Access to information, with every number measured
Hundreds of pages, redacted in an afternoon, and a person still decides every line
When someone asks a public body for its records, staff read every page and decide, line by line, what must be withheld and what must be released. The law turns on context: the requester's own details go back to them, an official signing a letter stays, a neighbour who made a complaint is withheld. Off-the-shelf privacy tools black out every name and number, which answers the request in form and refuses it in substance, so the reading is still done by hand. This tool reads the record the way a reviewer does, proposes every redaction with the section of the law it rests on, and leaves the decision on every page to a person, with a log nobody can quietly edit.
Reading this server's live count of decisions.
Every measured figure on this page is read from a table the measurement commands write, and carries a 95% confidence interval there; nothing is typed by hand. The one estimate, hours a year, multiplies the measured saving by the government's own published page count, and the study says how. The button above is the reviewer's own screen over six invented records, with nothing kept once you leave. See how it's built.
Why it is still done by hand
The same kind of detail, a name or a telephone number, is released on one line and withheld on the next, and nothing in the name says which. What decides it is who the person is in this record. That is the judgement privacy tools do not attempt, so they withhold everything, and a public body that used one would be refusing most of what the law says it must release.
| Who is named | What the law says | A blunt privacy tool | This tool |
|---|---|---|---|
| The person who asked | Their own information goes back to them | Withholds it from them | Releases it, matched to the request |
| An official signing a letter | Name, title and work contact relate to the position, so they stay | Withholds them | Releases them, from the title in the signature block |
| The minister on the letterhead | Already public | Withholds it | Releases it, from the letterhead |
| A neighbour who complained | A third party's personal information is withheld | Withholds it | Withholds it |
| "The client's sibling" | Identifies the sibling to a requester who knows the family | Releases it: no name, so nothing to find | Withholds it with the name |
| Someone the record cannot place | It depends, and a person has to decide | Withholds it, silently | Withholds it and says it needs a decision |
Federally this is the Access to Information Act and the Privacy Act; in Newfoundland and Labrador, ATIPPA. The categories and the sections they rest on are in The law, as data.
One memorandum, read the way a reviewer reads it
This is the tool's own output, produced when this page was built, on an invented human-resources memorandum from the demonstration: the findings of a confidential workplace complaint. The request was made by Fiona Kavanagh, a member of the public who is named in it. Every highlight is a proposal: the colour is what the tool would do, and pointing at one, or tabbing to it, shows the category and the reason.
Department of Justice and Public Safety Human Resources Division - IN CONFIDENCE MEMORANDUM To: Desmond Okonkwo, Director of Operations From: Bernadette Halleran, Program Officer Date: 11 May Subject: Complaint under the Respectful Workplace Policy - file HCS-2023-6373 This memorandum sets out the findings of the review I was asked to conduct. The complaint was made by Aaron Marchand, who is employed in this Department as Director of Operations. In the complaint Aaron describes a pattern of conduct over eleven months and states that the conduct has affected their health. A medical certificate was provided in support and is held separately; it records a period of leave from 3 February and gives a personal contact number of 709-555-1226. Aaron asked that their home address, 155 Blackmarsh Road, not be shared with the respondent. The complaint names Malcolm Rideout, Regional Director, as the respondent. The conduct described concerns the allocation of work and the conduct of team meetings, all of it within the respondent's duties as Regional Director. Malcolm responded in writing on 24 April and was interviewed on 2 May. Two further employees were interviewed. Both spoke about the operation of the unit rather than about either party personally, and both asked that their accounts be attributed to their positions rather than to their names. A member of the public, Fiona Kavanagh, who had a service complaint outstanding during the same period, is named in the complaint file at Tab 6 because a meeting about that complaint is one of the incidents described. Nothing in this review concerns Fiona's own matter and no finding is made about it. Findings: the allegation of an unreasonable allocation of work is not substantiated. The allegation concerning the conduct of meetings is partly substantiated. I recommend mediation and a review of the unit's meeting practice. Bernadette Halleran Program Officer bernadettehalleran@gov.nl.ca.example
Every category in this memorandum, with the tool's reason for its first use
- K1 Fiona Kavanagh: The requester's own personal information. The requester's own information: matched the configured requester on 'Fiona Kavanagh'.
- K2 Bernadette Halleran: An official of a public body acting in that capacity. Named with the title 'Program Officer' in signature, so this is an officer or employee acting in an official capacity.
- R1 Aaron Marchand: Third-party personal information. 'Aaron Marchand' is a party to the complaint in a confidential workplace complaint. That is about the person, not their position, and a title would identify them.
- E1 Desmond Okonkwo: Role undeterminable from the record. No title, no signature-block or letterhead mention, no register entry and no match to the requester. The record does not say whether 'Desmond Okonkwo' is an official or a third party.
The same words, different answers, because of who they belong to. "Director of Operations" appears twice. On the To: line it is the recipient's title, which says nothing about a person, so it stays. In the body it is the complainant's, and it is withheld with "Regional Director", the respondent's: they are the titles of the two parties to a confidential complaint, and a title released beside a severed name identifies the one person who holds it. The Program Officer who wrote the memorandum is an official doing their job, so their name and work email are released. The requester gets their own name back and nothing else of a file about other people.
When it is unsure, it withholds and asks. The memorandum's recipient, Desmond Okonkwo, is escalated, though a reviewer would release the name: the title sits beside it on the To: line, and the rules do not yet pair the two there. That is the direction the tool errs in on purpose, and the reviewer releases it with one key.
One page, in seven steps
The order is the design. The context is built before anything is decided, the rules decide what has a settled answer, and nothing is released until a person has acted on every page.
- 1ReadRecover the layout: letterhead, email headers, signature blocks, even from a scan.
- 2FindEvery name, number, address and title that could be personal information.
- 3PlaceBuild each person's context: their title, their block, every other mention of them.
- 4DecideFixed rules in a fixed order, each tied to a section of the law.
- 5AskOptionally, a model on what is left, seeing placeholders, never a name.
- 6ReviewA person approves or overrules every proposal, page by page, all of it logged.
- 7ReleaseRemove the text under every box, then test that nothing is left.
Step 6 is highlighted because it is the only one that can release anything. How it's built follows a span through all seven.
Does it work?
Two rates, published side by side, because either alone can be gamed
Measured on 210 invented records holding 5,460 labelled spans. The people were cast before the documents were written, so the right answer for every span is known by construction rather than by someone's opinion. Each part starts in plain language; the detail for someone who builds these systems is folded under For the technical reader.
Withholding what must go, releasing what must stay
A leak is a third party's information released. Over-redaction is something the requester was entitled to, withheld. A tool can drive the first to zero by withholding everything, and the blunt tool nearly does; the price is the second number, which is the one nobody publishes. Lower and further left is better on both.
The leak rate is above the blunt tool's, and it is meant to be visible. The gap, 2.5% against 1.1%, is what releasing anything at all costs. A tool that never released a span would have the lowest leak rate on the page and no reason to exist. A person reviews every proposal precisely because of that gap, and the study measures what is left after they do.
For the technical reader: the five configurations and the interval
| Configuration | Leak rate | Over-redaction | Escalated |
|---|---|---|---|
| The tool: rules only, as it ships | 2.5% 1.7 to 3.5% | 22.4% 20.9 to 23.9% | 37.3% |
| Rules, then a model on the rest | 4.6% 3.6 to 5.7% | 19.7% 18.1 to 21.2% | 22.7% |
| A model on everything, no rules | 15.4% 13.8 to 17.0% | 41.0% 38.4 to 43.6% | 40.8% |
| The rules, with the document's context blanked | 8.8% 6.9 to 10.8% | 49.3% 47.1 to 51.4% | 59.0% |
| Blunt: withhold every detected span | 1.1% 0.6 to 1.6% | 98.4% 97.9 to 99.0% | 0.0% |
- Denominators. The leak rate is over the 1,680 spans that must be withheld; over-redaction over the 3,780 that must be released. Escalations withhold, so they count against over-redaction and never against the leak rate.
- A partial box counts the way a reader would. A span is withheld only when every letter and digit of it is under a box, or what shows identifies nobody: four or fewer digits of a long number, or a job title with its role word withheld.
- Intervals are bootstrap percentiles resampled over the 210 documents, not over spans, because one invented person produces many spans and resampling them would count one fact many times. Project 03 implemented the same estimator independently; a test pins that the two agree.
- Tables generated 2026-10-10 by
sever report, spliced into the README by the same command.
The context is the whole argument
A person's capacity is almost never in the sentence that names them. It is three inches lower, in the signature block, or at the top, on the letterhead. So for every span the tool builds a context record: the kind of block it sits in, the heading above it, whether a title is paired with this person anywhere in the record, whether they are on a public register, every other mention of the same person, and whether they are the requester. Blank that record and keep the same rules, and over-redaction more than doubles.
Category by category
What each kind of release costs. Officials are most of what must be released, so theirs is the figure to watch, and most of it is escalation rather than a wrong answer: a name or a work address the record never pairs with a title, which the tool will not guess at. The twenty-year rule withholds about half of what it should release, because it acts only when the record states a date of death.
For the technical reader: what a published staff directory would do to the officials' figure
- Matched on the work email address alone, a directory takes the officials' over-redaction from 30.1% to 18.8% with the leak rate unmoved at 2.5%: the address is on the public body's own domain and nobody outside the directory shares it.
- Matched on names it multiplies the leak rate by nine, from 2.5% to 22.0%. See What did not work.
- The headline tables use no directory, because a directory is something a public body supplies. results/directory.md.
Where every leak comes from
A leak can only happen four ways: the detector never found the span; it found it and called it a place or an organisation, which is not personal information; the box was too short; or the decision was wrong. Counting them separately says which part to fix, and a two-way split once pointed a fix at the wrong module.
The rules themselves leak nothing. Of the rules-only row's 42 leaks, 17 were never found, 24 were found and typed as a place or an organisation, 1 left part of a name showing, and 0 were a wrong decision. The weak part is detection, and that is where the work goes.
Three people, timed
The claim in the title is about people, so it was measured on people. Each reviewer did twenty records: ten by hand, finding and categorising every severance themselves, and ten with the tool's proposals, with a practice page before each half. The halves were swapped between reviewers, so every record was reviewed both ways.
What that adds up to
Federal institutions processed 10,326,722 pages of access-to-information requests in 2023-24, by the Treasury Board's statistical report (Immigration, Refugees and Citizenship Canada excluded, as the report excludes it). At the 74 seconds a page the study measured, that is about 212,000 hours of reading a year, or 118 people reviewing full time (1,800 hours a year each).
This is arithmetic on a measurement, not a measurement. The seconds come from one-page invented records, and a real page can be denser or emptier; it counts only the reading and deciding, not finding the records, consulting other bodies or writing the response. Read it as the scale of the work the tool is aimed at.
Faster and more accurate, which is the combination that matters. By hand, the misses were the kind people make: a first name standing alone later in the text, the second appearance of a full name, a selection one character short of an email address. With proposals, the misses were the tool's own boxes, approved as drawn. Both kinds are why the screen now withholds every other occurrence of a value the moment a reviewer withholds one.
For the technical reader: what the study can and cannot say
- Three people, the first with eight years in access-to-information work. Every reviewer did the proposals half second, so some of the gain is familiarity with the templates; the interval resamples records, not people.
- The proposals are frozen as they stood for the first reviewer, so every later reviewer saw the same tool and the arms stay comparable as the tool improves.
- The reviewers changed the labels twice, and every table carries both rulings: a family word that points at a third party is withheld with the name, and the parties to a confidential workplace complaint are withheld with their titles. Where a reviewer departed from a label and the label stood, it counts against the reviewer.
- Seconds are the server's own, from serving a page to its completion row in the log. results/reviewer.md and docs/reviewer-study.md.
A black box is not a redaction
The most common redaction failure in the news is a black rectangle drawn over text that is still in the file, one copy and paste from public. Every export here removes the text under each box, and then a test tries to get it back two ways: from the PDF's text layer, and by rendering the page to an image and reading it with OCR.
Try it: review a record, release it, download the PDF and select the text under a box. There is nothing there to select.
Scanned paper
Many records arrive as scans. The same 50 records were printed to PDF, degraded the way a scanner, a photocopier and a fax degrade a page, and read back with Tesseract. What a good scan costs is not the characters; it is the layout. A scan has no signature block in it, only lines of text, so the tool reads the blocks back from the words a signature and an email header are made of.
A scanner or a photocopier costs about a point; a fax costs the record. Reading the layout back takes an office scanner's over-redaction from 39.9% to 23.1%, against 22.1% on the native file. At 65.2% of characters misread, a fax destroys the values before any detector sees them, and the leak rate is 78.7%. Fax-quality records need a person, and the tool's number says so.
Real released records
Invented records have exact labels and template-shaped text. Published federal releases are the opposite: real, untidy, and everything still visible was judged releasable by a person at Library and Archives Canada. So the question asked of them is one-sided but honest: of what a person released, how much would this tool have withheld? 15 packages, 288 pages, in three sets, the last two chosen by a rule written before any was downloaded.
This is the tool's weakest result, and it is published as such. On the Defence records it learned from, teaching it military ranks and reading layout back took it from withholding everything to 41.6%. On records it never saw, it still withholds 95.2% and 88.0%: heads of state it has no register for, family words in century-old immigration files that the archive opened because everyone in them was born more than 110 years ago. The gain did not travel, and the page says so rather than quoting only the first set.
For the technical reader: why a leak rate cannot be reported here
- One-sided by construction. What sat under the original black boxes was removed before publication, so there is no must-withhold denominator. The scoring code raises rather than return a leak rate on this track.
- A floor, not a forecast. These are OCR text layers of scanned forms with no layout and no configured requester, so every effect pushes towards withholding.
- The archive's own age rule (release once 110 years have passed since birth, Privacy Regulations s. 6(c)) was built and registered before the third set was opened; no page in it was old enough for the rule to read, so it decided nothing there, and that is reported too.
- releases.md, releases-heldout.md, releases-fresh.md, and the selection rules in docs/data.md.
The language model, measured and switched off
The rules decide what has a settled answer and escalate the rest. A language model can be asked about what is left, through the portfolio's own AI gateway, which replaces every name with a typed placeholder before the request leaves, so the model never sees one. It was built, measured at a threshold chosen on separate records, and then turned off by default, because of what the measurement said.
The gains go to a requester, sooner; the losses go to a stranger, for good. The model releases more of what requesters are owed, and leaks more of what third parties are owed protection from, and the two do not weigh the same: a requester can appeal a withheld line, and a third party's details in a stranger's hands cannot be recalled. The portfolio's release gate, asked the same question with the document as the unit, blocks it: 83.3% of documents leak-free with the rules alone, 66.7% with the model. Which to run is a public body's decision; the default is the rules.
For the technical reader: the warning worth more than the number
- A model measured on top of imperfect rules is partly being paid to fix them. The model's case was far stronger until the rules stopped escalating bare job titles, and nothing about the model changed. Anyone comparing "rules" with "rules plus a model" is measuring the rules they happen to have.
- Without the rules the model leaks 15.4% against 4.6% with them. Asked on its own, it releases "the client's sibling" almost every time, because a family word with no name beside it reads as harmless.
- No raw entity leaves. A test asserts that no detected value appears in any outbound request; the gateway records the data class of every call. 2,002 questions were asked for the published tables, and answers are cached so the tables reproduce for nothing.
The law, as data
Every proposal names a category, and every category maps to the sections of the legislation it rests on, in two jurisdictions. The mapping is a data file, not code, so a public body can read it, argue with it and change it. Each of the 39 provisions carries the quoted text, the date it was checked and the address it was checked against, and the file refuses one that has none.
| Category | Canada (federal) | Newfoundland and Labrador |
|---|---|---|
| R1 Third-party personal information | WithholdATIA 19(1), Privacy Act 3 | WithholdATIPPA 40(1), ATIPPA 2 |
| R2 Mixed span, third-party information entangled with the requester's | WithholdATIA 25, ATIA 19(1) | WithholdATIPPA 8(2), ATIPPA 40(1) |
| K1 The requester's own personal information | ReleasePrivacy Act 12(1) | ReleaseATIPPA 8(1) |
| K2 An official of a public body acting in that capacity | ReleasePrivacy Act 3(j), Privacy Act 3(k), Privacy Act 3(j.1) | ReleaseATIPPA 40(2)(f), ATIPPA 40(2)(g), ATIPPA 40(2)(h) |
| K3 Already publicly available | ReleaseATIA 19(2)(b) | Withhold and escalateATIPPA 40(2)(m) |
| K4 An individual dead more than twenty years | ReleasePrivacy Act 3(m) | Withhold and escalateATIPPA 40(5)(j) |
| K5 Consent given, or a discretionary benefit of a financial nature | ReleaseATIA 19(2)(a), Privacy Act 3(l) | ReleaseATIPPA 40(2)(b), ATIPPA 40(2)(l) |
| K7 An archival record, about an individual born more than 110 years ago | ReleaseATIA 19(2)(c), Privacy Act 8(3), Privacy Regs 6(c) | Withhold and escalateno provision cited: a person decides |
| K6 Not personal information | ReleasePrivacy Act 3 | ReleaseATIPPA 2 |
| E1 Role undeterminable from the record | Withhold and escalateno provision cited: a person decides | Withhold and escalateno provision cited: a person decides |
| E2 Mixed span the rules cannot sever cleanly | Withhold and escalateATIA 25 | Withhold and escalateATIPPA 8(2) |
| E3 Candidate non-personal exemption | Flag for the reviewerATIA 16, ATIA 20, ATIA 21, ATIA 23 | Flag for the reviewerATIPPA 29, ATIPPA 30, ATIPPA 31, ATIPPA 39 |
ATIA is the Access to Information Act; ATIPPA is Newfoundland and Labrador's Access to Information and Protection of Privacy Act, 2015. Where the two disagree, as on what is already public, the provincial file escalates rather than releases. The tool decides only personal information: advice, legal privilege, law enforcement and business interests are flagged for the reviewer with the provision and never decided.
What did not work
Looking officials up in the staff directory by name. Most of the over-redaction of officials is a name the record never pairs with a title, and the obvious remedy is to look the name up. Measured, it multiplied the leak rate by nine, from 2.5% to 22.0%. Some of that is the small invented name pool, but 35 of the leaks were the listed employee themselves: an employee who files a grievance is in the directory and is a third party in their own complaint. A directory is evidence about a string; the decision turns on the sentence. Only the work email address survived, and it is now the only thing a directory is matched on.
What this does not do
It does not release anything
Every page needs a person's explicit action, and the export asks the audit log, which refuses while any page is undecided.
It decides personal information only
Advice, legal privilege, law enforcement and business interests are flagged with the provision and left to the reviewer.
It is not legal advice
The categories map to two statutes and are configurable. A public body applies its own interpretation.
The corpus is invented
Exact labels need people cast before the documents are written. Real releases, above, are where that is checked, and they are its weakest result.
It does not read handwriting
The scanned track covers typed pages. A fax-quality page loses most of its values before detection.
The study is three people
Each did both halves, proposals second, so familiarity is part of the gain. The interval resamples records, not reviewers.
How it is engineered
- Python 3.13
- Presidio
- spaCy
- PyMuPDF
- Tesseract
- FastAPI
- Jinja
- DuckDB
- Pydantic
- YAML taxonomy
- Hash-chained audit log
- Bootstrap intervals
- Paired non-inferiority
- mypy --strict
- ruff
- pytest
- uv
- Docker
- Caddy
- GitHub Actions
- OVHcloud
- Cloudflare
For the technical reader: the rules the code enforces on itself
- No path to release without a person. The export asks the session's release gate, which reads the audit log and raises while any page lacks a completion row.
- Withhold and escalate when unsure. An undecided span is withheld and flagged; no rule releases a span because nothing matched.
- The audit log is append-only. A reviewer who changes their mind adds a row; each row's hash covers the one before it, and the log stores a hash of the withheld text rather than a second copy of it.
- Every model call goes through the gateway as personal data, and a test asserts that no raw detected value appears in an outbound request.
- Every reported number carries an interval and its track, and every table is written by the command that measured it. This page reads those tables, so it cannot disagree with them.
How this was built
Design, methodology, evaluation choices and judgement are Peter Parker's, from eight years inside a public body that answers access-to-information requests, where the question on this page is asked about every page released. AI coding assistants (Claude Code) were used for implementation and drafting, the way a senior engineer uses them in 2026. Every number on this page is reproducible from the repository.
Glossary
- Access-to-information request
- A request by anyone for a public body's records, which the body must answer, withholding only what the law allows.
- Confidence interval
- The range a measured rate would plausibly fall in if the measurement were repeated. A bare number without one is treated as a defect here.
- Context record
- What the tool knows about one span before deciding it: its block, its heading, the person's title anywhere in the record, other mentions, a register entry, and whether they are the requester.
- Escalation
- A proposal to withhold and ask a person, because the record does not say enough to decide.
- Leak rate
- Of the details that must be withheld, the share released.
- Over-redaction
- Of the details that must be released, the share withheld. The number blunt tools never publish.
- Severance
- One piece of a record withheld under a provision, listed in the schedule sent to the requester.
- Span
- One stretch of text the tool found and decided: a name, a number, an address, a title.