200 French medical documents, two thirds of them scans
Public PII benchmarks are plain text; a hospital's problem is a scanned Cerfa with handwriting across it. So we built the corpus: twenty patients, ten documents each, every identifier labelled down to its box on the page, and two thirds re-scanned with skew, noise and a stamp over the text. Then we sent all 200 through the API as a single job and scored what came back.
Built and run by Hexagone AI, September 2026. Clinical content from PARHAF, Health Data Hub / AP-HP, CC BY 4.0; every identifier in it is invented.
A corpus that did not exist
Public PII benchmarks are plain text. A hospital's problem is a scanned Cerfa form with a doctor's handwriting across it, a stamp over the patient's name and a fax header down the side. So we built the documents, from clinical reports written by clinicians, and labelled every identifier down to where it sits on the page.
Look at the documents| Level | Docs | Delivered as |
|---|---|---|
| easy | 67 | digital PDF, typed, clean |
| medium | 86 | scan at 200 dpi, tidy handwriting |
| hard | 47 | photocopy or fax, hurried hand, stamped |
Take the dataset
Every document, and for each one a JSON ground truth giving each identifier its label, its owner and its box on the page. Clinical content is CC BY 4.0 from PARHAF; everything identifying in it is invented.
Download medical-dataset-v1How the corpus was built, in four steps
- 1Start from real clinical writingPARHAF (Health Data Hub / AP-HP, CC BY 4.0) is fictitious patient reports written by clinicians. Real clinical language, no layout, almost no identifiers.
- 2Mint a consistent identityEach patient gets a name, an address, a NIR, an IPP and a set of doctors, identical across all ten of their documents. Phone numbers come from the ranges ARCEP reserves for fiction; the checksummed identifiers are valid but invented.
- 3Lay it out as the real thingFaithful Cerfa forms, prescriptions on a pad, lab tables, referral letters. The generator knows where it drew every value, which is what makes a box on the page possible at all.
- 4Then damage it, on purposeTwo thirds are re-scanned as images: handwriting with per-letter jitter, up to 2.8 degrees of skew, JPEG noise, greyscale photocopying, a stamp over the text. The text layer is gone; what is left is what a scanner would give you.
What came back
Two questions, kept apart. Did the API report a value covering each labelled identifier, which its own audit record answers. And is the original value still readable in the document it delivered, which only the delivered PDF answers. The second is the one that matters, and it is measured on the file, not on the claim.
What it found, by kind of identifier
| Identifier | On the pages | Found | Still readable |
|---|---|---|---|
| Names | 2,385 | 99.1% | none |
| Addresses | 1,370 | 98.6% | none |
| Dates | 1,251 | 99.4% | none |
| Health and admin numbers | 729 | 97.3% | none |
| Organisations | 319 | 99.4% | none |
| Contact details | 549 | 97.6% | none |
See all 23 labels
Age scores badly here on purpose. It is a quasi-identifier the dataset labels but does not require removing, and the default policy leaves it: it is reported rather than quietly excluded to flatter the average.
| Identifier | On the pages | Found | Still readable |
|---|---|---|---|
| SurnameSURNAME | 1,199 | 98.8% | — |
| Given nameGIVEN_NAME | 1,186 | 99.3% | — |
| DateDATE | 838 | 99.5% | — |
| CityCITY | 549 | 98.9% | — |
| Postal codePOSTAL_CODE | 401 | 99.0% | — |
| StreetSTREET | 360 | 97.5% | — |
| PhonePHONE | 323 | 96.0% | — |
| EstablishmentESTABLISHMENT | 307 | 99.4% | — |
| Date of birthBIRTH_DATE | 286 | 99.7% | — |
| Hospital patient number (IPP)IPP | 225 | 98.7% | — |
| Doctor's registration (RPPS)RPPS | 182 | 98.9% | — |
| TimeTIME | 127 | 98.4% | — |
| URLURL | 106 | 100.0% | — |
| Facility number (FINESS)FINESS | 101 | 86.1% | — |
| National health number (INS)INS | 101 | 99.0% | — |
| AgeAGE | 75 | 28.0% | — |
| FaxFAX | 60 | 100.0% | — |
| EmailEMAIL | 60 | 100.0% | — |
| Admission number (NDA)NDA | 60 | 100.0% | — |
| Place of birthBIRTH_PLACE | 60 | 100.0% | — |
| Health insurance numberAM_NUMBER | 60 | 100.0% | — |
| Lab referenceLAB_ID | 40 | 100.0% | 2 |
| EmployerEMPLOYER | 12 | 100.0% | — |
And by how hard the page is to read
| Difficulty | Identifiers | Found | Still readable after |
|---|---|---|---|
| easydigital, typed | 2,185 | 98.9% | none |
| mediumscanned, handwritten | 2,540 | 99.6% | none |
| hardphotocopied, stamped | 953 | 95.6% | 2 · 0.2% |
The hardest documents find the fewest identifiers and leak the least, and those two facts have the same cause. A photocopied page goes through OCR, and where OCR misreads a value the pipeline never sees it, which is a detection miss; but the misreading is not the value either, and the delivered page is rebuilt from what OCR read, so the original rarely survives into it. A low leak rate on a scan is therefore a weaker claim than the same rate on a digital PDF, and it is worth saying so.
Look at the documents
Every document of the run, the original and what the API delivered, with the dataset's own labels drawn where they sit. Drag the handle across the page to see one become the other. The figures above are an average of these; this is how you check one.
- Nothing matches.
Page images are renders of the real PDFs, the ones the dataset ships and the ones the API returned. Nothing on either side is retyped.
How to read this
Which documents are scored
All 200: every document was submitted, the ones with a superscript digit included, and every one was delivered.
What counts as found
The API reported a value that covers the labelled one. A full name reported once answers two labels, and a value read slightly wrong by OCR still counts, because a benchmark that punishes a scanner for being a scanner measures the scanner.
What counts as a leak
The original value is still in the text of the delivered PDF, on a word boundary, or its digits are still there with only separators between them. Read from the file, never from the audit record.
What was not measured
Re-identification. Every identifier can be gone and a rare diagnosis in a small town still name someone; that is a different study and this page does not claim it. The handwriting is also synthetic, convincing but not a real hand.
Run it on your own documents.
A benchmark is someone else's data. One week free, no credit card, and the files never leave your machine, so there is nothing to risk in checking for yourself.