PII-INDEXBENCH: 12,079 documents built to be hard
Every record through the production API, scored against the benchmark's own annotations alongside five other systems, one of which is Qwen3-14B prompted as a detector. The precision figure is taken apart against those same annotations further down the page.
Run by Hexagone AI, September 2026. Dataset PII-INDEXBENCH v1.2, CC BY 4.0.
12,079 synthetic English documents across 150 domains, built to be adversarial: identifier-dense support logs, clinical notes, machine transcripts, and the same person referred to three different ways within one document. It is not distributed on HuggingFace, which makes it a cleaner test for any system that might have memorised its own evaluation data during training.
Every record went through the production API in masking mode, and the output was scored against the benchmark's own annotations. The same five comparators ran on the same documents with the same scorer, four detectors and one prompted open-weights model. The primary metric is recall: of the 226,428 annotated personal values, what share did the output actually remove?
- Recall
- Share of the annotated personal values that the output actually removed, and the highest of the six systems compared here.
- Precision
- Share of what we removed that the benchmark also counts as personal data; the section below takes the figure apart bucket by bucket.
- Annotated values tested
- Across 12,079 documents and 150 domains, which is every record the benchmark ships rather than a subset of them.
Every kind of data the benchmark annotates
| System | Recallshare of personal data found | Precision | F1 | F1 · exact bounds |
|---|---|---|---|---|
| Hexagone AIapi.hexagone.ai, masking mode | 97.9% | 93.5% | 95.7% | 74.1% |
| Qwen3-14BQwen/Qwen3-14B, zero-shot, one request per document | 91.7% | 94.6% | 93.1% | 88.3% |
| OpenMed PIIOpenMed/OpenMed-PII-SuperClinical-Large-434M-v1 | 84.7% | 92.7% | 88.5% | 71.0% |
| OpenAI Privacy Filteropenai/privacy-filter, 1.5B, run locally | 77.5% | 94.0% | 85.0% | 41.9% |
| Microsoft Presidiopresidio-analyzer + spaCy en_core_web_lg | 65.4% | 91.7% | 76.4% | 42.5% |
| GLiNERurchade/gliner_multi-v2.1 | 51.0% | 90.0% | 65.1% | 62.4% |
This is the widest view and the least forgiving one: a system with no category for passports is charged every passport number as a miss. It is also the view that matches the product question, because a document does not stop containing passport numbers just because your model has no label for them. The last column scores the same predictions on exact character boundaries instead of any overlap, which is harsh on any tool that masks a name as two pieces.
| Kind of personal data | Spans | Hexagone AI | OpenMed | OPF | Qwen3-14B | Presidio | GLiNER |
|---|---|---|---|---|---|---|---|
| NamesPERSON | 46,589 | 99.9% | 99.8% | 98.4% | 90.8% | 84.5% | 81.9% |
| OrganizationsCOMPANY | 12,429 | 95.8% | 69.3% | no label | 86.9% | no label | 82.7% |
| Street addressesADDRESS | 20,351 | 95.5% | 76.1% | 79.8% | 92.2% | no label | 73.9% |
| PlacesLOCATION | 20,152 | 91.1% | 93.1% | 61.1% | 91.4% | 63.9% | 50.7% |
| Phone numbersPHONE_NUMBER | 15,673 | 99.9% | 96.9% | 99.1% | 94.7% | 78.1% | 34.8% |
| Email addressesEMAIL_ADDRESS | 17,145 | 100.0% | 99.7% | 99.9% | 94.5% | 46.9% | 10.6% |
| Social security numbersSSN | 3,222 | 100.0% | 99.5% | 95.4% | 96.7% | 99.2% | 40.5% |
| Dates and timesDATE_TIME | 20,163 | 95.4% | 98.2% | 70.3% | 78.7% | 82.5% | 70.3% |
| Card numbersCREDIT_CARD | 4,338 | 96.1% | 94.3% | 80.0% | 94.7% | 52.3% | 26.7% |
| Bank account numbersACCOUNT_NUMBER | 4,742 | 99.8% | 82.1% | 58.2% | 95.7% | 98.8% | 32.6% |
| National ID numbersNATIONAL_ID | 5,235 | 99.5% | no label | 71.1% | 97.1% | no label | no label |
| Passport numbersPASSPORT_NUMBER | 3,441 | 98.9% | no label | 83.9% | 96.4% | 99.9% | 47.5% |
| Driving licencesDRIVER_LICENSE | 3,088 | 97.9% | no label | 81.7% | 96.1% | 100.0% | 38.4% |
| Tax identifiersTAX_ID | 609 | 95.9% | 99.7% | 99.7% | 94.9% | 36.3% | no label |
| IP addressesIP_ADDRESS | 4,796 | 99.9% | 98.2% | 91.4% | 95.4% | 100.0% | no label |
| URLsURL | 3,625 | 100.0% | 77.8% | 11.0% | 96.2% | 89.3% | no label |
| Case, order and record numbersUNIQUE_IDENTIFIER | 28,682 | 99.9% | 53.9% | 74.3% | 95.2% | no label | 29.8% |
| Vehicle identifiersVEHICLE_ID | 795 | 93.1% | 88.3% | 97.2% | 95.1% | no label | no label |
| Passwords, keys and tokensPASSWORD | 6,776 | 99.6% | 99.6% | 97.6% | 98.3% | no label | no label |
| AmountsMONEY | 4,555 | 98.2% | no label | no label | 88.0% | no label | no label |
Best in each row is bold. « No label » means the system has no category for that kind of data at all, so it is neither scored nor penalised on that row, which is the generous reading of a missing vocabulary. The shading of each cell follows the value in it.
Where we do not come first. Restrict the scoring to the seven categories every detector in that table has a label for, and OpenMed edges ahead on both recall (97.9% against 97.7%) and F1 (93.6% against 92.7%). It is a supervised model with tight boundaries on the common types, and those seven are the common types. Our lead in the wider table comes from breadth instead: OpenMed has no label for national IDs, passports, driving licences, organisations, case numbers or amounts, and once those are counted its recall falls to 84.7%. Qwen3-14B sits outside that restricted view, since a prompted model has no fixed label set to restrict in the first place.
Dataset credit. PII-INDEXBENCH v1.2, released under CC BY 4.0. We did not build it and we do not control it. It is not published on HuggingFace, which is what makes it a reasonable test for any system that might have read its own evaluation set during training. Run of 2026-09-06, Hexagone production API, masking mode, with the benchmark traffic kept out of our database.
We remove more than the benchmark asks for
Precision counts a removal as wrong whenever it lands outside an annotated span, which only works as a definition if the annotation is complete. On a document where the same person is named four times and annotated once, it is not complete, and the three unannotated mentions are charged to us as errors.
So we took every span that cost us precision on this run and sorted it by rules that can be checked against the benchmark itself, rather than by whether we liked where each one landed. The buckets below are what came out.
The 29,597 spans that cost us precision
Every prediction on IndexBench that touched no annotated span, sorted by a stated rule.
- 37.4% A value the benchmark annotates elsewhere, or a piece of one (11,079)
- 39.7% Excluded from the annotations by design, or document scaffolding (11,760)
- 22.8% Genuine over-masking (6,758)
- 37.4%11,079 spans
A later mention of someone the benchmark annotates once
Rule. What we removed is a word of a value annotated elsewhere in the same document, a single letter, or a span that contains such a word without matching it. Nine in ten of them sit inside or beside a name the benchmark does annotate.
A complaint is signed Freya C. and the reply opens Dear Freya C. The benchmark annotates Freya Camara once, where she introduces herself, and neither of the two mentions that follow. The same shape covers an initial standing alone, as in « Xinyi H. » where Xinyi Hassan is annotated in the message header, and a boundary that lands on part of a name, as in « Sarah D » where the annotated value is Sarah Diallo. We remove all of them, since masking only the annotated mention would leave the name sitting in the document.
What Hexagone AI removed My name is Freya Camara and I am writing regarding the recent ⋯ Best regards, Freya C. --- Reply from Trust & Safety Team: Dear Freya C.,
What the benchmark annotates My name is Freya Camara and I am writing regarding the recent ⋯ Best regards, Freya C. --- Reply from Trust & Safety Team: Dear Freya C.,
Same excerpt on both sides, from PII-INDEXBENCH record doc-000001, verbatim. The ⋯ marks a cut between two passages of that one document.
- 34.0%10,075 spans
Document scaffolding masked conservatively
Rule. Speaker labels (Agent, User, Caller, Patient), field names (DOB, MRN, IP Address, State) and transcript timestamps.
We remove the name, the school and the town, which the benchmark asks for, and also the speaker labels and the transcript timestamps, which it does not. That is over-masking and we count it in full, though what it costs is readability rather than privacy, since nobody has ever been identified by the word « Agent » or by a timestamp. It is also the part of this figure we expect to bring down first.
What Hexagone AI removed [00:02] Agent: Good afternoon, I’m here to assist you today. Can I confirm I’m speaking with Edward Nguyen from Beacon University, located in Eastvale? [00:05]
What the benchmark annotates [00:02] Agent: Good afternoon, I’m here to assist you today. Can I confirm I’m speaking with Edward Nguyen from Beacon University, located in Eastvale? [00:05]
Same excerpt on both sides, from PII-INDEXBENCH record doc-000413, verbatim.
- 5.7%1,685 spans
Quasi-identifiers the benchmark excludes by design
Rule. What we removed matches a value the dataset maps to a label it drops: gender, job title, department, clinical terms.
The benchmark annotates the name, the date of birth and the email address, but deliberately not the gender, so a system is penalised for removing it. Under GDPR a gender combined with the other three is the sort of detail that narrows a person down, which is why we remove it anyway.
What Hexagone AI removed - Alex Lee, Gender: nonbinary, DOB: 11/22/1995, Email: alex.lee1942@example.com
What the benchmark annotates - Alex Lee, Gender: nonbinary, DOB: 11/22/1995, Email: alex.lee1942@example.com
Same excerpt on both sides, from PII-INDEXBENCH record doc-000028, verbatim.
- 22.8%6,758 spans
Genuine over-masking
Rule. Everything left over: relative dates (« yesterday », « next week »), acronyms (AI, FYI, ASAP, VIP), stray words.
« FYI » is not personal data and we should not have touched it. Roughly one in five of the flagged spans land in this bucket, and they set the honest floor under the precision figure: the other buckets have an explanation, this one does not.
What Hexagone AI removed FYI, your recent stay at hospit-179303 located in Lakeside, United Kingdom, involved treatment with medica-785190.
What the benchmark annotates FYI, your recent stay at hospit-179303 located in Lakeside, United Kingdom, involved treatment with medica-785190.
Same excerpt on both sides, from PII-INDEXBENCH record doc-000108, verbatim.
Which error we would rather make
Failing to remove a name is a personal data breach, while removing a field label that did not need to go leaves a document that reads slightly worse. We tune the system for the first of those and pay for it in the precision column. A tool with tighter precision than ours is usually making the opposite trade, agreeing with the annotation more often and leaving more names in your files, which is worth knowing before the two numbers are compared side by side.
Run it on your own documents.
A benchmark is someone else's data. One week free, no credit card, and the files never leave your machine, so there is nothing to risk in checking for yourself.