A re-identification benchmark built at Imperial College London, and a better question than most evaluations ask. We ran our pipeline against it alongside Azure and a GPT-4.1 masking prompt, using the authors' own scoring criteria unchanged.
Benchmark by Krčo, Yao, Meeus and de Montjoye, Imperial College London, 2026
Instead of measuring recall (did you find the name?), it measures re-identification risk: given the anonymized text, can an attacker still find you among 3 million real Americans?
Re-identification is declared when a direct identifier is recovered, or when the combination of inferred indirect attributes narrows the pool to fewer than 5 people out of 3 million (k=5, θ=0.2).
That is the GDPR standard. Not « did you remove the obvious things », but « is re-identification reasonably likely ».
| System | Re-id · level 1explicit, standard | Re-id · level 2obfuscated, non-standard | BLEU · utility | Direct identifiers leaked |
|---|---|---|---|---|
| Hexagone AI | 14% | 27% | 0.83 | None |
| Azure (best NER) | 22% | 33% | 0.55 | Some |
| GPT-4.1 + Anthropic PII purifier prompt | 26% | 58% | 0.88 | Some |
Re-identification rate: lower is better. BLEU: higher is better.
Re-identification risk is the primary metric, but BLEU tells the other half of the story: how much of the original text survives. A low BLEU means the tool redacts aggressively and indiscriminately. A high BLEU means it removes only what needs to go.
Azure reaches decent privacy by removing everything, including content that has nothing to do with re-identification. BLEU 0.55 means roughly half the document is gone. What is left is often too damaged for an AI to reason over.
GPT-4.1 with a purifier prompt preserves text well, then falls apart at level 2, when identifiers are expressed in slang or non-standard form. 58% of the texts stay re-identifiable. It also means sending the original document to a third party in the first place.
The hardest part of anonymization is not removing names. It is understanding that « I have been in California for as long as I can remember, I am not married anymore, and I work the night shift at the hospital » might be enough to find you.
Benchmark credit. RAT-Bench was designed and built by Nataša Krčo, Zexi Yao, Matthieu Meeus, Yves-Alexandre de Montjoye at Imperial College London (2026). We did not build it and we do not control it. Hexagone AI results were obtained by running our own pipeline against the public benchmark, using the authors' scoring criteria unchanged. Every figure above is measured on the benchmark's own test set. Read the paper on arXiv
A benchmark is someone else's data. One week free, no credit card, and the files never leave your machine, so there is nothing to risk in checking for yourself.