PrivacyBench v2: anonymizing whole workspaces, coherently
Tonic AI's benchmark of 21 workspace exports (email, Slack, PDF, Word, Excel, CSV), scored on two things at once: whether each personal value is found, and whether every way of writing one person or company is replaced by one consistent invented identity. Each export ran as one job, judged by the benchmark's own evaluation code.
Run by Hexagone AI, October 2026. Dataset TonicAI/Privacy-Bench v2.
What PrivacyBench measures
PrivacyBench v2, published by Tonic AI, is 21 synthetic workspace exports: one month of email, Slack and shared-drive documents (PDF, Word, Excel, CSV) for one fictional employee each, about 200 files per export. A system receives the raw export and must replace every personal or organisational value in place.
Detection is only half of the score. The other half is coherence: every way a person or a company is written (a full name, a nickname in Slack, an email address, a handle, an acronym) must be replaced by one consistent invented identity, across all files. An LLM judge checks that, per character and per organisation.
The bottom line is the combined score: the share of all ground-truth values that were both found and replaced coherently, averaged over the 21 exports.
Results on all 21 exports
| Pipeline | Detection (recall) | Coherent replacement | Combined |
|---|---|---|---|
| Hexagone AI Local | 93.7% | 94.0% | 88.1% |
| Tonic Textual (Graph Synthesis) | 93.9% | 91.8% | 86.2% |
| GPT-6 Astra | 91.6% | 87.5% | 80.1% |
| Claude Opus 5 | 93.6% | 85.5% | 80.0% |
| GPT-5.6 Sol | 89.9% | 81.6% | 73.4% |
| Claude Sonnet 5 | 89.0% | 79.3% | 70.6% |
| GPT-5.6 Terra | 84.9% | 78.6% | 66.7% |
| GPT-5.6 Luna | 54.7% | 77.4% | 42.3% |
Macro average over the 21 datasets, each weighted equally, as on the dataset card. The other rows are the dataset card's own published figures; ours were produced with the benchmark's unmodified evaluation code and its default judge model. Hexagone AI Local runs on small models that can be hosted on your own infrastructure.
Read plainly: 1.9 points ahead of Tonic Textual (Graph Synthesis), Tonic AI's own product, and 8.0 points ahead of the best frontier-model agent (GPT-6 Astra). The lead comes from coherence (2.2 points ahead); on detection alone we are 0.2 points of recall behind. Our weakest file type is Slack, where short forms of names and companies are hardest to tie to one identity.
By file type
| File type | Values | Recall | Coherent | Combined |
|---|---|---|---|---|
| 13,253 | 94.9% | 93.2% | 88.4% | |
| Slack | 18,282 | 92.0% | 88.7% | 81.6% |
| 44,876 | 93.2% | 94.5% | 88.1% | |
| Word | 15,733 | 92.7% | 95.4% | 88.5% |
| Excel | 13,960 | 94.6% | 94.5% | 89.4% |
| CSV | 26,639 | 94.3% | 96.4% | 90.9% |
By type of value
| Value | Count | Recall | Coherent | Combined |
|---|---|---|---|---|
| Given name | 46,607 | 97.6% | 91.3% | 89.1% |
| Family name | 27,580 | 97.7% | 99.6% | 97.2% |
| Email address | 14,395 | 99.3% | 98.7% | 98.0% |
| Phone number | 6,216 | 99.7% | 99.3% | 99.0% |
| Username / handle | 1,699 | 86.7% | 93.4% | 81.0% |
| Organization | 31,939 | 81.9% | 90.3% | 74.0% |
| Postal address | 1,699 | 74.0% | 73.2% | 54.1% |
| URL | 408 | 33.6% | 25.5% | 8.6% |
| Employee ID | 1,560 | 90.0% | 96.9% | 87.2% |
| Account number | 640 | 92.7% | 97.8% | 90.6% |
Counts are summed over the 21 exports (pooled), so frequent values weigh more here than in the headline average.
How it was run
- One job per export, as a customer would submit it: every file of the export in a single anonymization job, so coherence is the product's own, across mail, chat and documents.
- The benchmark's own evaluation code, unmodified, with its default judge model (claude-opus-4-7). We checked the judge path on the dataset's published example output and reproduced its documented scores within the judge's own variation.
What to keep in mind
- The ground truth is generated, not hand-labelled; the dataset card measures it against human annotations and finds it slightly incomplete. A judge is a model: on identical output, its verdicts vary by about a point.
- A first name shared by two different people (two colleagues called Sarah) receives the same replacement for both. This is our product's default, chosen because a bare "Sarah" in a chat cannot always be attributed to one of them; the judge checks consistency per character, and this choice keeps every mention of each character consistent.
The other studies
Run it on your own documents.
A benchmark is someone else's data. One week free, no credit card, and the files never leave your machine, so there is nothing to risk in checking for yourself.