Building RAG on sensitive documents

Retrieval-augmented generation gives a language model the right context at the right moment. It is the cheapest way to make an AI useful on documents the model has never seen. It is also a privacy minefield when those documents are sensitive. Here is the architecture we use to defuse it.

What RAG does, briefly

A RAG pipeline turns your documents into vectors, indexes them, and at query time retrieves the most relevant chunks to send to a language model along with the user's question. The model then answers grounded in those chunks. Compared to fine-tuning, RAG is fast to set up, easy to update, and works on small volumes.

The privacy problem

A standard RAG pipeline sends sensitive content to a third party at three different moments:

• Embedding. The original text is sent to an embedding model (OpenAI, Cohere, Voyage). Even when the API does not retain the text, the bytes left your network.
• Retrieval payload. At query time, the retrieved chunks are inserted into the prompt and sent to the LLM. Names, amounts and dates travel with them.
• Persisted index. The vector store often lives in the cloud. Encryption at rest is not enough when an admin or a court order can compel access.

Three approaches, ranked by safety

A. Fully local RAG with a local LLM

Everything runs on your machine: embedding, retrieval, generation. Strongest privacy guarantee. Hardest to operate at quality. Useful for highly sensitive workflows where good-enough quality beats best-in-class.

B. Anonymized RAG with a cloud LLM

Documents are anonymized before embedding. The vector store holds only protected chunks. At query time, the retrieved chunks (still protected) are sent to a cloud LLM. The answer comes back with markers, which are swapped locally to the real values. This is the architecture we recommend, and the one Hexagone AI plugs into.

C. Cloud RAG, raw documents

The default in most demos. Easy. Cheap. Catastrophic on sensitive data. Avoid.

The architecture, step by step

1. Ingest, locally. Hexagone AI scans each document with its local detection engine. Identifiers are replaced with stable markers (CLIENT_3, AMOUNT_8, DATE_14). The mapping is written to your local mapping store.
2. Embed the protected version. Only the protected text is sent to your embedding provider, or to your local embedding model. The vector store, wherever it lives, never holds your real client data.
3. Retrieve on protected. A user question is itself anonymized through the same mapping before being embedded, ensuring the query and the index speak the same protected language.
4. Generate. Retrieved protected chunks + the protected question go to your LLM. The LLM answers in protected language, using the same markers.
5. Restore, locally. On your machine, the answer is rendered with the real values. The user reads a clear answer. The cloud LLM never saw a real one.

Performance notes

• Anonymized embeddings retrieve as well as raw embeddings on most tasks. Loss is in the low single digits in our benchmarks.
• Stable markers are essential. Random tokens kill recall.
• Indexing throughput stays similar to a vanilla pipeline on modern hardware.

Where Hexagone AI fits

We do not replace your vector store, your embedding provider, or your LLM. We sit at the ingestion layer and at the query layer, on the user's machine. The rest of the stack stays exactly as it is. Your existing RAG, with the privacy properties most teams thought they had from the beginning.