Preventing RAG Data Poisoning: Source Provenance, Quarantine, and Re-Indexing Tests

A website chatbot can deliver a polite, linguistically convincing, and technically flawlessly generated response—and yet operate on a poisoned knowledge base. With RAG data poisoning, the formulation of an individual query is not the primary target of manipulation. Instead, false, corrupted, or inadequately verified content enters the persistent data chain: source, parser, chunk, metadata, embedding, and finally the production retrieval index. The error thus persists across many sessions and can also affect ordinary questions.

Effective protection therefore begins long before the prompt. Teams must be able to answer for every knowledge element: Where does it come from, who is responsible, which version was processed, what transformations occurred, and through which check was it approved for search? Source provenance provides this audit trail. Technically separated quarantine can prevent unverified changes from becoming immediately searchable. Targeted re-indexing tests subsequently check whether cleaned content has actually replaced old chunks.

What RAG Data Poisoning Is—and What It Is Not

The OWASP classification LLM04:2025 on Data and Model Poisoning describes manipulations of pre-training, fine-tuning, or embedding data as an integrity risk. For a website chatbot, the latter variant is particularly relevant: A document is ingested and broken down into sections; these chunks are embedded and stored as vectors in the retrieval index. If this document is intentionally or accidentally corrupted, it can appear as a seemingly relevant basis for matching queries.

The risks must be distinguished, but they can overlap: Prompt injection attempts to inject instructions or data at runtime in a way that alters the system's intended behavior; indirect prompt injection can also enter the context via retrieved documents. Data poisoning, on the other hand, alters the longer-lived knowledge base or its derivatives. Access rights also solve a different problem: They determine which person is allowed to see a document. Provenance and approval determine whether this document should enter the index as a trusted source of knowledge. In a resilient architecture, all three risks require their own controls and coordinated transitions.

The Attack Surface Spans the Entire Data Chain

A RAG index rarely originates from a single, manually reviewed collection. Crawlers read web pages, connectors synchronize cloud folders, users upload files, and APIs import product data. Added to this are parsers, OCR, language cleaning, chunking, and metadata enrichment. Every stage can ingest incorrect content or detach an originally correct statement from its context.

Typical causes include a compromised source system, a newly linked mirror document, an accidentally published draft file, a misassigned tenant, or a parser update that assigns table values to the wrong headings. Comparing a cryptographic content hash with a trusted reference value can detect discrepancies; however, a matching hash proves neither the truth, topicality, nor approval of the content.

Provenance as an Auditable Data Record

A provenance record should belong to every document and every chunk derived from it. In practice, useful elements include at least a stable source ID, canonical origin URL, responsible owner, retrieval timestamp, document version, content hash, approval status, trust level, parser version, chunking version, embedding model, and index generation. For manual uploads, the role of the uploader and verified license are added. For synchronized systems, it is also important which authenticated connector delivered the file.

The NIST AI 600-1 Generative AI Profile treats content provenance, traceable documentation, as well as testing and evaluation as key building blocks of generative AI risk management. Transferred to RAG systems, this means: It's not just the current index that counts. The traceable relationship between source revision, processing run, and published index generation is also part of operational documentation.

Quarantine Separates Ingestion and Publication

A central architectural building block is strict separation: New or modified content does not immediately become searchable. It first lands in an ingestion zone. There, the pipeline validates origin, file type, size, signature or expected hash, allowed tenant, metadata completeness, and scope of change. Only then are text and chunks generated in a non-production index generation.

Rules should be risk-based. A change on an authenticated, internally owned FAQ page can be approved after automated tests. A new domain, an unusually large text change, an unknown file owner, or a source without a responsible person, on the other hand, trigger quarantine and human review. If required information is missing, 'fail closed' applies: The old, verified generation remains active; the new state is not quietly published.

Approval as an Immutable Index Generation

After review, a production index is not overwritten incrementally. A better approach is a new, versioned generation with a manifest: expected documents, expected chunks, source hashes, transformation versions, and timestamps. Only when the tests pass does an alias or routing configuration switch atomically to this generation. The previous generation remains rollback-ready for a limited, defined period.

The procedure resembles a controlled migration. Our article on changing a RAG embedding model shows why parallel index generations and comparative tests are also useful during technical changes. In the event of suspected poisoning, the security question arises: Which source revision and which derived chunks must be blocked?

Fictional Example Scenario: An Incorrect Return Window Reaches the Support Bot

Suppose a retailer operates a chatbot for product and service inquiries. The knowledge base synchronizes the official help center and a few approved vendor portals every night. Following a link change, a connector follows a redirect to an unapproved mirror page. There, a visually plausible PDF states a return period of 90 instead of 30 days. The file is broken into chunks; several sections land in the index with high semantic similarity.

The next morning, the bot promises the wrong return window for return queries. The language model was not reprogrammed, and users did not enter malicious instructions. Retrieval simply provided an incorrect foundation. Monitoring triggers an alert because a new domain appears as an answer source for the first time and a golden set test for the return period deviates from the expected source evidence.

Controlled Quarantine and Recovery

1. The team halts only the affected ingestion source and freezes the current index generation against further changes.
2. The suspicious document ID, all derived chunk IDs, and their query matches are logged in the incident report.
3. The mirror domain is blocked, and its chunks are moved to quarantine. For questions regarding the return period, the bot temporarily provides a safe notice pointing to human support or the verified policy page.
4. The alias is rolled back to the last provably clean index generation. Other, unaffected knowledge domains remain available.
5. The connector is restricted to the canonical source. The pipeline then builds a new generation from the verified manifest.
6. Only after re-indexing tests and subject-matter approval does this generation go live.

This sequence limits damage without prematurely shutting down the entire chatbot. Crucial is the link between origin data and derivatives: Without mapping documents to chunks, it would be unclear which vectors need to be removed.

Re-Indexing Tests Must Show More Than a Successful Pipeline

A green job status only proves that the process completed technically. It proves neither that old chunks have disappeared nor that correct sources win out for realistic queries. A resilient test suite therefore checks inventory, retrieval, and response behavior.

1. Manifest and Deletion Audit

Compare the new generation against the approved manifest. Every expected document version must be present; blocked document and chunk IDs must not exist. Particularly important are 'tombstones' for deleted or replaced content. Simply appending new embeddings leaves old, poisoned matches active in the index.

2. Retrieval Tests with Expected Sources

For critical questions, an expected answer text is not enough. Additionally define allowed and forbidden source IDs, a minimum hit count, and exclusion rules. For example, the return period must originate from the canonical policy; the quarantined mirror domain must not appear in the top hits or the model context. How to structure such test suites is explained in the article on answer quality with Golden Sets and RAG tests.

3. Negative and Manipulation Tests

In an isolated test environment, teams can ingest a clearly flagged, unapproved test source. The pipeline must hold it in quarantine; production-like search must not retrieve it. In addition, unusual domain switches, missing owners, extreme content differences, and contradictory date specifications are tested. The NIST report AI 100-2 on Adversarial Machine Learning classifies poisoning as an attack category in its taxonomy and emphasizes that countermeasures and their limits must be considered systematically.

4. Pre- and Post-Switch Comparison

Run the same queries against the last clean generation and the new generation. Compare hit sources, ranking, answer citations, no-answer rate, and expert evaluation. A small canary slice can provide additional production signals as long as users gain no access to unverified sources. The production switch occurs only when defined security and quality thresholds are met.

Monitoring: Detecting Anomalies Early

Do not monitor answer ratings alone. Meaningful metrics include new or rare source domains, the proportion of unverified sources in the ingestion stream, unusual document sizes, large hash or text differences, spikes in new chunks from a single owner, shifts in top sources within the Golden Set, and answers lacking verified citations. Metrics should point to provenance IDs rather than unnecessarily stored full user queries.

Content freshness also remains relevant. An old, long-superseded policy is not intentionally poisoned, but it can have the exact same effect. The article on knowledge base freshness and crawl QA complements security controls with cadence, ownership, and deletion paths.

Checklist for Secure RAG Ingestion

• Does every source have a stable ID, canonical origin, responsible owner, and trust class?
• Are hash, document version, parser, chunking, and embedding model logged together?
• Do new or drastically modified sources remain outside production search until verified?
• Do new domains, missing signatures, or implausible content changes trigger quarantine?
• Are approved indexes published as versioned generations with a rollback-capable alias?
• Does re-indexing demonstrably remove replaced chunks instead of merely appending new data?
• Does a Golden Set test answers as well as expected and forbidden sources?
• Is a safe fallback available for topics whose sources are blocked during an incident?
• Are distinct roles assigned for ingestion, business approval, incident response, and republication?
• After every incident, is it documented which control failed and which regression test was added?

Conclusion

RAG data poisoning cannot be solved with a single prompt rule. Protection stems from an auditable supply chain for knowledge: document origin, inspect changes in quarantine, version indexes, safely purge old derivatives, and test retrieval against expected sources. Start with the highest-risk document classes and a small Golden Set. This combination alone makes it visible which source backs an answer—enabling a targeted rollback path before a corrupted state becomes the permanent status quo.

Sources

• OWASP GenAI Security Project: LLM04:2025 Data and Model Poisoning
• NIST AI 600-1: Artificial Intelligence Risk Management Framework – Generative Artificial Intelligence Profile
• NIST AI 100-2: Adversarial Machine Learning – Taxonomy and Terminology of Attacks and Mitigations