rag-integrity-guard

The problem

Standard RAG trusts whatever the vector store hands back at query time, no questions asked. If that store gets tampered with, through a compromised ingestion pipeline, a poisoned document, or a direct database edit, the LLM has no way to tell real context from an attacker's substitute. It just answers confidently from whatever it was given.

How it works

• Every chunk gets a dual signature at ingestion: a cryptographic hash (BLAKE3/SHA-256) of the normalized text for exact-match tamper detection, plus an L2-normalized semantic embedding for similarity comparison.
• IntegrityGuardRetriever wraps any LangChain BaseRetriever as a drop-in and re-verifies every chunk it returns. A hash match is VALID, full stop. A hash miss where the embedding is still highly similar gets flagged SEMANTIC_DRIFT: the content changed just enough to stay plausible, which is exactly what a disguised poisoning attempt looks like. A hash miss where the embedding has diverged too is the blunter HASH_MISMATCH. Only VALID chunks are ever treated as trustworthy.
• A configurable StrictnessMode decides what happens next. STRICT blocks the whole query, FILTER quietly drops the offending chunk, and LOG_ONLY returns everything annotated for audit.
• A zero-dependency HashingEmbedder ships by default: deterministic hashed n-grams, no API key, no downloaded weights, runs anywhere at zero cost. A real OpenAIEmbedder swaps in when genuine semantic comparison is worth the cost.

Benchmarked, not asserted

I ran it against 500 real (claim, gold-evidence) pairs from the FEVER dataset, split across a benign baseline and three attack types: exact mutation, prompt-injection payload insertion, and wholesale semantic-drift replacement.

Overall: 100% true-positive rate, 0% false-positive rate, at a mean verification overhead of 2.73ms (7.07ms p95) using the zero-cost default embedder. That's fast enough to sit in the retrieval path by default, not as an opt-in toggle you'd disable the moment load got heavy.

One caveat worth stating plainly: the 100/0 numbers aren't a probabilistic score, they're a property of hash verification. Any byte-level change fails the hash, by construction, so that part was never really in doubt. The interesting number is the HASH_MISMATCH vs. SEMANTIC_DRIFT split. A threshold-sensitivity sweep shows the zero-cost HashingEmbedder struggles specifically with the sneakiest attack, a wholesale topical swap, because it's reading lexical n-gram overlap rather than meaning. A real semantic embedder would probably make that number worse before it gets better, and that's the part that actually matters for a poisoning defense to get right.

What it looks like

A live dashboard ships alongside the library. Ingest sample facts, run a query, then launch an attack on an ingested chunk and watch the guard classify it in real time.

Engineering, not just a notebook

It's a proper monorepo: the core Python package (hasher, an embedder protocol matching LangChain's own Embeddings interface, the fingerprint store, the LangChain retriever wrapper), a FastAPI backend deployed as a Vercel Python serverless function, and a Next.js dashboard. Those last two run as two independent Vercel projects on purpose, after the one-project version silently lost Next.js framework detection. 89 tests, 96%+ coverage, ruff and mypy in strict mode enforced in CI on every push, across Python 3.11 and 3.12.