Drift enters through several doors. Documents get edited while the vector index still holds the old chunks. A new authoritative page is published while three superseded versions remain retrievable. Query patterns shift as adoption grows, pulling in corners of the corpus nobody evaluated. Each of these degrades answers without touching a single line of your pipeline, which is why launch-time benchmarks give false confidence that expires quietly over the following quarter.
The reason drift is hard to catch is that it doesn't look like an error. Retrieval returns a relevant, well-formed passage; generation stays faithful to it; the answer reads perfectly. Only someone who knows the current policy can tell it's out of date. By the time that person notices, the assistant has repeated the stale answer hundreds of times — which is why drift has to be monitored on the corpus, continuously, rather than discovered from support tickets.
The signal to watch: not answer quality, but corpus divergence. When two retrievable documents answer the same question differently, drift has already happened — regardless of what your evaluation set says this week.
How to monitor retrieval drift
1. Track index freshness against source-of-record edits
Every retrievable chunk should have a traceable last-synced timestamp and a link to its source document version. Alert when a source is edited but its embeddings are not re-generated. Unsynced chunks are the most common and most fixable form of drift, and they are invisible to answer-level evaluation.
2. Watch conflict rate as a time series
Continuously compare passages that answer the same question. A rising count of contradictory pairs per topic is the earliest reliable warning that your knowledge base is diverging — usually weeks before it shows up in user complaints or in per-query metric dashboards.
3. Monitor query coverage as usage shifts
Log which topics real users ask about and compare against what your evaluation set covers. As adoption spreads to new teams, traffic moves into parts of the corpus that were never validated. Expanding the evaluation set to follow live query distribution keeps your metrics honest.
Monitoring the knowledge, not just the pipeline
Retrieval drift originates in documents, so it has to be monitored where documents live. Alignode watches the knowledge layer continuously and turns divergence into actionable work.
- Continuously scan Confluence, Notion, SharePoint, Slack, and Drive for newly introduced contradictions and stale duplicates.
- Surface conflict rate and staleness as tracked metrics so drift is visible before users report it.
- Reconcile superseded content against a validated source of truth and route fixes to the owning team.
Frequently asked questions
What is retrieval drift?
Retrieval drift is the gradual degradation of RAG answer quality caused by changes in the knowledge base rather than in the pipeline. Sources get edited, superseded pages stay retrievable, duplicates accumulate, and query patterns shift — so retrieval starts returning content that no longer reflects current truth even though nothing in the system changed.
How do you detect drift in a RAG system?
Monitor the corpus rather than only the answers: track index freshness against source-document edits, track the number of contradictory passage pairs per topic over time, and compare live query topics against your evaluation coverage. Rising conflict counts and unsynced chunks are the earliest reliable indicators.
Is embedding drift the same as retrieval drift?
Embedding drift is one cause of it. If documents are updated but their embeddings are not regenerated, the index holds stale vectors and retrieval returns outdated chunks. Retrieval drift is broader — it also covers superseded documents that remain published, accumulating duplicates, and shifting query distribution.
How often should we re-index our vector database?
Re-index on source change rather than on a fixed schedule wherever possible: trigger re-embedding when the source document is edited, and reconcile deletions so removed content stops being retrievable. Where event-driven sync isn't available, the cadence should match how quickly the underlying policies actually change.
Can observability tools catch retrieval drift?
They catch its symptoms — degraded faithfulness scores, more thumbs-down, higher escalation rates — once answers are already wrong. They don't see the corpus, so they can't tell you that a new policy page now contradicts three older ones. Catching drift before it reaches users requires monitoring at the knowledge layer.
Related guides
Stop accuracy decay before your users report it.
See how Alignode monitors your knowledge base for drift, contradictions, and stale content — continuously.