A useful RAG evaluation suite separates retrieval quality from generation quality. On the retrieval side, context precision asks how much of what you retrieved was actually relevant, and context recall asks whether the passages needed to answer the question were retrieved at all. On the generation side, faithfulness asks whether each claim is supported by the retrieved context, and answer relevance asks whether the response addresses the question. Diagnosing with these four separately matters, because a low faithfulness score usually means a retrieval problem, not a prompting problem.
The plateau shows up like this: context recall is high, faithfulness is high, and support tickets still say the assistant gave the wrong policy. That combination is the signature of a corpus that contradicts itself. Retrieval found relevant passages, generation stayed faithful to them, and the passages disagreed. Every metric in the standard set is computed per query against the context it retrieved — so none of them can see the second, contradicting document.
What to add: corpus-level metrics. Conflict rate per topic, staleness distribution, and coverage of authoritative sources. These are measured on the knowledge base, independent of any query, and they explain the ceiling your per-query metrics keep hitting.
A metric stack that actually diagnoses
1. Instrument retrieval and generation separately
Track context precision and context recall on a fixed evaluation set alongside faithfulness and answer relevance. When quality drops, the split tells you immediately whether to tune chunking, embeddings, and reranking, or the prompt and the model. Aggregated single-score dashboards hide exactly the information you need.
2. Require citations so faithfulness is auditable
Compute faithfulness at claim level with a judge that must cite the supporting passage. This makes the metric debuggable — you can read the passage it relied on — and it exposes the case where a claim was scored faithful against a document that should never have been authoritative in the first place.
3. Measure the knowledge base, not only the answers
Add conflict rate (how many topics have contradictory sources), freshness (how much of the retrievable corpus is stale), and authority coverage (how many high-traffic questions map to a validated source of truth). These metrics move slowly, predict incidents, and are the only ones that improve when you fix documentation rather than code.
The metrics you can't compute per query
Conflict rate, staleness, and authority coverage require scanning the whole corpus and comparing documents against each other. That's the layer Alignode operates on.
- Quantify contradictions and stale content across Confluence, Notion, SharePoint, Slack, and Drive.
- Track conflict rate over time so knowledge quality becomes a monitored metric, not an assumption.
- Route each detected conflict to the owning team so the corpus improves instead of drifting.
Frequently asked questions
What are the core RAG evaluation metrics?
Four metrics cover most needs: context precision (how much retrieved content was relevant), context recall (whether the necessary passages were retrieved), faithfulness (whether each generated claim is supported by the retrieved context), and answer relevance (whether the response addresses the question). Tracking them separately is what makes them diagnostic.
How do you measure faithfulness in RAG?
Decompose the answer into atomic claims and have a judge model label each one supported, contradicted, or unsupported against the retrieved context, with a required citation for the supporting passage. Faithfulness is then the share of claims that are supported, and the citations make every score auditable.
What is the difference between context precision and context recall?
Context precision measures how much of what you retrieved was actually useful — it drops when the retriever returns noise. Context recall measures whether the passages needed to answer the question were retrieved at all — it drops when chunking, embeddings, or top-k settings cause misses. Precision problems and recall problems require opposite fixes.
Why do our RAG metrics look good while users still get wrong answers?
Because every standard metric is computed per query against the context that query retrieved. If two documents in your corpus contradict each other, retrieval can return either one, generation stays faithful to whichever it got, and all four metrics pass — while half of users get the wrong policy. Detecting that requires corpus-level conflict measurement.
How often should RAG evaluation run?
Run per-query metrics on every release against a fixed evaluation set, so regressions in chunking, embeddings, or prompts are caught immediately. Run corpus-level metrics continuously, because knowledge drifts on its own schedule: documents get edited, policies change, and duplicates appear without anyone touching the pipeline.
Related guides
Raise the ceiling your metrics keep hitting.
See how Alignode measures and reduces conflict in your knowledge base so downstream RAG scores can actually improve.