RAG Evaluation Checklist: What to Test Before a Private Knowledge System Launches

A RAG demonstration can look convincing after five carefully selected questions. A launch decision needs a test set that reflects real work, missing information, permission boundaries, conflicting sources, and operational failure.

The goal is not to prove that the system can answer something. It is to determine when the system retrieves the right evidence, produces an acceptable response, cites that evidence, refuses when necessary, and stays within the user's access rights.

Define the Unit of Success

“Accuracy” is too broad. Evaluate the stages separately.

Retrieval quality

Did the system retrieve the passages needed to answer the question? Record whether the required source appears in the candidate set and whether irrelevant results crowd it out.

Grounded answer quality

Does the answer follow from the retrieved passages? A readable answer is not necessarily a supported answer.

Citation quality

Do citations point to the source that supports the associated statement? Check source identity, page or section, and whether the quoted or highlighted passage is actually relevant.

Refusal quality

When the approved sources do not contain an answer, does the system state that clearly? A well-calibrated “not found” response is part of quality.

Workflow usefulness

Can the intended user act on the result? A technically grounded answer may still be too vague, too long, or missing the format the workflow requires.

Build a Representative Test Set

Start with questions collected from the people who perform the work. Include more than the normal path.

Useful groups include:

• Common questions with one clear source
• Questions that require two approved sources
• Ambiguous wording and internal terminology
• Recently changed policies
• Similar documents with different versions
• Questions whose answer is absent
• Questions outside the system's scope
• Questions from users with different permissions
• Conflicting or outdated source material
• Attempts to make the system ignore its rules

Store the expected evidence and acceptance notes for each question. Do not write only one “perfect” reference answer when several phrasings would be useful.

Evaluate Retrieval Before Generation

If the right passage never reaches the model, prompt changes will not reliably fix the problem.

For a failed example, inspect:

1. 1Was the source successfully ingested?
2. 2Was relevant text extracted?
3. 3Did chunking preserve the section and its context?
4. 4Was metadata correct?
5. 5Did permission filtering remove the source?
6. 6Did search retrieve it?
7. 7Did reranking move it into or out of the final context?

This trace turns “the bot got it wrong” into an engineering diagnosis.

Test Permissions as a Retrieval Property

Do not depend on the model to hide content it should never have received.

Test users from each permission group against:

• Sources they may read
• Sources they may not read
• Questions that hint at restricted information
• Documents that changed ownership
• Revoked users and revoked documents
• Cross-tenant or cross-client identifiers

Record permission decisions in the application and retrieval layers. The prompt is not an access-control boundary.

Test Conflicting and Stale Sources

Private knowledge collections often contain duplicate policies, old proposals, and copied SOPs. The evaluation should establish:

• Which source is authoritative
• How versions and effective dates are represented
• Whether superseded material is excluded or visibly labeled
• What the answer does when two approved sources conflict
• Who owns source cleanup

Sometimes the correct outcome is not a model change. It is a content-governance task.

Test Untrusted Instructions in Documents

A retrieved document may contain text that looks like an instruction to the model. This can be accidental or malicious.

The OWASP LLM Prompt Injection Prevention Cheat Sheet describes direct and indirect prompt injection, including instructions hidden in documents and external content. It recommends defense in depth rather than relying on one filter.

Include adversarial fixtures that try to:

• Override the system's task
• Request secrets or hidden instructions
• Trigger an unauthorized tool
• Insert an external link or exfiltration attempt
• Masquerade as a trusted policy
• Poison future retrieval

Keep untrusted content separated from instructions, restrict tool permissions, validate proposed actions, and require approval for consequential operations.

Measure Operational Behavior

Quality is only one part of launch readiness. Observe:

• End-to-end and stage-level latency
• Model, retrieval, OCR, and infrastructure cost
• Timeout and retry behavior
• Empty retrieval and provider failure
• Logging without prohibited data
• Index freshness and failed ingestion
• Usage by team and question category
• Escalations and unresolved topics

Report distributions and sample sizes where possible. Avoid turning one fast test into a public latency claim.

Assign Review Ownership

Every failed example should have a category and owner:

The evaluation set should evolve when real users expose a new failure pattern. Preserve old examples so improvements do not silently reintroduce previous defects.

Launch Gate

Before a bounded launch, require:

1. 1Versioned sources and test set
2. 2Agreed scoring rubric
3. 3Permission tests
4. 4Unanswerable and adversarial cases
5. 5Safe failure behavior
6. 6Cost and latency observations
7. 7Named source, system, and review owners
8. 8A post-launch review cadence

NIST's AI Resource Center provides testing, evaluation, verification, and validation resources for operationalizing the AI Risk Management Framework. Use such guidance to structure decisions around the actual risks of the workflow rather than treating one benchmark as universal proof.

Review the private RAG service or request an assessment.