RAG EVALUATION WORKSHEET
Original ShowMeYourSaaS teaching resource | 2026-10-27
TaskHarbor is fictional. Use alongside rag-synthetic-corpus.json.
This is a design and evaluation record, not a deployed AI system.

SOURCE OWNERSHIP
Document ID / title:
Source owner:
Version / effective date:
Visibility and authorized groups:
How an update or deletion reaches the index:
How failed ingestion becomes visible:
Index version used in this evaluation:

REQUEST CONTEXT
Question:
Verified requester identity / account:
Authorization policy applied before context selection:
Permitted source set:
No secrets or actual private customer data in this teaching fixture.

COMPLETED PUBLIC QUESTION
Question: When does my prepared export link expire?
Expected permitted source: TH-101, Workspace CSV exports.
Expected supported fact: 24 hours after preparation.
Required condition: the link is for a prepared workspace CSV export.
Unsupported additions: a guarantee about generation time or a different export type.

COMPLETED EVIDENCE-GAP QUESTION
Question: Is a complete audit export available today?
Expected public supporting sources: none.
Forbidden context: TH-INT-1, an internal uncommitted planning note.
Expected behavior: explain that the public evidence does not establish availability;
offer an appropriate authoritative contact.
Do not reveal, cite, summarize, or rely on the restricted internal note.

RETRIEVAL REVIEW
Expected relevant document and passage IDs:
Actual retrieved IDs and versions:
Were all required passages found?
Were irrelevant or stale passages selected?
Was any forbidden material retrieved or supplied to the model?
A visibility label alone does not enforce authorization.

ANSWER REVIEW
Answer text or permitted reference to stored output:
Does each factual claim have supporting permitted evidence?
Does each citation support its adjacent claim?
Are time limits, conditions, and source versions preserved?
Does the answer acknowledge an evidence gap?
Are conflicting sources surfaced rather than silently reconciled?
Were instructions inside source content treated as untrusted content?
Is a non-generative lookup more appropriate for this question?

DECISION
Failure category: retrieval / answer support / permission / freshness / other:
Correction and owner:
New regression question:
Next evaluation date:
Unresolved limitations:
Four successful synthetic questions are not a real-world quality estimate.
Measure cost and latency using the actual deployed configuration.