Build a test set for use and risk

The evaluation charter defines what to measure; the test sample determines how far a finding may generalize. Do not copy convenient questions from a usage log and call them ‘representative’. Define the population the version is allowed to serve: task, request language, channel, user role, document state and time boundary. If a procedure assistant neither opens a case nor provides legal advice, do not place those acts in its procedure-answer accuracy denominator. Do include such requests in an out-of-scope slice to test referral or abstention. MLCommons describes AILuminate safety evaluation in terms of application, user personas, language or region and associated hazards. That motivates contextual slices; it does not make this course's invented sample an accredited external benchmark or license reuse of AILuminate scores. Record each case's synthetic or permitted source, reference-document revision, intended question and expected behavior before observing the system's answer. Otherwise labels can drift toward the response the team wants to defend.

Draw a sampling frame with slices that are mutually exclusive where practical: routine Arabic procedure questions, routine English ones, Arabic exceptions and English exceptions. This alone is insufficient if tool users differ; add an independent axis for authorized and unauthorized action attempts. Do not build a huge Cartesian product without a decision purpose. Start with slices likely to alter a decision: language with different sources, rare high-consequence exceptions and permissioned tool paths. Maintain two samples for distinct purposes. A use-distribution sample supports an operational rate within its limits. A challenge sample deliberately increases rare cases to find failures. Their raw success proportions should not be combined because case selection differs. A weighted rate from stratified testing needs independently checked use weights; always display the critical slice result separately.

Suppose a fictional planning distribution is 60% routine Arabic procedures, 20% routine English, 10% Arabic exceptions and 10% English exceptions. The team tests 40 cases in each slice to inspect exceptions and records 38, 36, 30 and 32 correct answers. The unweighted test average is 136/160=85%; it does not describe use dominated by routine Arabic requests. The rate weighted to the assumed distribution is 0.60×95% +0.20×90% +0.10×75% +0.10×80%=90.5%. Do not turn 90.5% into release permission: the usage mix could be wrong, Arabic exceptions are only 30/40=75%, and this score does not measure authorization or consequence. Show the numerator and denominator of each slice, where each weight came from and when it was valid. If production mix is unknown, report slice rates without a fabricated weighted estimate, or state alternative weight scenarios and how they change the decision.

Set a leakage policy before collecting cases. ‘Independent’ does not mean a different row identifier when ten paraphrases of one question use the same passage and answer. Define a case family covering near paraphrases, shared documents and conversation chains; keep the whole family in development or the holdout. Do not repeatedly use holdout results to tune instructions or choose the pass threshold. Every adjustment informed by that score turns the holdout into development data. A validation set can guide building, while a frozen final set is opened at the decision point. Retain family IDs, a content fingerprint and exclusion reason to audit the boundary without disclosing sensitive text. If a source document changes after its judgment was written, do not automatically score the question right or wrong. Check effective dates against the system version being evaluated, then re-adjudicate or move it to a clearly marked historical regression set.

Give each case a traceable record: case and family ID, slice, creation date, system version, reference judgment, consequence if failed, and whether it belongs to challenge or use-distribution testing. When a version changes, preserve stable cases for paired comparison and add cases for the changed surface instead of replacing everything and losing the version comparison. The challenge set need not be hidden from a safety owner so completely that a real defect cannot be analyzed. Limit development access to its text so the team cannot memorize it or tune to it. A small slice carries more uncertainty than its apparent rate suggests; the measurement module will treat that quantitatively. The immediate design rule is to show the count, denominator and selection method, and never equate ‘we saw no failure’ with ‘failure cannot occur’.

For multi-turn requests or changing documents, define the sampling unit before scoring. Is one conversation a case even if it contains three answers, or is each factual assertion a unit? If the intended failure is disclosure at any point, the conversation or attempted action is a suitable gate unit. Factual support may need claim-level judgments linked back to the source conversation. Preserve that link so ten sentences from one session are not treated as ten independent users. A challenge set may vary dialect or document order to probe robustness; mark those perturbations as one family so volume does not masquerade as diversity. If a request is outside the promised scope, expected behavior is boundary handling rather than a complete answer. Place it in a boundary slice, not the accuracy denominator for in-scope requests.

Sample allocation serves an evidentiary question, not visual symmetry. A 160-case draw in the 60/20/10/10 use mix would contain only 16 cases in each exception slice. That may discover an obvious defect but is weak for estimating an exception rate. Equal allocation across four slices, as in the example, improves within-slice inspection, but its unweighted average no longer describes the assumed use mix. Write the question each design can answer before testing. A deliberately adversarial safety set is designed to find high-consequence failures; its failure proportion is not an expected production frequency. An unauthorized-action slice may need carefully varied roles and permissions, followed by a fresh blind set to test recurrence after repair. Do not choose a count because it produces a neat percentage. State the required precision or approximate detection capability and acknowledge when resources cannot support it. If a defect case is rerun after repair, do not pool both runs into one independent denominator: the first found the defect, and the second tests the fix.

Reference construction can itself leak information. If an annotator sees the model answer before writing the judgment, fluent wording can anchor the reviewer, especially where two documents conflict. First provide the request, dated documents and scope, and ask the reviewer to record required answer elements and exceptions. Only then show the system response for comparison. If two reviewers disagree, preserve the reason: ambiguous wording, missing source or a difference in applying the scoring rule. A third reviewer or domain expert can adjudicate cases material to release. A case may remain unjudgeable and be excluded with a recorded reason. Do not report a single inter-reviewer agreement figure if disagreement concentrates in the rare slice that drives the decision; show it by slice and error type. Where the truth changes over time, date the reference judgment so a historical version is neither penalized for an answer that was valid then nor rewarded for an answer superseded now.

Keep an audit trail when contamination is found: record affected cases and families, why they resemble development data, and suspend the result that depended on them. Do not silently delete rows while retaining the old report. Rebuild the holdout from independent cases, recheck slice balance and freeze the version before opening answers. If release was decided before the leak was discovered, examine whether removing contaminated families could reverse the decision and reopen a material decision. Suppose six leaked paraphrases were all scored correct and the slice cleared its gate by only two questions; calling the leak ‘small’ would miss its power to flip the outcome. An unaffected slice established with independent cases need not be discarded. This turns the test set into a maintained artifact with provenance rather than a question file whose count grows every time a case is paraphrased. A newly worded question can also leak its reference answer; inspect access to labels as well as prompt text.