Adjudicate reference answers and reviewer disagreement

A test cannot establish evidence if the ‘correct answer’ is unknown or changes with time. For a retrieval system, write the reference judgment before seeing the system response: the request, permitted context, supporting document, its effective date, required answer elements and any exception that changes the result. ‘The response looks good’ is not a rubric. Fluent wording may cite an old edition, or give a decision without its decisive condition. ACL research on retrieval evaluation distinguishes answer correctness from grounding in relevant passages; we use that methodological distinction without transferring any study score to this fictional service. Name the unit being judged, whether an individual assertion, whole answer or downstream action. When no source resolves the question, the reference behavior may be an explained abstention and referral rather than a plausible invention from a nearby passage.

Use a compact rubric with behavioral anchors. For this teaching example, score 2 when the answer supplies the outcome and decisive condition and uses a document applicable to this case and date. Score 1 for partly correct information missing a nonmisleading element that still prevents the user's decision. Score 0 for a consequential wrong answer, inapplicable support or a claim that an action occurred when it did not. Record disclosure or unauthorized action on a separate critical flag; do not reduce it to one lost point. This scale is not a universal AI standard and must be tailored to consequences. Provide boundary examples: a short answer containing the condition can earn 2 despite plain wording; a long answer mentioning the wrong exception does not earn 2 because of its length. A citation's presence is not proof of support: its passage may be irrelevant. Conversely, a factually right answer without a visible citation may still fail a product promise to show its source.

For two reviewers, mask the system version and promotional outcome where feasible. Give them the request, dated reference material and use boundary, and require independent scores with case-specific reasons. Do not ask for an impression before defining the grades. Then inspect disagreement. Reviewers may agree because both relied on an obsolete document; agreement measures consistency, not reference validity. They may disagree because the request never states whether an exception applies, making the test item incomplete rather than one reviewer wrong. Classify each disagreement as interpretation of a condition, document currency, request ambiguity or application of the scale. Escalate cases that affect a release gate or rare slice to a qualified third adjudicator with the evidence packet, and preserve the earlier judgments and reason for revision. Mark a case that cannot be resolved from available sources as temporarily unjudgeable and remove it from the accuracy denominator, while reporting how many were removed: selective deletion can inflate a score.

Distinguish three rates before reporting. Reference coverage is cases that can be adjudicated divided by candidates. Initial reviewer agreement is agreement among adjudicable cases. System success is scored only after disagreements are resolved. For example, out of 50 candidate cases, five lack sufficient documentation. Reporting 40/45 system answers correct without showing reference coverage of 45/50=90% conceals a blind spot. If reviewers initially agree on 36 of 45, their 80% agreement is not the system's accuracy. Adjudicate the other nine and then recalculate system success on the 45. If disagreement clusters in an exception slice of ten cases, a high easy-case agreement figure must not erase that problem. A chance-adjusted agreement statistic can be useful, but no agreement statistic creates truth when both reviewers use a bad document. Investigate causes and scope before choosing a more flattering number.

Preserve the adjudication trace as part of the test asset: case ID, document revision, two reviewers' scores and reasons, final adjudicator decision, judgeability state and linked claim. If a rubric is corrected after seeing system scores, version the new rule and apply it to every affected case, not only failures. Calibrate reviewers on training cases outside the holdout, then keep holdout keys from development staff. If documents change later, retain the historical judgment for the older system and create a new one for current use. An update should not silently rewrite history. Also ensure authorized reviewers can reach the reference documents and that excerpts do not expose personal data irrelevant to evaluation.

Turn adjudicated failures into an interpretable error record for release decisions. Score 0 alone does not tell the team whether retrieval failed to find D2, the model received D2 but ignored its date, or the interface ranked D1 above D2. All three can produce the same wrong text yet need different repairs and regression tests. Preserve the passage that should have been used and the error's location in the system path without assuming that the answer alone reveals the cause. Retrieval traces, ranking logs and instruction versions may be needed. If these traces are absent, mark the cause unknown and test hypotheses rather than declaring that the model ‘hallucinated’. When the reference response is abstention, also inspect whether the system detected missing support or simply refuses every question. Broad refusal can conceal a service-coverage failure.

Test the rubric's construct validity before scaling annotation. Give reviewers two near-identical cases differing in one decisive fact: a day-10 question before D2 takes effect and a day-12 question after it. If they assign the same judgment only because the wording resembles, the date rule is unclear or unused. Require each score explanation to name the governing document, its date and the condition that changes the outcome; ‘the source is official’ is not enough. Include an answer with the right numeric period but a citation to a passage that does not support it. It may pass factual value and fail grounding. If the product promises visible trustworthy evidence, that second failure matters even though the number happens to match. Preserve factual correctness and grounding as separate axes before deriving a composite decision, so the team can tell whether retrieval or answer composition needs repair.

Set a rule for cases that lack information needed for a factual judgment. Do not penalize a system merely for requesting a missing completion date when that date is decisive. Do not reward a categorical answer that accidentally matches an unknown truth either. Separate appropriate behavior under missing information from factual outcome correctness. The first can be judged from the request; the second remains unknown until the facts are supplied. Report the number and share of such cases among candidates and why information is missing. If missingness concentrates in one slice, it may reveal a sample or interface design problem, not grounds to delete that slice. If the assistant asks a clarifying question and answers in the next turn, treat the sequence as one path when assessing referral behavior and retain the link between its answer claims. A three-turn conversation is not three independent chances to pass.

When reviewing results with the decision owner, ask what changes if five cases move from score 2 to score 0 after a reference judgment update. If the slice threshold lies between the old and new result, suspend approval and recalculate on the new judgment version; a textual note is insufficient. If the aggregate decision is unchanged but the five failures concern a high-impact action, separate containment may still be required. This is sensitivity analysis for a defined decision, not grade polishing. Record the five case IDs, the reason for judgment change, the approving adjudicator, and old and new outcomes. If an automated judge assists scoring, compare its decisions against a human-adjudicated sample rich in difficult cases. Easy-case agreement alone cannot validate it, and its output is not a final truth source without known failure modes. A written, challengeable rubric remains the evaluation basis rather than the scoring tool's name. Preserve each original system response; rewriting it before rescoring would silently test a different version and obscure the evidence change.