From an assessment decision to an evidence argument
10 min read
Lesson objectives
- The learner states a bounded claim about a skill, specifies the evidence it needs, tests rival explanations, and limits the decision before selecting a tool.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
Start with the decision to be made, not with the test format. 'Measure understanding' does not say what anyone will do with the score. 'Decide who needs additional instruction in interpreting learning-progress data' identifies a use that can be evaluated. State the interpretive claim narrowly: given a short dataset, the learner can explain a change in a group rate, identify the denominator, and qualify the comparison. That is narrower than 'can analyze data'. One task cannot license a judgment about every analytic skill. Identify the decision maker and the consequences. A revisable recommendation for instruction differs from a final, high-stakes selection decision. As consequences rise, the case for diverse evidence, quality review, and fairness scrutiny becomes stronger.
Build a five-link chain: claim, observable behavior, task that elicits the behavior, scoring rule, and permitted use of the result. For a claim about interpreting a changing rate, the behavior is more than choosing the larger number. The learner identifies numerator, denominator, and period, then asks whether the groups are comparable. A task can present two periods with changing sample sizes. A response shows reasoning when the learner explains why two raw counts should not be compared as rates. Give distinct rubric credit for identifying denominators, calculating rates, and qualifying an interpretation. If a denominator error invalidates a later inference, specify how the scoring rule handles that dependence rather than merely adding points.
Map rival explanations of a score before relying on it. A learner may fail a mathematically sound task because its language is ambiguous or the table cannot be accessed, rather than because the target concept is missing. A learner may succeed by guessing or with unrecorded assistance. These explanations cannot be separated by a single score. Review the wording with intended users; record time, display mode, and permitted supports; then inspect actual or documented pilot responses. A group difference alone does not establish tool bias, while equal group averages alone do not establish fairness. Seek evidence about access and equivalence of the measured construct. Keep subgroup sizes and missing responses visible.
Do not jump from an observed response to a justified claim. An assessment argument needs a warrant explaining why the observed behavior represents the target skill. If learners calculate a rate from a ready-made table, success may support their ability to perform that operation in this format. It does not establish their ability to select suitable data from conflicting sources. If the intended skill is to interpret learning results, a task rewarding only the numeric answer misses reasoning about the unit of measurement and missing data. Write a testable warrant: 'If a participant identifies the correct denominator, calculates the rate, and explains the effect of changing sample size, the response provides bounded evidence of interpreting a similar table in a training setting.' This states the task, components of the response, and the context to which the inference may cautiously extend. Then ask what guessing would look like, what even a correct response would fail to show, and what additional evidence would be needed if the use changed from instructional support to promotion. Multiplying weak items does not repair a missing warrant. Coverage begins with task design rather than a larger pile of points.
Distinguish underrepresentation of a skill from an irrelevant influence on performance. In underrepresentation, the skill is broad but the tasks evoke only a narrow part of it: claiming to measure 'interpretation of progress data' from one rate calculation without testing comparison or communication. In the other case, performance depends on something outside the intended skill: ambiguous wording, a chart inaccessible to a screen reader, or a time limit that turns an interpretation task into a typing-speed test. Do not assume that every accommodation changes the measured construct or that identical presentation is automatically fair. Define the construct first. If reading prose is essential, text-to-speech may alter the interpretation. If calculation and interpretation are the targets and prose merely delivers the data, it may remove an irrelevant barrier. In either case, examine comparability after the change rather than declaring it. Record what support was available. A professional decision identifies the barrier, target skill, proposed remedy, and remaining uncertainty instead of relying on the label 'equity'.
Separate three numerical objects before writing the report. A raw count answers how many responses were correct. A rate answers what share of a specified denominator was correct. A percentage-point gap answers how far two rates differ. With 18 of 30 and 14 of 20 correct, group A has more successes but group B has the higher rate: 60% and 70%, a descriptive gap of ten percentage points. Do not write that B 'improved by 10%': B has no before-and-after comparison here. Relative difference using 60 as a base would be about 16.7%, a different calculation answering a different question. Nor should you average 60% and 70% to estimate the rate among all participants: the pooled rate is 32/50=64% because group sizes differ. The unweighted 65% is the average of two equally weighted group rates. It could answer a different question if those weights were declared. Arithmetic can be correct while the associated decision is wrong because its unit and intended population were left unstated.
Close the evidence argument with a conditional use rule rather than a categorical label. For a low-stakes example, if an individual completes two different rate-interpretation tasks and names denominators and comparison limits on both under a published rubric, offer a targeted correction activity or optional enrichment. If responses conflict or one task cannot be accessed, do not turn an automated average into a description of the person. Review the item and performance conditions, then collect another sample of evidence. Specify what must be retained: task version, scoring rule, administration date, assistance, and rationale for the decision. Avoid collecting unrelated personal data merely because it is available, and do not project a group judgment onto an individual. In the report, separate 'observed', 'inferred', and 'action'. A colleague can then locate a disagreement: perhaps both agree on the arithmetic but differ on the inference or the cost of the proposed action. A sound argument supports such disciplined scrutiny and names the further evidence that could change the decision.
Inspect the evidence sample before inspecting the average. An example with 40 responses in one week and 30 in the next may or may not include the same people. If the tasks differ in difficulty, 75% versus 80% does not measure a net learning change even if participants are the same. If ten people are absent in the second period, absence might be related to performance; do not silently assume they resemble respondents. Record the unit of observation, eligible population, number responding, selection mechanism, task version, and missing values in an interpretation card. Then choose a sentence the data permit: 'The observed rate was 80% among 30 responses in the second period,' rather than 'All teachers improved.' No elaborate statistical model is needed to recognize that a changed denominator or task limits the claim. Design a follow-up with visible sample, version, and timing so a changed result can be interpreted instead of becoming a persuasive but uninformative number.
Test the decision with a countercase that changes one concrete fact. For 18 of 30 in A and 14 of 20 in B, the descriptive comparison is 60% versus 70%. If a documented review finds that three A answers were wrongly marked incorrect while group size stays fixed, A becomes 21/30=70% and the descriptive rate gap disappears. That does not prove fairness or equality of every learner. It shows that a decision may be sensitive to the recording of three responses. Conversely, if A's denominator grew to 33 while its 18 successes were unchanged, its rate would fall to about 54.5%; that would not demonstrate decline among the original people without knowing who joined. Use this sensitivity to specify checks before action: confirm entry accuracy, sample definition, and item version. Set a procedural review trigger, such as rechecking group records whenever correction of three entries would change a support decision, instead of turning the trigger into an automatic judgment about a learner. A defensible argument exposes what could change the decision rather than hiding behind one figure.
Lesson summary
The learner states a bounded claim about a skill, specifies the evidence it needs, tests rival explanations, and limits the decision before selecting a tool.