Audit evidence quality and disaggregate a signal
6 min read
Lesson objectives
- Produce a reviewable output for audit evidence quality and disaggregate a signal in an educational case with checks and authority limits.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
A dashboard tells a specialist that the school's average reading score rose by five points. The news sounds encouraging, yet the average cannot tell us whether every pupil progressed or whether this term's measured group resembles last term's. Good evidence is not simply a larger pile of numbers. It is a measure whose construction and question are clear. Before proposing action, audit four features: the indicator definition, the measured population, the collection method, and comparability over time. Then read the quantitative signal alongside work samples or task observations so that a number does not become its own explanation.
Start with the definition. If “proficient” means 70% on the first test and 80% on the second, the standard changed even if learning did not. A raw reading score assembled from items of different difficulty does not automatically represent growth. Write the indicator as “number meeting a stated standard divided by the number validly assessed,” then record the standard, date, and exclusions. Do not let absentees silently disappear from the denominator: their absence may relate to the difficulty under study. When data are incomplete, say the signal is provisional and request a targeted check instead of claiming precision.
Disaggregation means examining groups that matter educationally instead of relying on an overall average. Consider 20 of 40 pupils meeting a standard at the start of term and 36 of 60 at the end. The proportion increased from 50% to 60%, but 20 new pupils entered the measured group. We cannot call the ten percentage points growth by the same pupils. Suppose 24 of the original 40 met the standard at the end. That group's later proportion is 24÷40=60%, while 12÷20=60% of newcomers met it. The equal later proportions do not make their histories identical. Show stable and incoming groups separately, with a source for each count, before telling a growth story.
An average can conceal a contrasting subgroup pattern. Suppose 75% pass overall, while 90% of 40 pupils who attended every support session pass and 60% of 40 nonattenders pass. It is tempting to call the 30-point difference the program's effect. Yet attendance may reflect prior motivation, access, or greater need. A subgroup table helps frame an inquiry but does not remove selection bias. Keep “the groups differ” separate from “the intervention caused the difference.” To test the latter, examine a comparable baseline, how participants were selected, implementation records, and work samples alongside the test.
Once definitions and denominators are sound, check collection reliability. Did teachers use a common marking rubric? Were test conditions similar? Did results come from an authoritative system or a manually copied sheet? Trace three sampled records from source to report without opening more identifiable information than the task requires. If one cell was copied incorrectly, fix it and examine whether the transfer method itself can repeat the error. That is a safeguard for decisions about pupils' and teachers' time, not an accusation against colleagues.
Use an evidence matrix with a row per claim and columns for source, definition, denominator, exclusions, collection reliability, what the evidence supports, what it cannot support, and next check. The limitation column needs content. “Two class averages improved” does not establish that every pupil benefited or that a new textbook caused the change. The US Institute of Education Sciences frames data use as an ongoing cycle of instructional improvement rather than a verdict from one snapshot. The audit can therefore conclude in one of three ways: sufficient evidence for a bounded action, evidence requiring a targeted check, or evidence unsuitable for the current comparison. Each conclusion calls for a different next step.
When briefing the school leader, use two separate sentences: “What we know now” and “What remains unknown.” Match the proposed decision to confidence in the evidence. Comparable items and marking may support a small instructional trial. A changed standard or one-third of pupils missing calls for repair of measurement first. A persistent difference affecting a particular group calls for fair support without labeling the pupils or making a sweeping judgment about the group. The ability to set limits on an inference is part of educational leadership, as much as choosing an intervention is.
Lesson summary
Audit evidence quality and disaggregate a signal with a worked case and reviewable output.