Set evaluation criteria, stakeholders and distributional questions
8 min read
Lesson objectives
- Set evaluation criteria, stakeholders and distributional questions
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
After the team has framed a causal question and mapped the change pathway, the sponsor asks, “Is the program good?” There is no single arithmetic value of “good.” Participants may find mentoring respectful and useful, a manager may see high cost per person reached, and a partner may observe that people without internet access rarely enroll. An evaluator translates these perspectives into explicit questions and judgment criteria. The loudest or best-funded voice should not quietly become the definition of impact.
The OECD DAC evaluation criteria provide useful lenses: relevance, coherence, effectiveness, efficiency, impact and sustainability. They are to be applied thoughtfully for the evaluation’s purpose and context, not as a compulsory six-part checklist. In the mentoring case, “Does the service address the intended young people’s needs?” concerns relevance. “Does it complement or duplicate existing job-search support?” concerns coherence. “Did it reach its target group and achieve near outcomes?” concerns effectiveness. “What resources are required per documented result?” concerns efficiency. “What change can be attributed to the initiative under a defensible design?” concerns impact. “Does résumé skill persist after sessions end?” concerns sustainability. Each question may require a different evidence source and design.
Do not turn impact into a synonym for any improvement. More attendees indicate reach. Better résumés may be a near outcome. Estimating an initiative’s impact requires reasoning about what would have happened without it under a defined question and comparison. A team may reasonably judge a program relevant while lacking a causal impact study, if it states what is known and unknown. That judgment gives no license to claim improved employment. Criteria organize inquiry; they do not override evidence limits.
Map stakeholders by their relationship to the decision, not just by seniority. A sponsor allocates funds; delivery staff know referral constraints; participants bear travel time and disclose information; eligible people who did not enroll may know access barriers; a partner organization may receive difficult referrals. For each group ask: Which decision affects them? What distinct knowledge do they hold? What risk does asking them for data create? Listening to all groups is not a ritual of collecting signatures. People least able to attend an evening interview may hold the most important explanation for exclusion by the current design.
Construct a four-column evaluation matrix: criterion, specific question, required evidence, and decision use. For relevance, ask whether far residents have a viable way to attend; short interviews with nonregistrants may help, with their selection limits recorded. For delivery effectiveness, ask what share of eligible invitees at each distance completed, using a single defined invitation and attendance register. For efficiency, ask the cost per completer under each delivery model, using a consistent cost boundary that includes travel support if the initiative pays it. For impact, ask whether the offer changed job-search outcomes; that needs a comparison design the attendance table does not provide. The impact question can be deferred if repairing reach is the immediate decision.
Equity is not an appendix added after the average is reported. Mean outcomes can differ because an initiative serves people who face harder circumstances; an overall mean can also look favorable because those unable to attend disappear from follow-up data. Choose a disaggregation with causal and operational relevance, such as distance here, before seeing the final table. Avoid a profusion of tiny subgroups that creates false precision. When rates differ, show counts and denominators, then ask what would establish a reason and what change could be tested safely. Do not hide the gap in the average or infer its cause from the gap alone.
Trade-offs become real when the budget cannot answer every question. Suppose the team can fund twenty interviews or audit the full attendance roster and add distance to it. Interviews may surface barriers but cannot estimate the prevalence of each barrier in the eligible population if purposively selected. A complete roster describes the gap better but does not explain it. A defensible sequence may be to audit the roster first, because it anchors denominators, then interview far nonregistrants and near dropouts. Record why this order was chosen and what will remain unknown. If the sponsor asks about employment impact, this formative work does not replace comparison design; it prepares for it.
Write a decision rule before the next result arrives. For example: “If the near–far completion gap remains after the roster audit, pilot a closer venue for a specified period and track both completion and access cost.” That is a test rule, not a promise that a venue will close the gap. If purposively selected interviews suggest travel cost is a major barrier, do not assign their frequency to all far residents. Seek broader confirmation or test bounded travel support and observe the response. Weigh participant travel burden as well as the initiative’s own expense. Who bears a cost matters for both efficiency and fairness.
Some questions cross criteria. Low participation among far residents may reflect poor fit of the service, weak operational reach, and an efficiency trade-off if relocation uses more resources. Do not count the same observation as three impacts. Cite it once and show how the judgment changes under the three lenses. If stakeholders disagree, record the rule for trading outcomes off. An organization may choose to serve the least-reached group despite a higher cost per completer, within a known budget limit. That is an explicit value choice, not a statistical conclusion extracted from the attendance table.
Lesson summary
Apply set evaluation criteria, stakeholders and distributional questions to a hypothetical case and justify a bounded decision.