Problem framing and theory of change · Lesson 4

Test evaluability and issue a go, redesign or defer recommendation

9 min read

Lesson objectives

  • Test evaluability and issue a go, redesign or defer recommendation
Reading tools

At the end of framing, a team may feel pressure to launch an “impact evaluation” at once. Naming the task does not make it evaluable. Evaluability asks whether the intervention, population and outcomes are specified sufficiently, whether data and a comparison exist or can be built to answer the question, and whether implementation, ethics and decision timing permit the work. An early assessment does not aim to stop learning. It identifies what can be learned now and which repair would support a stronger question later. UNICEF’s evaluability guidance connects this review to improving program logic and monitoring, but is written for its own setting. We can use the principle without presenting internal UNICEF procedures as general law.

Use four gates. First, program clarity: who is eligible, what is offered, and when does exposure end? Second, measurability: does the outcome have a denominator and time, and can it be collected comparably across relevant groups? Third, identification: is there a group-entry mechanism or another design that supports a causal interpretation under testable assumptions? Fourth, protection and use: is data collection proportionate to sensitivity, and will a decision maker use the answer? Do not mark an overall “yes” when a critical gate fails. Name the defect, the question it affects, who can repair it, and when.

A defect can be repairable before measurement or blocking for a particular question. Duplicate attendance names may be cleaned while preserving an audit trail. If nonparticipants were not asked the outcome at the same time, a prospective follow-up may be designed; missing past observations cannot be created retroactively. If motivation to participate is unmeasured, adding age after the endline does not automatically remove selection. Say “we can describe change or delivery” when that is supported; do not claim that nothing is known simply because an effect is unidentified. Evaluability governs the claim the evidence earns, not the worth of the people who collected it.

The initial decision is to withhold an effect percentage from respondent means. Group entry, roster completeness and response patterns may all change the comparison. A conditional descriptive report is feasible: count documented invitations after verifying, rather than guessing, duplicates; trace registration and attendance; and show confidence differences among respondents with explicit limits. During the four weeks, inspect invitation dates, define the person and invitation unit, assess whether follow-up with nonrespondents is appropriate, and choose an outcome relevant to expansion. If past data cannot be repaired, design a future round with eligibility, invitation, baseline and consistent follow-up records, then select a comparison feasible both ethically and operationally. Deferring an effect estimate need not defer service improvements.

An evaluability memo can carry three separate judgments: **go** on a question supportable now, **redesign** where a specified change can close a gap before new data collection, and **defer** a claim blocked by comparison or protection. These judgments can coexist. The team may evaluate delivery reach now, redesign outcome tracking, and defer employment impact. Do not stamp the entire initiative “unevaluable” when some questions can be answered. Conversely, success in a delivery report does not allow its measures to be renamed “impact.”

Make recommendations executable with a “defect → repair → owner → date → verification” table. For example: “Possible duplicate invitation IDs → reconcile selected records with their original source without automatic deletion → data lead → before freezing the analysis file → change log and unique-person count.” Another row might read: “Confidence outcome missing for nonrespondents → consistent, voluntary follow-up plan → field lead → before the next round → a tested form on people from each path.” For a comparison gap, specify a design decision rather than writing the word “statistics.” If there is no acceptable option, that is an explicit reason to narrow the claim.

Consider the cost of requesting evidence that will not change a decision. Long interviews with young people who lost work may reveal sensitive circumstances without fixing a comparison defect. Add them only for a defined question, appropriate protection and clear use. Separate identifiers from the analysis file, minimize contact fields, and specify who may see raw responses and for how long under the organization’s applicable policy. Do not invent a universal legal retention period. Check that inviting nonregistrants to follow-up cannot imply service depends on answering. Ethics review is not a last box; it may alter the design or make collection of a particular item unjustified.

An evaluability table distinguishes the presence of data from its fitness. A column named “employment result” may exist while half its cells are blank and mentors use different definitions of employment. Do not mark the question ready because a column exists. Check the data dictionary, provenance, collection time and missingness pattern. Ask whether the defect is distributed evenly or concentrated in a group whose results could change the decision. More rows of completers cannot fix the absence of outcomes for noncompleters. An audit of a few original records may reveal that an outcome date precedes the first session, a time-order problem that a new average cannot solve.

A manager may ask for a quick trial in one center before investing in a full measurement system. That can be useful if labeled a feasibility and process test. Can the team record invitation, consent and follow-up consistently? How long does this take? Why is contact lost? Successful process testing must not be recast as an impact estimate. Specify a transition rule: if eligibility is defined and follow-up reaches the needed groups without unacceptable burden, proceed to a planned comparison; otherwise revise the question or collection method. Each data investment is thus linked to a later decision that can be checked.

At the end of framing, someone who missed the meetings should be able to read three things: the question answerable now, the question contingent on a change, and the question the current evidence cannot support. Put beside each judgment the failed or passed gate and a concrete observation that would change it. That is more informative than “the program is 70% evaluation-ready.” A single collapsed score hides the possibility that a missing comparison blocks the impact question despite tidy documents elsewhere. Quality is not the count of checked boxes; it is the match between question, evidence and decision.

Lesson summary

Apply test evaluability and issue a go, redesign or defer recommendation to a hypothetical case and justify a bounded decision.