Problem framing and theory of change · Lesson 1

Separate a decision question, descriptive question and causal question

8 min read

Lesson objectives

  • Separate a decision question, descriptive question and causal question
Reading tools

An evaluation team receives a brief: “Two hundred forty young people attended mentoring and their confidence in job search rose. Demonstrate the program’s impact.” The first move is to name the decision. Is the sponsor deciding whether to continue delivery, expand to more sites, or repair referral and attendance? Each decision asks for different evidence. Session completion and absence can inform delivery. Whether mentoring caused a change in confidence is a causal question. Post-program counts alone cannot answer it.

A descriptive question asks what happened among those observed: “What share completed four sessions?” A causal question compares the outcome under mentoring with what would have happened to the same target population during the same period without mentoring. We cannot observe both outcomes for the same person at the same time, so we construct a comparison under explicit assumptions. An evaluative question adds a judgment: “Is a possible gain worth the cost and participant burden?” Delivery may improve without demonstrated impact; a credible small effect may still leave a genuine value decision. Keeping these questions apart prevents a delivery measure from being used as an effect estimate.

Define the decision unit before selecting a method. It may be an eligible young person, a center, or a neighborhood. State the target population, when exposure begins, the outcome and the follow-up horizon. “Beneficiaries” is ambiguous: it might mean everyone invited, everyone registered, or those who attended four sessions. Comparing completers with noncompleters can entangle the program with prior motivation, because motivation may affect both attendance and job search. That comparison describes observed groups; it does not automatically identify an effect.

The arithmetic 7.2 − 6.1 = 1.1 is correct as a difference between the two responding groups. The causal sentence does not follow. Some invitees chose to register; some registrants persisted; only a subset of each group responded. The groups may have differed in confidence or motivation before the program, and nonrespondents may differ from respondents. There is no shared baseline measure or credible allocation mechanism. A defensible sentence is: “Among respondents, mean confidence was 1.1 points higher for four-session completers than for nonregistrants; this difference does not identify the effect of mentoring.” For a delivery decision, 180÷400 = 45% is the share of invitees completing, provided the invitation roster is deduplicated. A causal expansion decision needs an appropriate comparison design before an effect claim.

Build a three-column matrix: decision, question, and required data or comparison. A delivery row asks how many of all invitees completed and requires one consistent invitation roster. An experience row asks why attendance stopped and uses transparently selected interviews that include dropouts; interview themes are not prevalence estimates for all invitees. An expansion row asks about the effect of being offered mentoring on confidence or employment in the eligible group and requires a predefined outcome, credible comparison, and follow-up plan. The same dataset can answer one row while leaving another unresolved.

Check the matrix before releasing it. Is the action decision explicit? Are the population, denominator and period named? Does the claim match the question type? A before-and-after measure for only the treated group describes change over time; it cannot remove labor-market trends or maturation. Better data collection improves descriptive accuracy but does not itself create a counterfactual. These limits make the recommendation auditable when more evidence arrives.

Classifying a question is not a word game. Add “What would change my decision?” to every row. For delivery, a manager can set a preliminary completion threshold and watch its trend, but must check whether the definition of an invitee changed between periods. For expansion, improved confidence may not be enough: the program’s professional aim may be job applications or employment, while confidence is only an intermediate step. “Did confidence improve?” and “Did the offer lead to more jobs?” ask about different outcomes. The first can move without the second. In the next lesson, a theory of change will make the proposed links between activities, near outcomes and final outcomes testable.

Do not answer the sponsor with a vague request for “more data.” Name a defect and a feasible response. An eligibility and invitation roster dated before the decision is needed to define the target group. A privacy-preserving identifier can support deduplication. Completion needs a stable definition, and the outcome needs a specified follow-up time for all feasible comparison members. Collecting these fields alone will not establish a valid causal design; assignment to groups and threats to their comparability still need examination. If no credible comparison is feasible this cycle, the organization can publish a transparent descriptive performance report and defer the impact claim.

Another error is changing the unit after seeing the numbers. Completion may look low among invitees and high among registrants. Both calculations can be useful, but one addresses reach and the other retention. Do not select the denominator that flatters the program. Draw a flow: eligible → invited → registered → completed → responded at follow-up. Put a person count at every node and record known reasons for missing outcomes. The flow exposes attrition and selection. Repeated invitations do not turn one person into two people in a person-level completion rate; document the deduplication rule before computing it. If eligibility rules changed during delivery, separate the periods or show how the changed rule affects comparability.

Lesson summary

Apply separate a decision question, descriptive question and causal question to a hypothetical case and justify a bounded decision.