Problem framing and theory of change

Review questions

Problem framing and theory of change — Integrated Summary

The module turns a broad request to prove a social initiative successful into distinct questions for distinct decisions. Learners separate reach description, implementation explanation and the effect of an offer on a defined outcome. They build a theory of change connecting resources, activities, near outcomes and final outcomes, recording assumptions and rival explanations rather than drawing unsupported causal arrows. They weigh relevance, effectiveness, equity and cost against the decision and its stakeholders, including people missing from routine data. An evaluability review then asks whether eligibility, denominators, outcomes, a comparison and suitable permissions are available. The output is a question matrix and a conditional go, repair or defer memo. A before-and-after change alone remains descriptive.

Review questions

Test your understanding with these questions. Marking and saved results are available with a free account.

  1. Question 1Hypothetical training scenario in a fictional city: 400 eligible young people are invited to mentoring. Of these, 260 register and 180 complete four sessions. At three months, 150 completers and 90 invitees who never registered answer a confidence scale from zero to ten. Responding completers average 7.2; responding nonregistrants average 6.1. The sponsor asks to publish: “Mentoring raised confidence by 1.1 points.” What is the most defensible conclusion about separate a decision question, descriptive question and causal question?
  2. Question 2Sessions were offered to 400 eligible young people; 180 completed four sessions. In a review of 60 completers’ files, 42 met a résumé standard set in advance, while 25 of the file owners submitted at least one application during the month. A manager says, “Since 70% met the résumé standard, 70% of all invitees became ready for work.” Two leaps are present. The 42÷60 = 70% rate describes reviewed files from completers, not all invitees. Readiness for work is also broader than one résumé standard. A defensible statement is: “Among 60 reviewed completer files, 42 met the résumé standard; we need to know how files were selected and assess other skills before generalizing.” Then ask why only 25 in this set applied. Were vacancies unsuitable, the application platform difficult, or follow-up support inadequate? These are rival explanations to test, not ready-made excuses. What is the most defensible conclusion about draw an intervention logic with assumptions and rival explanations?
  3. Question 3Mentoring was offered to 400 eligible people: 240 live near the center and 160 farther away. Four sessions were completed by 144 near residents and 36 far residents. Completion is 144÷240 = 60% among near invitees and 36÷160 = 22.5% among far invitees. Overall completion is 180÷400 = 45%. The sponsor says, “We beat the 40% reach target, so let us expand.” The overall rate is correct but masks a 37.5 percentage point gap. Its cause is unknown: travel cost, session time, alternative opportunities, or differences in willingness to enroll could all matter. It is professional to report an access gap worth investigating, not to assert intentional discrimination or differential impact. If equitable reach is a scale objective, testing a closer venue or different schedule while tracking each group is more defensible than simply scaling the current model. What is the most defensible conclusion about set evaluation criteria, stakeholders and distributional questions?
  4. Question 4An expansion committee sees 400 invitations, 260 registrations and 180 completers. Some invitations lack dates, and 30 records may be duplicates. Confidence was asked after three months of 150 completers and 90 nonregistrants, with no baseline. Mentors recall registration starting a week before the public announcement, but no record shows who heard the offer early. The sponsor wants an effect percentage before a budget meeting in four weeks. What is the most defensible conclusion about test evaluability and issue a go, redesign or defer recommendation?
  5. Question 5In a hypothetical training scenario in a fictional city, 400 adults are eligible for job-search skills support. Capacity permits offering it to 200 now and deferring the offer to 200 until the next round. For the example, assume the offer is randomly allocated before service starts and outcome collection is complete and comparable. One hundred twenty offer recipients register and attend at least one session. Within three months, 60 people in the offer group submit at least one job application, versus 48 in the group not yet offered a place. The offer-group rate is 60÷200 = 30%, the comparison rate is 48÷200 = 24%, and the gap is six percentage points. Under valid allocation and follow-up, this is an initial estimate of the **offer** effect in this sample. It is not the effect of completing sessions, nor is the difference automatically precise enough for scaling without uncertainty analysis. What is the most defensible conclusion about define treatment, eligible population, estimand and counterfactual?
  6. Question 6A skills initiative wants to evaluate an offer of four sessions across six centers, each serving 80 eligible people. Trainers run group sessions, and learners at each center share a messaging platform. Three trainers can begin in three centers now. A manager suggests drawing 40 names within each center and calling everyone else “controls,” while leaving the messaging platform open to everyone. This individual allocation will not yield a clean contrast of the offered service. Session materials may reach noninvitees, who may also experience exclusion while watching others take part, and trainers may alter how they support everyone at the center. What is the most defensible conclusion about assess random assignment feasibility and contamination?
  7. Question 7A hypothetical training initiative in a fictional city prioritizes mentoring for people scoring at least 50 on a work-readiness test. Application outcomes after two months are available for 80 people scoring 48–49 and 75 scoring 50–51. Thirty-two in the first group and 36 in the second applied, rates of 40% and 48%. The raw eight-point gap is between groups near the threshold. It should not yet be published as “mentoring raised applications by eight points.” A referral officer may have raised some scores to 50; a separate digital service may also begin at 50; or follow-up may differ. Check allocation records, the distribution of scores around 50, parallel-program records and outcome loss. If the cutoff rule, lack of manipulation and smoothness of other factors are credible, a suitable discontinuity analysis may estimate a local effect with uncertainty. This case supplies none of those validation results. What is the most defensible conclusion about compare quasi-experimental designs and assumptions?
  8. Question 8In a hypothetical training scenario in a fictional city, mentoring is offered to 200 eligible people and compared with 200 whose offer is delayed. After three months, 160 offer recipients and 190 comparison people respond. Job applications are reported by 64 offer respondents and 57 comparison respondents. The respondent rates are 64÷160 = 40% and 57÷190 = 30%, a ten percentage point observed respondent gap. But outcomes are missing for 40 offered people and ten delayed people. Publishing “the offer effect is ten points” from respondents alone treats unequal loss as if it did not matter. What is the most defensible conclusion about plan sensitivity to selection, attrition and interference?
  9. Question 9In a hypothetical training scenario in a fictional city, a social center offers guidance to 240 eligible households. In a first survey, 200 households answer “Did you obtain an appointment in the last 30 days?” Sixty say yes, or 30% of respondents. Three months later, 200 households answer “Have you ever obtained any appointment?” Ninety say yes, or 45%. A draft says the program “raised access from 30% to 45%.” Even if the same households answered, the questions use different time windows. If respondents also differ, composition adds another change. A defensible sentence is: “The first round recorded 30% under a past-30-day question and the second 45% under an ever-use question; these are not growth in the same indicator.” Inspect the instrument, select a stable question for the next round, and record the wording change in the data dictionary rather than hiding it. What is the most defensible conclusion about operationalize constructs with validity and comparability checks?
  10. Question 10A hypothetical initiative in a fictional city wants to estimate access to a confirmed appointment. Field staff estimate 600 eligible households in the service area, but the contact list has 500; the hundred missing are thought more likely to avoid the digital channel used to build the list, a hypothesis requiring verification. A random 200 from the list are selected, 160 respond, and 64 report a confirmed appointment within 30 days. The respondent rate is 64÷160 = 40%; response among selected households is 160÷200 = 80%; estimated frame coverage is 500÷600 ≈ 83.3%. “Forty percent of every eligible household obtained an appointment” is unsupported. Unlisted and nonresponding households may differ in access. The repair plan starts by verifying the estimate of 600 and investigating who is missing, then building an appropriate way to include nondigital households and improve follow-up. What is the most defensible conclusion about construct sampling frame and reason about precision?
  11. Question 11In a hypothetical training scenario in a fictional city, a field lead receives 500 forms in one week. Twenty-two are flagged for possible duplicate household numbers, 30 for age outside eligibility and 40 for a missing primary outcome. Do not announce “92 bad forms” by summing 22+30+40. Flags can overlap; a repeated number may describe a second appointment for the same household or a typing error. Build a form-level table with an authorized identifier, flag, entry date, interviewer and result of checking the source. Prioritize missing outcomes by center and interviewer, duplicates that threaten denominators, and out-of-range ages that may be eligibility or entry mistakes. Retain raw data with limited access, then make a cleaning copy and change log; never silently rewrite the raw value. What is the most defensible conclusion about build field quality controls and missingness response?
  12. Question 12A hypothetical initiative in a fictional city wants to know whether guidance sessions help households obtain a service appointment within 30 days. A proposed file collects household head name and phone number, children’s names and ages, exact monthly income, reason for seeking service and appointment date. Three partners each request a complete weekly copy. The primary question shows no need for children’s names, exact income or universal file sharing. A separate study ID, invitation and appointment dates, and access outcome may suffice, while protected contact information stays with a small follow-up team if follow-up is necessary. If age-related equity is a well-defined question, discuss broad age categories rather than names and full birth dates, subject to the relevant organization’s rules. Partners receive appropriately aggregated decision information. What is the most defensible conclusion about apply proportionate consent, privacy and safeguarding?
  13. Question 13In a hypothetical training scenario in a fictional city, a team defines the primary result before launch as at least one application within 90 days. Application count and résumé quality are secondary. After collection, the primary gap is small, but a large résumé-quality gap appears among 23 people who attended extra sessions. The sponsor asks to lead with “the program worked for highly committed attendees.” This is a post-offer group selected on receipt, a small sample and a secondary outcome. Motivation or earlier opportunity may explain it. Report the planned primary result and its limits first. Present résumé quality as exploratory, with the subgroup count and reason it was examined, rather than upgrading it to a causal effect of the initiative. What is the most defensible conclusion about pre-specify estimands, missing data and subgroup analysis?
  14. Question 14In a hypothetical training scenario in a fictional city, an initiative allocates session offers to people under a documented plan. Its preliminary report shows applications from 70 of 200 offered people and 50 of 200 comparison people at 90 days, an observed ten-point gap. For the example, the design-appropriate analysis supplies an uncertainty interval from −2 to +22 percentage points. It remains compatible with a small harmful effect, no difference and an improvement above the scale committee’s eight-point threshold. This does not mean every value is equally likely, nor does a conventional frequentist interval mean a fixed 95% probability that this particular true effect lies inside it. It means these data and this method do not narrow the decision enough to separate those possibilities under the analysis assumptions. What is the most defensible conclusion about calculate and interpret a bounded impact estimate?
  15. Question 15In a hypothetical training scenario in a fictional city, a roster of 300 invitees has 180 near a center and 120 farther away. Four sessions are completed by 108 near and 36 far residents, rates of 60% and 30%. Twelve purposive interviews are conducted, six with completers and six with dropouts. Seven interviewees mention conflict between session times and part-time work, and four mention trouble reaching digital vacancies. Do not say “58% of all participants had a schedule conflict” because 7÷12 describes selected interviews only, and these interviews were not necessarily intended to estimate barrier frequency even within interviewees. A careful sentence is: “Schedule conflicts arose in seven of twelve purposive interviews. Far residents’ completion was 30% versus 60% nearby, so session timing deserves a test; we do not yet know how much it explains the gap.” What is the most defensible conclusion about triangulate qualitative accounts with quantitative findings?
  16. Question 16In a hypothetical training scenario in a fictional city, a job-application outcome is recorded for assigned offer and comparison groups. Among people near the center, 45 of 100 offered and 35 of 100 comparison people apply, a ten percentage point gap. Among far residents, 20 of 100 offered and 18 of 100 comparison people apply, a two-point gap. The arithmetic difference between subgroup estimates is eight points. These totals alone do not show whether that difference is reliable or whether allocation and follow-up were valid within groups. One hundred per cell may be too little to distinguish modest differences, and distance may mark different referral channels or job opportunities. Present counts, outcome horizon, gaps and planned uncertainty, then ask whether distance changes receipt or usefulness of sessions. What is the most defensible conclusion about evaluate heterogeneity, mechanisms and external validity?
  17. Question 17In a hypothetical training scenario in a fictional city, a nonprofit evaluates six months of job mentoring. Before launch, its primary outcome was “job application within 90 days,” measured in both groups under a plan. The sponsor sees a small application gap but higher satisfaction among completers. They request making satisfaction the primary outcome and dropping the comparison arm because “people dislike being called controls.” This is substantive, not editorial: satisfaction among completers does not estimate the effect of offering mentoring to all eligible people, and dropping comparison removes the basis of the causal question. Log the request, explain its effect, and refuse to rename the primary outcome after results. Satisfaction may be reported as a secondary service-experience result with its denominator and limits. Participant-facing language can be revised respectfully without deleting the analytic comparison record. What is the most defensible conclusion about contract governance, independence and protocol changes?
  18. Question 18In a hypothetical training scenario in a fictional city, a mentoring evaluation estimates a six percentage point offer difference in submitting a job application by day 90, with an interval from −2 to +14 points under its chosen analysis. The committee had set an eight-point minimum for broad scaling and a cost ceiling of 500 monetary units per served person under a common cost definition. Observed cost is 430 units per completer but excludes volunteer time and participant travel; “completer” also differs from the committee’s “served person” denominator. Completion is 60% near the center and 25% farther away. Selected interviews mention varying schedule problems but do not estimate their prevalence. “Impact is proven and efficiency achieved” is unsupported. The interval reaches below zero and above the threshold, cost scope is incomplete, and the access gap needs examination. What is the most defensible conclusion about synthesize evidence using fit-for-purpose criteria?
  19. Question 19In a hypothetical training scenario in a fictional city, a nonprofit’s preliminary report estimates a six percentage point gap in submitting an application within 90 days, with an interval from −2 to +14 points. Follow-up loss is 12% in offer and 5% in comparison. Completion is 60% near the center and 25% far away. A draft summary says, “The initiative proved it raises employment chances, and evaluators recommended immediate expansion.” That sentence slips four ways: the measured outcome is applications, the interval does not settle the scale threshold, unequal loss threatens interpretation, and the access gap questions expansion in the same form. An executive sentence can say: “Data show a positive estimated application gap, but they do not settle an effect sufficient for broad scaling; we recommend a limited access test and stronger follow-up before the scale decision.” The technical report then gives denominators, allocation, interval method, attrition sensitivity and transfer limits. What is the most defensible conclusion about write transparent limitations and communicate to audiences?
  20. Question 20In a hypothetical training scenario in a fictional city, a nonprofit has eight centers, each able to invite 100 eligible young people. Capacity permits starting at four centers now and four after four months; some centers share a public messaging platform. A committee asks whether the **offer** of four mentoring sessions raises submission of at least one application within 90 days by at least eight percentage points, to decide on expansion. The old invitation register has duplicates, and its outcome item asks only “Did you try to look for work?” without a time window. Center cost differs according to whether participant travel is covered, and there is no deletion plan for contact data. A ready-made form cannot solve these defects; sequential repairs are needed before the first invitation. What is the most defensible conclusion about assemble and defend the integrated impact protocol?