AI Strategy and Value · Lesson 2

From Problem Framing to an AI-or-No-AI Value Hypothesis

15 min read

Lesson objectives

Frame a solution-neutral problem and compare AI with non-AI alternatives.
Build a value hypothesis with a baseline, target, mechanism, guardrail, and continuation decision.
Calculate a simple workload baseline and distinguish association from a causal claim.

Separate the problem from the solution before designing a value hypothesis

A good problem statement describes a performance or decision gap without assuming AI is the answer. Instead of “We need a model to predict delays,” say, “The procurement team cannot detect delay signals early enough to change the plan, which causes repeated rescheduling.” The second statement opens several interventions for comparison: clean delivery-date data, change alert rules, improve supplier commitments, or use a predictive model. This comparison reduces solution bias, where every problem is interpreted as a reason to build a model.

The Map function of the NIST AI RMF is useful here because it emphasizes understanding context, purpose, benefits, costs, and risks before moving ahead. In this diploma we use that idea as a decision tool: the less we understand the problem, stakeholders, data, or impact of error, the weaker the value hypothesis. This does not mean waiting for perfect certainty. It means identifying what must be known before taking on greater cost and risk.

From problem to a testable value hypothesis

Hypothesis element	Practical question
Problem	What is the current gap, and who is affected?
Baseline	What is the current state in a measurable unit?
Intervention	What change is proposed: AI, non-AI, or a combination?
Target	What improvement is expected during the test period?
Value mechanism	How is the intervention expected to create the improvement?
Guardrail	What quality, safety, or cost outcome must not be sacrificed?
Continue/stop decision	What evidence would make us scale, redesign, or stop?

Formula

Rework load in the case = number of requests × rework rate × rework minutes ÷ 60. This does not estimate every process cost, but it converts a vague problem into an understandable baseline that can be compared after an intervention.

Guardrails prevent one metric from improving at the expense of the system

Processing time can fall because the system accepts lower-quality answers, or because staff postpone difficult work. Use at least one value metric and one guardrail when an important trade-off exists. Examples include response speed with classification accuracy; reviewer productivity with returned-case rate; cost reduction with escalation rate; or coverage expansion with privacy and security boundaries. A guardrail is not a long disclaimer. It is a measure or condition tied directly to a credible failure mode.

Exercise

Exercise 1 — Rewrite this statement: “We want a chatbot because competitors use one.” Replace it with a specific problem, one baseline, two interventions including one non-AI option, an experimental target, and one guardrail. Then state what would make you choose the non-AI option.

Check your answer

Guidance: “Thirty-five percent of simple customer questions reach an employee even though a documented answer exists. Intervention A improves search and navigation in the help center; intervention B is a conversational assistant retrieving from the same content. We test reduction in employee transfers while measuring answer accuracy and escalation.” If search improvement alone achieves the required value, or if the knowledge is stale and unstable, the non-AI option may be the better decision or the necessary first step.

Exercise

Exercise 2 — In the numerical case, assume mandatory fields reduce rework from 30% to 22%, then an AI assistant is introduced and the rate falls to 19%. What can you say confidently, and what cannot be attributed to AI without a stronger comparison? Propose a simple way to improve the inference without turning the experiment into an academic study.

Check your answer

Feedback: we can describe the sequence: the form change was associated with an eight-percentage-point decline, followed by a further three-point decline during the AI stage. The sequence alone does not prove AI caused the three points; experience, case mix, or volume may also change. Improve inference by keeping metric definitions and periods stable, observing a similar channel or group where practical, or using phased introduction with a clear before/after comparison and a log of other changes that could explain the result.

Lesson summary

Separate the problem from the technology and turn it into a testable hypothesis. Use a baseline, guardrail, and decision rule, and distinguish an observed result from a claim that AI caused it.