Diagnosing adoption opportunity and work context · Lesson 5

Select use cases under constraints

6 min read

Lesson objectives

  • Compare eligible adoption opportunities after non-tradeable constraints, testing ranking sensitivity and learning value.
Reading tools

Once work, baseline, and affected groups have been mapped, plausible opportunities usually exceed an organization’s capacity to pilot them. A good choice begins with a consequential result under manageable constraints, not the vendor’s most impressive demonstration. Define a unit of use for each option: a specific request class, user, decision or draft, and success or stop condition. “Use AI in human resources” is too broad to test. “Help an HR specialist locate the leave-policy passage for review before replying to general questions” is narrower and reveals data, harm, and authority questions. The narrow scope supports honest learning; it is not a promise that later scaling will be safe by default.

Run eligibility before ranking. May the data be used for the proposed purpose? Does the team have access and publication authority? Is there a realistic way to catch an error before it causes harm? Is someone empowered to stop the use? If a material condition is unknown, do not mark it “passed” with an arbitrary low risk score. Hold that option and request specific evidence while other options proceed. A held use case may become eligible through a different data source, a narrower task, or an effective human review. This separates “may and can we control it?” from “which eligible option is best now?” It is neither permanent cancellation nor an attempt to turn a mandatory condition into a tradable score.

For eligible options, announce criteria and weights before seeing the result. Outcome value, knowledge readiness, measurability, implementation cost, and what the organization will learn from a pilot may be useful. Scores are internal estimates, not measured impact. Ask each owner to explain a score and name the information that could change it. A weighted total can hide uncertainty behind decimals; plausible ranges may reverse a ranking. Separate the attractiveness of a use case from organizational readiness. A high-value case may be a poor first pilot if its data are disordered or reviewers are unavailable. An easy, low-value case may make an attractive demonstration while teaching little about a consequential scaling decision.

Assume A and C then become eligible. A joint team scores five points per criterion, with higher better: outcome value 35%, measurability 25%, work readiness 20%, and learning for later scale 20%. A scores 3, 5, 4, 4. Its total is 0.35×3+0.25×5+0.20×4+0.20×4=3.90. C scores 5, 3, 2, 4, totaling 1.75+0.75+0.40+0.80=3.70. A leads by only 0.20, not by an objective law. If C’s readiness rises from two to three after policy repair, it reaches 3.90 and ties. If C’s true value is four rather than five, it falls to 3.35 before the readiness repair. This sensitivity indicates the next evidence to gather: readiness and value of C matter more than a longer argument about chart design.

Do not let the total erase a non-tradeable condition. If testing B could influence real employment opportunities without an acceptable basis for data use or a route to challenge a decision, a hypothetical 4.8 on a scorecard does not qualify it. “A person reviews every screening result” is no automatic cure. The reviewer needs relevant information, enough time, and genuine authority to reject an unreliable ranking. Even A’s lower sensitivity does not eliminate harm: a wrong maintenance summary could produce an unsafe order. Bound the pilot to noncritical equipment, require a technician to verify the source passage, and keep error and stop records. Choosing the first use case means choosing learning and risk that can be managed, not the most attractive title.

The MANAGE function of the NIST AI RMF weighs benefits and risks and asks whether AI is appropriate for the purpose. MAP asks for context and consideration of alternatives, including non-AI ones. Our decision uses those ideas, but the weights and scores in the example are instructional estimates, not a NIST specification or market forecast. A short decision record lists the three options, each eligibility condition, assumptions behind scores, success measure, stop owner, and reconsideration date. Include a non-AI option too: a maintenance index might outperform generated summaries if search is the only problem. New evidence may revise the choice. One successful pilot cannot authorize similar deployment in recruitment or customer service by analogy.

When options are close, do not demand precision that a scorecard cannot provide. Ask about opportunity cost: which pilot cannot run during the six weeks if A is selected? C may teach the organization more about customer-facing content, while A may be preferable now because review can be tested without exposing sensitive information. Turn “learning value” into a testable question: can a technician identify an incorrect maintenance-policy passage within two minutes? What share of errors is found only after a work order? When does the team stop? State the transition gate too: after A succeeds, which new safeguards are needed before trying C? A’s score and technicians’ training do not transfer automatically to service staff. Record a no-AI decision when the nontechnical alternative wins. If repairing the maintenance index solves search at lower cost while preserving order correctness, leadership has solved the problem even without a model. Name what evidence would reverse that conclusion, such as complex searches the index cannot serve.

Lesson summary

Compare eligible adoption opportunities after non-tradeable constraints, testing ranking sensitivity and learning value.