Allocating an AI opportunity portfolio

A portfolio is more than a ranked idea list At the end of a quarter, a product council sees three attractive opportunities: repair support routing, assist with reply drafting, and improve knowledge search. Giving each idea an “impact” and “ease” score and adding them is not enough. The opportunities may serve the same customers and use the same data. An earlier project might be a prerequisite for a later one. A portfolio decision says where money and scarce team time go now, what waits for evidence that could change the choice, and which risks cannot be traded away. Treat figures as conditional comparison tools, not a supposedly objective ranking to two decimal places. Suppose this quarter's discovery budget is 60,000 monetary units and two teams can each run one experiment at a time. Opportunity A, repairing intake and routing, needs 12,000 and four weeks. B, a drafting assistant, needs 28,000 and six weeks, but depends on current policy documents. C, knowledge search, needs 24,000 and five weeks, and competes with B for the data team. These are discovery experiment costs, not the full costs of product development. B and C together cost 52,000 and fit the cash limit, yet compete for the same people. Draw resources and dependencies on a calendar as well as a budget. Claiming that all three can be funded is inaccurate: 12+28+24=64 thousand, exceeding the budget before staff constraints are considered.

A decision criterion, not a magic score For each opportunity, identify a user outcome, reachable volume, possible value, evidence quality, test cost, non-compensable risk and dependency. A aims to reduce transfers; it has a 72% baseline, a plausible cause supported by a sample and a relatively inexpensive test. B aims at faster correct resolution, but the measurement lesson exposed a correction risk from stale documents. C aims to reduce agents' search time, yet the team lacks evidence that search is the bottleneck in delayed cases. A therefore rises in test priority. Document governance becomes a condition for B. C needs a short task-time study before a broad build experiment. This does not prove that C has lower potential value; it identifies a different evidence gap. A council member might demand a unified score. A descriptive table can help: possible effect high or medium within a stated range, confidence high or low according to evidence type, and reversibility of the decision. Do not multiply “high=3” by “confidence=2” to produce six as if it were currency. Ordinal labels can organize a discussion but cannot handle B's dependence on clean documents or equate a low-harm failed test with disclosure of a sensitive request. Put beside each ranking a reason and a condition that would reverse it. The council should see what is known and unknown rather than submit to an elegant spreadsheet.

The value of information before the value of execution Sometimes the best short investment is a question that prevents the wrong project. Consider B in a simplified illustration. If policy documents are clean and current, a scaled assistant test might create net value of 100,000, a state assigned a fictional probability of 40%. If documents remain stale, it might lose 30,000, assigned 60%. Expected value of expanding B now is 0.40×100−0.60×30=22 thousand, before non-negotiable harm limits. With perfect knowledge of document condition, the team would expand only in the good state. Expected value with perfect information is 0.40×100+0.60×0=40 thousand. The difference, 18 thousand, is an upper bound on the value of perfect information under this model. A proposed document audit costing eight thousand might be worth considering, but its actual information value is at most 18 thousand because it will not be perfect. Do not label the audit's value “18” without estimating how accurately it distinguishes conditions and which decision its result changes. The probabilities are not facts. Forty percent is a teaching assumption, not market research. In practice, inspect a sample of documents, their effective dates, topics and share of requests depending on them. If the audit cannot change the decision because the team will run B regardless, its decision value is small even if its report is interesting. Conversely, a low-probability opportunity may justify a tiny reversible test instead of prolonged research. Information is valuable when it changes a consequential choice, not merely because it fills pages.

Test whether information earns its cost The audit result also changes the budget. The first 24,000 paid for A, the audit and observation, leaving 36,000. A positive document signal, if harm limits permit, can release 28,000 for B's experiment, leaving 8,000 and no room for C's 24,000 test this quarter. A negative signal could redirect part of the remainder to C, provided the search observation establishes a different need and the data team is available. These are conditional branches, not automatic promises. Give each branch a review decision, required evidence and spending cap. Unspent money is not failure when it prevents building a solution before its job is understood. Improve the document-audit example by allowing the audit to be wrong. Suppose it reports “good documents” for 80% of genuinely good cases and falsely reports “good” for 10% of bad cases. These are assumed test properties, not verified performance. The probability of a positive report is 0.40×0.80+0.60×0.10=0.38. After a positive report, the good-state share is 0.32/0.38, about 84.2%. Expected expansion value then is about 0.842×100−0.158×30=79.5 thousand. After a negative report, the good-state share is 0.08/0.62, about 12.9%, giving expansion value near 0.129×100−0.871×30=−13.2 thousand. Do not expand in that branch. Under a policy of expanding only after a positive report, expected decision value before audit cost is 0.32×100−0.06×30=30.2 thousand. Expanding immediately had expected value 22 thousand, so the imperfect audit's information value is 8.2 thousand. Subtract its eight-thousand cost and only 0.2 thousand remains under these assumptions. This is fragile: a small change in audit accuracy or cost reverses the case. It also cannot authorize a product that breaches an independent harm limit. The calculation suggests that a smaller probe or a better audit may be more useful than a large report. Show the state counts and resulting action beside the equations so one probability does not conceal the assumptions.