AI Governance Foundations · Lesson 1

The learner distinguishes probabilistic AI output from deterministic rules and chooses a control proportionate to harm and context

5 min read

Lesson objectives

  • The learner distinguishes probabilistic AI output from deterministic rules and chooses a control proportionate to harm and context.

Governance starts by defining the system and its function, rather than attaching an AI label to a product. A deterministic rule returns the same result for the same input when the rule and environment remain fixed. A statistical model may return an estimate or generated output that varies, and its behavior can depend on training data and the distribution of new cases. This distinction shapes testing: inspect rule logic and exception paths for the former; examine model error, performance across groups and drift as data change for the latter. A deterministic rule is not automatically safe: a wrong exclusion rule can also cause harm at scale. In a service application triage example, an overall accuracy of 92% is insufficient to approve use. First ask whether the output advises an officer or takes effect as a decision. Who bears an error? Can a person appeal or obtain meaningful human review? Then define the harm from false positives and false negatives separately. If an eligible application is classified as ineligible, access to the service may be delayed. If an ineligible one is classified as eligible, scarce resources may be consumed. The relative severity depends on the service, so an acceptance policy cannot be inferred from one average accuracy score. Responsible governance principles become useful when translated into testable questions about fairness, safety, transparency and accountability. Fairness does not require every number to match across groups without considering sample size and decision context. Measure the error rate associated with the relevant harm for each group and set a review threshold when the difference matters. Safety calls for testing unusual cases and a stop plan if serious harm appears. Transparency calls for a comprehensible explanation of the system's role and limits to affected users. Accountability requires an owner who authorizes release and examines incidents. Having an employee beside a screen does not establish effective human oversight if that person cannot change the outcome. Map the decision chain from input to human action and appeal. Record data provenance, model type and version, metrics, acceptance thresholds, user groups, and the owner of each measure after deployment. The chain separates model risk, such as poor performance outside test data, from use risk, such as turning internal advice into automatic denial without

authority. If the purpose changes, an unchanged model is not enough to justify continuing: reassess impact and controls because the context and harm have changed. The previous test set may also be unsuitable for the new purpose. For a practice question, read the facts before the answers: identify the system function, decision autonomy, affected people, error type and available evidence. Select an action that addresses the evidenced risk, rather than a broad slogan. If the evidence does not show how errors are distributed, seek disaggregated measurement or constrain use instead of declaring fairness from an aggregate score. If direct harm is already occurring, begin appropriate containment alongside investigation; do not wait for a perfect report. Exam-style reasoning often fails when the denominator for harm to one group is replaced by all applications. An urgent miss rate uses all truly urgent cases that should have been detected. It does not use all applications or only those labeled urgent by the tool. Positive predictive value uses the latter denominator and answers a different question: how many escalated cases were truly urgent? A smaller number is not a reason to substitute one metric for another. Every rate also needs a defined measurement period and enough cases; a striking difference from a tiny sample calls for uncertainty review before a lasting policy. A compact risk record identifies intended use, affected people, mechanism of harm, measurement evidence and control owner. In triage, delayed urgent service from low ranking is distinct from review overload caused by false escalation. The first may require prompt human review of every low-priority case; the second may call for threshold adjustment or resource planning. Do not compress them into a vague model-quality indicator. Test the control itself: does a person with authority receive the case before the critical deadline? Record receipt time, override time and rationale. If the organization cannot deliver that review, saying there is a human in the loop does not control the risk. Release decisions can be conditional. A limited pilot with no effect on rights and daily checks may be acceptable while using ranking to delay service remains on hold. Record who approves the pilot's boundaries, monitoring period and stop condition so it cannot drift into routine use without another review.

Lesson summary

The learner distinguishes probabilistic AI output from deterministic rules and chooses a control proportionate to harm and context.