Form risk hypotheses from conflicting evidence

When a financial crime programme's indicators conflict, do not average them into a verbal compromise. Locate each indicator in the causal chain. Suppose alerts per thousand transactions fell while the proportion passed by investigators to a case team rose. The rule set may have become more selective. It may instead have lost a category of easily detected transactions. Or investigators may have tightened the referral standard. The first indicator compares alerts with eligible transactions; the second compares referrals with alerts whose reviews were closed. Either can change because its denominator changed, without any change in underlying risk. An adviser should construct rival hypotheses that could be wrong and ask for evidence that distinguishes them. Declaring either 'improvement' or 'failure' before this step skips the diagnosis the institution needs.

Create an evidence ledger with four columns: observation; source, period and definition; possible interpretation; and what the observation does not establish. For example, 24 of 200 closed alerts were referred this quarter, against 30 of 500 in the previous quarter. If the populations are comparable, the referral share increased from 6% to 12%. That does not prove that real risk doubled or that investigators became better. The referral threshold, selection of the population or backlog may have changed. Ask who counted closed cases, whether referral is a preliminary or final outcome, and whether cases opened in an earlier period entered the new denominator. Put 'unknown' in the ledger when the answer is missing. Zero means a measurement found no cases; unknown means no sound measurement has been obtained.

Write three rival hypotheses as a proposed cause and an observable consequence. An improved rule should remove low-value alerts while retaining coverage of higher-risk patterns. A broken data feed should cause a concentrated drop in one channel or field, perhaps excluding transactions even from the eligible-volume denominator. A change in investigation policy should shift referral rates even among alerts generated by unchanged rules. Give each hypothesis a test capable of weakening it. Like-for-like cohort comparisons probe operational change; replaying a known test set through old and new logic probes detection differences; and reviewing investigator instructions and training records probes a change in referral decisions. Do not call a test proof of coverage if the sample was selected by the rule designers or contains no missed cases.

Prioritise tests by their effect on the decision, not by which dashboard is easiest to export. If the loss of an international-transfer feed could create an urgent blind spot, examine completeness first even when the probability of that defect is uncertain. Then test whether a new rule improved referrals; next assess changes in investigator behaviour. Use an explicit qualitative scale, such as potential impact high, medium or limited and evidence confidence strong, moderate or weak. Multiplying ordinal labels does not create a scientifically measured probability. Define a trigger for immediate referral to the authorised control owner, for example loss of an entire transaction category or potentially prohibited activity. Such escalation protects the control. It does not make the adviser the adjudicator of an individual case that the responsible team has not examined.