Evidence lineage and rival explanations

In the week after a loan-system migration, reported 30-day arrears rise from 4% to 7%. The credit head says underwriting has deteriorated; the collections head says cash receipts have not fallen. Do not select the louder account. First define the ratio. Is it the number of accounts more than 30 days late divided by all active accounts at a specified date, or the balance on late accounts divided by total principal? One large account can move the balance ratio without moving the count ratio much. Is age measured from the original due date or the date of a rescheduling? Two points are not comparable if their measurement units or ageing rules differ, even when the columns share a name.

Build an evidence lineage. The first row states the claim: ‘weaker underwriting raised delinquency.’ The second states the indicator definition and extraction date. The third names the source system, data owner and conversion rule. The fourth records a repeatable check, such as reconciling account count and principal to the loan ledger at the same date. The fifth lists independent or contrary evidence: cash collections, restructuring movements and samples by origination vintage. The sixth records a provisional decision and confidence limit. A polished chart is insufficient. A loan timestamp from the new system and cash receipts from the old one may describe different periods. Record that timing difference before drawing a conclusion.

Test three hypotheses with observations that distinguish them. An underwriting hypothesis predicts worse results for newly originated vintages than earlier vintages at the same age. Compare the share late two months after origination in each vintage; do not use a pooled portfolio rate that changes as loans age. A customer-stress hypothesis predicts deterioration across exposed industries or regions and may also affect collections. A migration fault predicts a jump at the conversion date, inconsistency between loan and cash records, or duplicated due dates. Each hypothesis needs its own discriminating test. Running the same query three times with different labels does not test alternatives.

Stable cash is not a certificate of sound credit. A customer may pay late but before month end; early payments from other accounts can hide a decline in a narrow segment. Distinguish evidence that refutes a claim from evidence that simply cannot settle it. Record the coverage of collections data, the share of missing files and who can amend records. If a manager’s interview conflicts with a committee minute and a transaction log, prefer the contemporaneous record and test it against sampled transactions; memory after the event is not equivalent to a reproducible trail. Two causes can coexist: a technical measurement effect and a smaller real decline in a recent vintage.

Present the result to the sponsor in three columns: established, plausible, and untested. For example, the denominator changed after migration; part of the reported jump plausibly reflects arithmetic; the last quarter’s vintages have not yet been shown to be sound. Attach an action to each: repair the dashboard definition, run like-for-like vintage analysis, and defer a higher credit limit until the sample is verified. This prevents opposite mistakes: blaming the underwriting team for an unproved failure or closing every credit concern because a data defect exists. Good risk advice does not erase uncertainty in prose. It makes the decision consequence and the date for reducing uncertainty visible.

Make each hypothesis test explicit with a predicted pattern, contrary evidence and observed result. A migration defect might cluster newly late cases around days 29–31 without a comparable change in cash receipts. If added cases extend evenly to 90 days, a boundary error may not explain most of the rise. For weaker underwriting, guard against vintage-age bias: a loan originated last month has not had time to become 90 days late. Hold loan age, observation date and product type constant before comparing cohorts. Unchanged cases can serve as a reference group only if they passed through the same data route. Report the share of records that cannot be linked across systems. A 98% count match could conceal the remaining 2% among the largest balances, so reconcile by value as well as count.

Document contradictions while analysis is still underway. If cash records show payment but the arrears report still counts the customer, check whether the receipt applies to that invoice or another debt and whether it posted after report extraction. Preserve transaction number, two dates and amount in a restricted reconciliation record. One rare case cannot be generalized; a repeating pattern at the 30-day boundary warrants a query across all boundary cases. Disagreement between teams becomes a measurable hypothesis rather than blame. For records still unlinked, show a result range indicating how much the decision metric could move once they are resolved.

Check missing-data bounds before claiming confidence in a ratio. If due dates are absent for 500 of 10,000 accounts, a reported 7% delinquency rate cannot automatically represent the whole book. The missing cases might be new and sound or concentrated in a troubled branch system. Show rough extremes: none late versus all late, explicitly as sensitivity bounds rather than forecasts. Their spread tells the decision maker how much the data gap could change the result and how urgently it needs repair. Inspect balances too, because five per cent of accounts might carry a much larger share of principal.