Architecture decision under outage and growth

An architecture choice assigns responsibility

A payments firm may contract with a customer app, payer account provider, system operator, settlement participant, payee provider, and network carrier. When a payment stalls, the customer asks for one result while each party sees a different part of the chain. A diagram of technical boxes is insufficient. Label each connection with the sender, receipt acknowledgment, owner of the authoritative status, party able to correct an error, and response deadline. These boundaries determine dependable service as much as the technology does. More intermediaries usually increase the need for a shared identifier and reconciliation. Fewer intermediaries may move operational and liquidity burdens onto the firm itself.

Reliability is not simply server uptime. Imagine an app available 99.99% of the time while a settlement partner sends state changes an hour late during peak demand. The customer can open the app but cannot tell whether retrying will duplicate a payment. The app is available in a narrow technical sense and poor in the functional sense. Measure the proportion of instructions reaching a comprehensible outcome. Then separate valid rejections, technical outages, funding waits, and uncertainty about status. A valid rejection is not a system outage, yet it still belongs in the customer's experience measure. Decision makers need both views, because improving one does not guarantee improvement in the other.

Choose recovery according to the money's state

If a messaging route fails before an instruction is accepted, it may be safe to resubmit with a stable identifier after establishing that no acceptance occurred. If it fails after acceptance but before visible finality, a switch to another partner is dangerous: the first partner may complete the payment while the second also processes a retry. The rule is not "send through the backup immediately." Freeze re-execution, query the original identifier, establish the settlement state, then choose a documented retry or correction. If settlement is final but customer notification failed, do not reverse the money to repair the screen. Rebuild the notification from an authoritative event record. Recovery differs because the payment has crossed different financial boundaries.

Design a finite state machine: CREATED, SUBMITTED, ACCEPTED, SETTLED, AVAILABLE, REJECTED, and UNKNOWN_PENDING_RECONCILIATION. A user-interface timeout must not move UNKNOWN straight to REJECTED. A status value in a database is not evidence by itself. Connect each transition to an event from the authorized party with a timestamp, order, and source. On the customer screen use understandable language instead of internal codes, such as "We are checking the result of your payment," with a promised update time and contact route. If the customer needs an immediate purchase, explain that a pending instruction should not simply be repeated. Temporary credit would be a separate decision within the firm's approved limits.

Growth changes the meaning of backup

A backup link tested with 100 instructions per hour may fail to carry 5,000 per hour when the primary route goes down. Separate message throughput, settlement liquidity, and exception-team capacity. A backup may carry messages but lack enough funding at the peak. It may have funding while manual review of new customers becomes the bottleneck. Run a controlled failover test. Move a small, authorized share of new instructions during a known window and verify identifiers, reconciliation, and safe return to the primary route. Record real time to recover the customer service, not merely the seconds required to flip a technical switch. Do not conduct a test that places real customer funds at unapproved risk; use a simulation or a tightly authorized live slice.

Two partners do not automatically mean diversified resilience. Both may depend on the same carrier, settlement bank, or fraud data service. Map shared dependencies down to the level that can stop service, then include them in the stress scenario. If the partners use different meanings for "accepted," implement a documented translation layer with tests instead of assuming equivalent words. Sometimes a short, clearly explained pause is safer than automatic failover. The judgment changes if waiting creates greater harm than duplication and if the switching, reconciliation, and funding controls have been proven.

Write a decision a board can inspect

A product team may offer four options: retain one partner and improve reconciliation, add a manually operated backup, add conditional automatic failover, or build direct access. Do not rank them using undefined scores. Begin with an eligibility gate: any option unable to demonstrate that a pending instruction cannot execute twice during switching is withheld until that risk is addressed. Compare eligible options by time spent uncertain, maximum pending value, liquidity cost, operating cost, and ability to explain status to customers. Set the relative importance of these measures from the firm's objective and risk limits before seeing the figures. Changing weights after a favored option emerges turns analysis into post-hoc justification.

Use fictional decision figures. The present system leaves 0.8% of instructions unresolved for more than five minutes on a stress day, with a maximum pending value of 1.6 million. Better reconciliation costs 120,000 annually and might reduce the rate to 0.3% if linkage tests hold. A manual backup costs 300,000 and aims for 0.2%, but may remain slow at the peak. Automatic failover costs 550,000 and aims for 0.1%, yet has not passed a duplicate-prevention test. Do not choose automatic failover merely because its forecast rate is lowest; it has failed the entry gate. Reconciliation could be preferable if the firm's limit is 0.4% and the remaining exposure fits its approved ceiling. The figures are teaching assumptions, not market forecasts. The pilot must actually test the 0.3% expectation before value is claimed.

Specify what would reverse the choice. If the residual 0.3% of pending instructions concentrates in high-value transfers and exceeds the 1.6 million exposure ceiling, reconciliation alone may fail despite a good count rate. If duplicate prevention is proven and automatic failover becomes less costly at higher volume, it returns to consideration. If both partners share a dependency that makes their outages correlated, backup value falls even when the links appear separate on a diagram. Put these conditions in the decision memo with a plan to obtain the evidence. The choice can then be corrected rather than defended as an unconditional promise.

Measure recovery from the affected person's perspective

Suppose intake of new instructions stops for ten minutes, while 40 existing instructions remain of unknown status for two hours. "Ten minutes of outage" describes intake only. It does not describe completion of the service. Measure three recovery layers: new-instruction acceptance, resolution of pending instructions, and restoration of reconciliation and balance reporting. If the first recovers in ten minutes, the second in two hours, and the third the next day, customer harm and financial risk did not end at minute ten. Assign a measure and escalation threshold to each layer. That prevents an incident from being closed as soon as servers turn green.

Describe impact without false precision. Count unique customers awaiting a result, unresolved value, merchants whose delivery was delayed, and confirmed duplicate complaints. Do not multiply transaction count by an undocumented average loss and call it a confirmed loss. If a money estimate is needed for a decision, give a range with explicit assumptions, such as support cost per unresolved case and a defined early-credit amount, then test whether the option ranking changes across the range. Separate improvement cost from a rare but severe incident loss. An everyday average must not erase a potentially large tail event. This is language in which non-engineers can understand investment in reconciliation and limits before investing in a faster screen.