Design strategy measures resistant to gaming
16 min read
Lesson objectives
- Design a measure with a stable request population that reveals work displaced outside its calculation.
- Interpret faster processing alongside quality and backlog changes and formulate a conditional decision.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
The portfolio meeting opens with reassuring news: request completion time has halved. Minutes later, another manager shows a long list returned to applicants before acceptance. Has service improved, or has the number improved? Consider the alternatives before choosing an explanation. Staff may genuinely work faster, difficult work may have moved outside the measurement boundary, or both may have happened alongside a counting change. Your task is to make that distinction visible. Trace the whole request journey, then design a small set of measures supporting investment decisions rather than rewarding one attractive field. All figures below belong to original fictional teaching cases, not observations from an actual organisation.
Begin with the promise that matters to the user. If the aim is a correct service within a suitable time, counting records closed in a day describes only part of that promise. Users also need fair admission, a usable answer and a workable route when information is incomplete. Write the outcome before choosing the number. For example, a complete eligible application receives a correct answer within the agreed period, with suitable help to complete missing information. Speed, correctness and access are different dimensions. Decide which outcome is measured directly and which measures protect the rest. This distinction connects benefit and reporting principles in project-delivery guidance with process and outcome examination in evaluation guidance.
Map one request journey before aggregating hundreds. It can begin with an attempt to access the service, followed by registration, completeness checking, processing, a decision and use of the result. Define clock starts and stops at each transition. One measure starts at acceptance, another at initial submission; the difference is real waiting time experienced by the applicant. A returned application submitted again needs a connection between attempts. Issuing a fresh identifier each time may make one long journey appear to be two quick requests. Protect privacy through an authorised internal linkage rather than publishing personal identities. Trace only what the question requires, with access and retention controls appropriate to that record.
Separate a processing measure from a coverage measure. The first describes applications admitted to the workflow; the second asks what happened to requests needing service at entry. Retain both when entry rules can change the population. A return may be legitimate because the request is outside remit, or useful because an essential document is missing. Record reasons, completion time and available help to distinguish quality protection from exclusion of difficult cases. When eligibility definitions change, show the previous and revised definitions and their effective dates. The board can then see whether it is comparing different performance, different populations or both. An attractive percentage should not silently choose the population against which its own success is assessed.
Create a measure specification. State its name and decision question, numerator, denominator, period and source. Define treatment of returned, duplicate and pending requests, and identify the correction owner. Explain what happens when an end timestamp is missing or two systems disagree about registration date. Two organisations can use the same label while one counts attempts and the other counts people; that alone may change the conclusion. Keep the specification concise enough to use and precise enough to reproduce. Preserve why each revision occurred and its consequences for earlier comparisons. Its value appears when a second reader can calculate the rate and understand it in the same way without relying on the original analyst's memory.
Distinguish record closure from problem resolution. A closure target can encourage a status choice that ends the record while the user still needs another contact. Look for observable signals: repeat calls, more reopened requests, work passed to another team or disappearance of a group from accepted cases. A signal warrants examination; an accusation of deliberate gaming requires further evidence. Inadequate training or a system offering incomplete categories could explain the same pattern. Ask staff how they follow the instructions, then compare that account with actual request journeys. This supports correction of the mechanism rather than turning every discrepancy into a dispute with the delivery team. The inquiry should be capable of confirming genuine improvement as well as discovering displacement.
Before linking a measure to rewards, ask a reverse-incentive question. If staff wanted to improve the number quickly using the same resources, what would be the shortest route? They might prioritise easy cases, return incomplete applications or stop the clock at transfer instead of resolution. Some actions are legitimate and helpful; others improve presentation more than outcomes. Examine each route separately, tracing effects on users and receiving teams, then test whether companion measures would reveal it. This is a design exercise, not an advance allegation against future staff. People who understand the shortcut can help strengthen the definition before it becomes established practice or a dispute about performance pay. The discussion should result in a more informative measure rather than a presumption of misconduct.
Choose a specific protection measure for each material shortcut. If speed could undermine correctness, track rework caused by error. If easy cases might be favoured, compare completion by complexity using a classification defined beforehand. If handoffs stop the local clock, retain end-to-end journey time. Avoid a dashboard crowded with near-identical ratios: each measure needs a question, an interpretation owner and a decision that could change. Remove two measures answering the same question if one provides a more reliable answer. Keep a protection measure separate when it can reveal harm hidden by improvement in the main outcome. Its distinct purpose matters more than whether the display can be compressed into one overall score.
Examine reporting timing as well. Closing simple cases before month end while leaving difficult ones open can improve completed-work duration as the open workload ages. Show opening balance, arrivals, completions and closing balance, then examine ages of remaining requests, especially those approaching a critical limit. This reveals work that has not reached the closure measure. Next month's result can change when old cases close together. Explain the movement rather than penalising a period merely for clearing an inherited backlog. The aim is to understand service over time, distinguishing real performance change from the reporting boundary. A consistent cohort or an ageing view can supplement the flow measure when that question matters to a decision.
When combining units, make the aggregation rule explicit. An unweighted mean of branch rates gives a small branch the same influence as a large one, while summed numerators over summed denominators describe requests collectively. Both questions can be useful; choose according to the decision. Allocating support to reduce the number of delayed cases makes delay volume important. Examining fairness of service standards may require seeing every branch even where its volume is small. Connect the aggregate to its distribution and preserve exceptions needing a separate decision. This prevents a large branch taking all attention while a small service remains unacceptable beneath a reassuring average. An overall score should organise interpretation rather than remove consequential differences from view.
Treat voluntary satisfaction responses carefully. Respondents may be easier to reach or more motivated than nonrespondents. Retain numbers invited and responding, invitation method and intended population. Where authorised data permit, examine response differences by request type or service location. Responses inform understanding of experience within their population, with generalisation limits stated. Evaluation guidance explains the importance of representative survey samples; the numerical cases here remain original teaching designs. If the invitation channel changes, inspect channels separately before attributing the whole satisfaction movement to an initiative. A new channel can improve reach and still break comparability with an earlier series, requiring a transition explanation rather than suppression of either result.
Specify who corrects measurement and who examines the consequences. An analyst finding duplicates may repair counting, but excluding an entire group changes what the outcome means and warrants broader review. Record reason, previous and revised values and affected period. Then revisit decisions based on the earlier figure: does correction change expansion eligibility or resource allocation? Explain why a decision remains valid if it does; otherwise identify the authorised owner and corrective step. This makes error discovery useful rather than a personal admission to avoid. The board can reconstruct when its knowledge changed without erasing what was available at the earlier meeting. Correction authority should fit the actual consequence of the change, not merely the size of the spreadsheet edit.
Before adopting a new measure, test boundary cases. Select a returned application, a transferred request, a reopened case and work completed after reporting cut-off. Ask two analysts to calculate results using the same specification and compare disagreements. If clock starts are interpreted differently, clarify that definition and retest the affected case. This examines shared understanding of the measure; it is not a statistical effectiveness test. Monitor the first period because real registration may reveal an omitted pathway. A specification that works on clean examples still needs evidence that available data can implement it consistently at reasonable cost. Document the unresolved cases and the limited provisional use, if any, that the decision owner permits.
Tomorrow, choose one number used for an important decision. Trace a user's case across its start and finish, then identify where cost or need could disappear from the report. Propose a small definition repair or one protection measure if needed. Discuss it with the people recording the data and those using it to decide. This turns dashboard review into protection of the outcome the portfolio promised. The investigation may show that the present measure is suitable; retain it with a clear explanation and direct improvement effort elsewhere. A useful review changes understanding or action through evidence, rather than requiring another metric merely to demonstrate that a review took place.
Lesson summary
The lesson connects measure definitions to the behaviour they reward. It distinguishes accepted requests from incoming demand and corrections from new services. Quality limits, case tracking and backlog reconciliation support a review of apparent improvement before expansion is funded.