Decision question and experiment protocol Review questions Decision question and experiment protocol — Integrated summary This module begins with a professional decision that evidence could change, rather than a wish to test a tool. Learners specify feasible alternatives, eligible units, allocation unit, outcome window, useful-effect threshold and harm guardrails before seeing results. They choose a concurrent comparator that answers the decision and distinguish the effect of a message from an added call or priority support. In fictional booking and learning cases, they calculate rate differences with the correct assigned denominators and see why opening or receiving a message changes the question. They build a risk register for contact permission, disclosure and withdrawal, redesigning when a safer comparison is needed. Finally, they freeze a dated protocol and change log, separating a documented measurement repair from a favorable outcome selected after results appear.
Review questions Test your understanding with these questions. Marking and saved results are available with a free account.
Question 1 Which question frames the decision? Can we send more messages? Will message opens rise if the color changes? Among eligible users, does the new reminder reduce missed visits versus current service within a fixed appointment window, subject to a complaint guardrail? Do staff prefer the new app? Question 2 Which denominator matches an assignment-policy effect? One hundred in each original assigned arm. Eighty new and one hundred current after dropping undelivered users. Message openers in each arm. Attendees alone. Question 3 What does the bundle test estimate? The shorter page alone. Extra staff help alone. Page color after controlling for help. The page-and-help bundle versus current service. Question 4 What feasibility action precedes assignment? Start small; sample size resolves permission. Hold sending and establish permitted use, objection route and minimum data before trial. Delete numbers only after the trial. Make message opening primary instead of attendance. Question 5 Which amendment preserves the protocol? Extend the window to ten days to increase successes. Delete every outcome from the fault week without examination. Log a deduplication repair, preserve raw records and retest refresh before analysis. Always select the second event because it is newer. Question 6 Which unit fits shared operating instructions? Assign requests individually and ignore information sharing. Assign workers but analyze thousands of requests as independent draws. Reassign each worker daily despite sharing with colleagues. Consider team assignment and base uncertainty on independent teams, not request count. Question 7 What protects allocation concealment? Fix eligibility first, then reveal a stored assignment under restricted access and audit log. Let staff mark a request ineligible when an unwanted arm appears. Publish the full sequence to make it transparent. Draw anew on every retry for the same ID. Question 8 How are block contrasts weighted? Use simple mean +6 regardless of target mix. 0.3×10+0.7×2=+4.4 points for the decision population. Add ten and two to get twelve points. Use morning effect alone because it is larger. Question 9 What comparison fits staged rollout? Compare week one with week three without contemporaneous controls. Treat requests within each team as independent draws. Compare started and not-yet-started teams in the same period, accounting for period and team. Infer effect from start count alone. Question 10 What is the factorial interaction? Interaction +20 because 60−40. Interaction +10 because that is A's standalone effect. No interaction because both factors help. Interaction +5 points because 60−50−45+40=5. Question 11 Which metric dictionary is valid? Count an assigned person successful once if a valid first event occurred within seven days; retain all raw events. Count every event because raw logs are trustworthy. Use page openers as denominator. Include late events if they improve the difference. Question 12 What is break-even? +2 points because 800/400=2. +20 points because value is twenty. +10 points because 800/20=40 successes and 40/400=10%. +40 points because forty completions are needed. Question 13 How does expected attrition change allocation? Assign 200 per arm because attrition comes after the draw. Assign ceil(200/0.8)=250 per arm, then check eligible flow. Assign 160 because 80% of 200 is sufficient. Remove the useful threshold to shrink sample. Question 14 What must be fixed before outcomes are opened? Choose the highest effect and register its method as prior. Drop unknown outcomes without description. Use an individual-person interval for a branch-randomized design. Fix transformations, denominator, missingness check, interval method and decision threshold; label later sensitivities. Question 15 How should an unplanned interim look be handled? Claim success immediately because 0.03<0.05. Log the look, do not treat it as a planned success stop, and assess its inferential implications. Erase the interim result from records and memory as if it never happened. Switch primary outcome to a more significant secondary. Question 16 Which assertion detects a display defect? Check only total draw counts. Open the link with a synthetic control ID and compare stored arm, visible page and exposure log. Ask an operator whether a bug is expected. Inspect later new-version success counts. Question 17 What is the valid assignment contrast? 55/80−50/100 because twenty were not reached. Reassign the twenty to current. 55/100−50/100=+5 points, reporting 80% reach separately. No contrast can be calculated without perfect compliance. Question 18 What does an unknown outcome mean? Classify them unknown and calculate bounds and sensitivity before scale judgment. Code failure because the event is absent. Code success to compensate for the fault. Delete them from the raw file. Question 19 What should happen to an ID in both arms? Choose the arm that increases estimated effect. Keep the ID in both arms to enlarge sample. Delete both raw rows without a trace. Pause inference for it, preserve originals, recover first eligible assignment independently or declare uncertainty. Question 20 How are harm containment and analysis separated? Leave it until the outcome window ends to protect sample size. Disable now, log deviation time and scope, then judge analytic effect without overwriting the original protocol. Delete complaints so they do not spoil the average effect. Switch primary outcome to page opening. Question 21 How do you read an interval above zero but below useful benefit? Launch because the entire interval is positive. Declare zero effect because five is unproved. The method supports positive direction, but lower end one does not establish useful five. Say there is a 95% probability the fixed effect lies in this realized interval without qualifications. Question 22 What is modeled net value with guardrail retained? One hundred extra successes worth 3,000, modeled net +500 subject to uncertainty. Net +3,000 because fixed added cost need not count. Net −2,500 because a rate difference can never be valued. Five hundred extra successes because two points means ten percent. Question 23 What condition limits effect transport? Weight eight and two only; missing capture is irrelevant. Assume every target site is busy to raise projection. Label new sites low demand without checking. Repair and verify comparable outcome measurement before using a weighted effect for target-site decisions. Question 24 Which decision follows the launch rule? Launch because effect clears the useful threshold. Do not launch: complaint gate fails despite effect passing; repair cause and verify. Move complaint cap to +1.5 after seeing data. Declare no completion effect because complaints are high. Question 25 Which link makes the dossier auditable? Add a methodology source logo instead of counts. Rely on manager signature because the number was discussed orally. Link claim to event dictionary, raw rows and transformations, arm table, interval and limits. Display three decimal places for the claim. Question 26 Which information deserves attention first? Views of the tool advertisement. Reasons for noncollection and the assumed value before funding the tool trial. The dashboard color. A precise tool effect in five cities before examining cost assumptions. Question 27 What decision value can this trial provide? Improve message wording within an already settled purchase, stating that purpose explicitly. Prove that the trial caused the tool purchase. Treat contract signing as the randomized outcome. Cancel all learning because purchase is settled. Question 28 Which date allows the evidence to be complete? End of day ten. End of day twelve. End of day fifteen. End of day seventeen. Question 29 How should information priority change? Fund a more precise financial difference alone. Choose reminders because their range is wider. Check support capacity because it may change implementation feasibility. Treat page superiority as proof of adequate support. Question 30 What should be repaired first in the success definition? Declare every electronic response ineffective. Keep sending as the only metric because it is easy to count. Separate sending, understanding and task completion, then choose an outcome matching the wording decision. Always count the second visit as another success. Question 31 Which interview prompt is least leading? What did you do after reading the message, and what did you need to find out? Didn’t you call because the technical word was bad? Do you agree removing the word will solve the problem for everyone? How much did our suggestion improve your experience? Question 32 What do five interviews establish? Every new user has the difficulty. Population prevalence equals its interview frequency. The new message causally solved the problem. An explanation worth examining, with prevalence and nonreturning-user experience still unresolved. Question 33 Which action improves consistency of the definition? Choose the criterion from aggregate results of the better version. Specify an action and window and test borderline cases before freezing the guide. Hide disagreement in an average rating. Retain full user identity even if unnecessary for classification. Question 34 Which conclusion is supported? Attendance rose ten points. Procedural readiness within tested cases, with attendance effect unknown. Positive annual benefit is established. Every possible operating condition passed. Question 35 Which question suits an operational pilot? Does the message increase attendance in every city? Does the sponsor prefer the tool? Can each tested appointment outcome be retrieved under the correct ID by the deadline? Does a small count prove both arms equivalent? Question 36 What progression decision follows this defect? Pause expansion, repair eligibility and replay both canceled and valid cases. Expand because it occurred only once. Remove the case from the report. Cancel the project criterion after seeing failure. Question 37 What is delivery among eligible bookings? 83.33%, or 30 of 36. 75%, or 30 of 40. 72%, or 36 of 50. 60%, or 30 of 50. Question 38 Who should act on this urgent incident? Wait regardless of continuing exposure. Pause within assigned authority, retain the time and notify the safety lead. Replace the primary outcome with complaints. Delete the incident log and resume sending. Question 39 What review does this change require? Treat it as spelling repair with no comparative consequences. Analyst masking alone is enough. Record package version and sponsor inspection, then review question and delivery implications. Erase the first exposure version. Question 40 Which approval does not establish calculation correctness? Sponsor signing alone proves the formula correct. Cost acceptance removes the need to state assumptions. All figures may be labeled independently audited. The signature is a scoped management decision; calculation review remains separate. Question 41 What establishes substitute readiness during absence? Have the substitute run the task with appropriate access and retain reviewable output. Only listing the name. Send the absent colleague’s password. Rely on reading an old file. Question 42 What is the allocation unit? Each request because it has an ID. Each measured minute. The system operating period, with requests measured inside it. The operator even though operators were not randomized. Question 43 What weakens rapid switching suitability? Learning persists despite technical rollback, requiring design review. Activation timestamps are retained. Unique request IDs are used. The queue is visible. Question 44 Why is always-new mornings versus always-current evenings insufficient? Means cannot be calculated in the morning. Equal period lengths remove every time effect. Request counts must be odd. Method is confounded with temporal demand; design must appropriately separate them. Question 45 What is the equally weighted paired contrast? Negative nine minutes. Negative three minutes. Positive three minutes. Negative one minute. Question 46 Which information can define pairs before allocation? Postlaunch usage because it is closest to intervention. Prior volume with consistent definition and period, reviewing outcome relevance. Satisfaction after viewing version results. The final contrast making pairs appear balanced. Question 47 Which pairing is closer on volume? 80 with 420; 95 with 400. 80 with 400; 95 with 420. 80 with 95; 400 with 420. Four sites cannot form pairs. Question 48 What can disqualify an apparently similar pair? Shared operation requires allocation-unit review or other sites. Similarity on two rates eliminates instruction transfer. Assign the preferred branch new without a draw. Delete rotating staff after seeing performance. Question 49 What does randomization do after matching? Prove equality on every unknown feature. Make each within-branch request an independent draw. Remove the need to define outcomes. Choose new within pairs without selecting on future outcomes. Question 50 What does the comparison measure? Count alone with every other element fixed. Wording alone. Differences between the described count-and-timing policies. Reading effects among readers automatically. Question 51 What does the third reminder add? Three points over two. Thirteen points over two. Ten points over two. Seventy-three points over two. Question 52 What follows for the highest intensity? Accept it because success is higher. Remove the complaint safeguard after results. Accept after rounding 5% to 3%. It fails the complaint condition despite descriptively higher success. Question 53 Which comparison is missing? Three versus two. A reminder policy directly versus no sending. Two versus one. Descriptive ranking of observed arm rates. Question 54 What should be checked first? Immediately add both reported effects. Map eligibility, assignment, exposure and shared resources before interpreting separate effects. Assume independence because teams differ. Delete shared users after observing success. Question 55 What do two absent cells imply? Four drawn boxes establish factorial design. Separate component effects need no assumptions. Available arms compare packages and do not support full separation as though all combinations were observed. Manually move half the users into missing cells. Question 56 What is the page effect with help? Five points. Twelve points. Fifteen points. Ten points. Question 57 What is point-scale interaction? Positive seventeen. Positive seven. Zero because both effects are positive. Negative seven, help effect contrast minus usual. Question 58 How is an open request recorded at cutoff? Zero completion time. Five observed days without an observed completion, with follow-up end reason. Actual completion on day five. Permanent failure for all later time. Question 59 What does the completer mean describe? All four have a two-day mean. Open requests necessarily failed permanently. Two days describes completers, not completion time for everyone. Everyone’s mean is zero because some times are unknown. Question 60 What is five-day completion? 60%, with the four later endings unknown. 100% because follow-up is complete. Zero because some remain open. 40%, the open share. Question 61 How does late completion affect an earlier window? Retroactively count success on day five. Remove it from the denominator. The five-day threshold is scientifically disproven. It remains unsuccessful within five days; day-six completion may inform another horizon. Question 62 Which outcome answers whether a person received the service? Successful visits divided by visits Mean duration of successful visits Completing people divided by all eligible assigned people Opened pages divided by people Question 63 What is the attempt success rate? Forty percent Sixty-five percent One hundred percent Fifty percent Question 64 How should an assigned person who never visits be handled? Exclude them for not seeing the interface Add a fictitious failed visit Reassign them every day Retain them in the assigned denominator as not completing Question 65 What is needed before treating visits as independent? Use the exported row count alone A method accounting for dependence within people Delete person linkage immediately after export Call every visit a new participant Question 66 Which variable is provisionally a baseline covariate? Requests in a month ending before randomization Help clicks after version exposure Satisfaction after completion Time spent in the new version Question 67 What is the adjusted value? Seventeen Eight Fourteen Eleven Question 68 What does a fractional adjusted count mean? Half an actual request was added An analytical value while the raw count remains ten All values must be rounded before comparison Necessarily an error because requests are integers Question 69 What is the adjusted contrast under this fixed formula? Three requests Zero One request Four requests Question 70 What is the contrast direction? Negative five points Positive five points Positive five percent relative Zero because both are high Question 71 Which margin rationale is appropriate? Whichever later makes the result pass Whichever minimizes sample size A published margin from any unrelated product Consequences and alternatives reviewed and approved in advance Question 72 Is noninferiority established? Yes under the rule, without establishing superiority No because the estimate might be negative Yes, proving exact identity No because zero is included Question 73 What happens when the interval touches the margin? Pass after rounding Pass because equality is preferable The strict criterion is not met Change the rule to include equality Question 74 Which classification fits the seven-day outcome? Exclude because receipt follows day seven Count completion on day nine Count by day-six occurrence after source checks Convert to permanent failure Question 75 What does a stable count for two hours establish? It does not establish daily-source completeness It proves no new events exist It automatically permits final status It means both arms are equivalent Question 76 Which message governs status? Latest arrival regardless of version First success because it favors the user Average completion and cancellation Authoritative version two with retained history Question 77 How should negative lag be checked? Clip values to zero Review clocks, time zones and field meanings while preserving originals Delete records without documentation Immediately declare deliberate manipulation Question 78 What does negative mean here? A bad user outcome A numerical value below zero A recommendation to cancel the trial A control role based on a causal assumption Question 79 Which candidate is weak as a negative outcome? Fixed prior activity through shared linkage Help openings during form use A synthetic message with a known expectation Identifier checks on a fixed historical file Question 80 What is needed beyond no treatment path? Relevant sensitivity to the target fault or disclosure of weak coverage Rename it an approved check A high p-value alone Treat it as a guarantee for every source Question 81 How should a historical difference between randomized arms be interpreted? Randomization failure is proven Treatment changed the past Review size, precision and linkage while allowing chance imbalance Delete the more active group Question 82 What comes first in adjudicating resolution? Define resolution, evidence and boundaries Learn which version the sponsor wants to win Set final agreement percentage in advance Always classify the longest reply as success Question 83 How is an absent reference classified? Correct because wording is confident Wrong because reference is missing Unverified under the insufficient-evidence category Remove the person from assignment records Question 84 How should masking be described? Complete masking of everyone Knowledge roles need no description Guaranteed independence because the name is removed Partial masking with reviewer knowledge stated Question 85 How are two judgments kept independent? Show the first decision before the second judges Record both initial decisions before discussion Overwrite originals with final consensus Let the version designer choose their judgments Question 86 Which field defines the assignment-based comparison? Last exposure instead of original assignment Current analyst name Original assigned arm with exposure retained separately Whichever version accompanied success Question 87 What establishes recoverability? An authorized open-and-reproduce rehearsal The name alone File size without contents The departing person says it once worked Question 88 How should backup-key custody transfer? Print it in a general receipt Send it with the file to every analyst Publish a shared password Use an authorized secure mechanism and test recovery Question 89 What does a matching assignment hash establish? Fairness of every eligibility decision Byte equality without proving fair randomization Effectiveness of the new version Every field is semantically correct Question 90 Which layer should be paused? Every organizational service automatically Efficacy conclusions relying on missing measurement while diagnosing source Original assignment by deleting it All earlier valid records Question 91 How is incident start recorded? Ten only because that is when noticed Nine without evidence Necessarily the last deployment Record established occurrence and detection times separately Question 92 Who can be affected outside the assignment-hour cohort? That person based on exposure history Only people assigned during the outage Nobody because original assignment is correct Only people who complain Question 93 Which test targets wrong-arm linkage? Open the home page Increase row counts Compare known cases with source and reject a deliberately swapped arm Read the repair filename Question 94 Which decision fits charter scope? Solve every citywide seat shortage Adopt for comparable cohorts, modify or retain current practice Prove employment improves for every applicant Change priority rights without approval Question 95 How is a priority-change proposal handled? Implement under the message label Hide it in the dictionary Call it purely technical Record it as a policy change needing separate authority Question 96 What limits confirmation-only evidence? It describes a process step rather than establishing attendance Attendance necessarily improved Skill learning is proven Workshop follow-up can be removed Question 97 Which mechanism assumption is exposed? Every message necessarily arrives Capacity doubled Reoffering may fail because list updates are too slow Eligibility policy automatically changed Question 98 How many unique people are represented? Seventy-four Eighty Seventy-five Sixty-nine Question 99 What capacity interference occurs? Proven linkage failure Complete independence A seat-mediated effect crosses arms New physical capacity Question 100 Which eligibility criterion precedes randomization? Responding to the treatment message Workshop prerequisites verified at cutoff under policy Later attendance Whichever version the applicant liked Question 101 What follows merging three cohorts? Necessarily ten independent units remain Seven units Thirty units Eight groups, with other links still reviewed Question 102 What should be shared across branches? Every action until no addition remains Better basic service only for treatment No assistance for comparison Correct information, assistance and rights under the same policy Question 103 How is actual treatment described? A specified reminder-and-early-release bundle Reminder wording alone Identical current practice A text effect isolated from every other change Question 104 How is an assistance request handled? An explicit decline releasing the seat A reason to switch arms Shared-rule assistance with a record A reason to remove the person from study Question 105 What is the original weekly task load? One hundred twenty minutes Two hundred ten minutes Three hundred minutes Ninety minutes Question 106 How many seats were used? Two One Zero because the first declined Three based on messages Question 107 What does the dictionary need? Only its name Numeric type alone Column color Definition, evidence, unit, source and missingness rule Question 108 What can joining attendance to two offers do? Double attendance if summed without respecting granularity Increase real attendance to two Require deleting the second valid offer always Create two independent people Question 109 How is an unclear signature interpreted? Ninety percent confirmed Reduce denominator to nine Eighty confirmed, possibly ninety after resolving the unknown A statistical confidence interval from eighty to ninety Question 110 What is the preliminary calculation unit? Every message as a person Raw request count Assignment group with its occupancy outcome Every nominal seat as an independent cluster Question 111 How many complete groups does the base approximation require? Seventy-two after rounding to thirty-six each Thirty-five total Seventy after rounding down Thirty-six for the whole study Question 112 What recruitment allowance is needed? Sixty-five Seventy-two unchanged One hundred forty-four Eighty, while missingness analysis remains necessary Question 113 What follows twenty rather than fifteen standard-deviation points? Required size falls because values are larger It rises to sixty-three complete groups per arm It must remain unchanged Power becomes one hundred percent Question 114 What is correct after replay handling? Nine people in ten seats Ten people because there are ten rows Eight after deleting both copies Only nine available seats Question 115 What is wrong with dropping the last row? Correct because the desired total matches Correct if nobody checks IDs Only the title needs clarification It reaches a total with the wrong set and leaves replay logic unfixed Question 116 How is conflicting content under one ID handled? Randomly remove one Apply source version/conflict rules Sum two attendances Choose whichever favors the arm Question 117 What is an independent expected result? Copy software output into expected values Write pass for every case Derive the result from the case by hand before comparison Choose the larger number Question 118 What is the base occupancy contrast? Twenty percentage points Ten percentage points Five percentage points One hundred fifty percent Question 119 What units inform uncertainty? Group outcomes under their design Every confirmation message independently Four hundred seats as independent randomizations Complaint count instead of attendance Question 120 Does the preliminary interval meet a four-point lower-bound rule? Yes because estimate is ten Yes because direction is positive Yes because seats are numerous No under this conditional reading, with final method still needing review Question 121 What follows discovery of unknown outcomes? The prior interval stays final unchanged Make every case present Reclassify and perform suitable inference on the revised version Remove them from denominator to complete the rate Question 122 What should lead the decision presentation? The history of every written message A short promise of victory Proposed action, conditions and evidence limits Acceptance without a receipt Question 123 How is a valid denominator objection handled? Correct it, trace dependencies and preserve the prior version Defend it because the title looks good Change the source to match arithmetic Restart everything regardless of impact Question 124 Which objection concerns values? The quotient is nine The source denominator is missing The report incorrectly labels minutes as hours Eighteen minutes per extra attendee is not worthwhile under organizational priorities Question 125 How are self-generated objections described? Completed independent review Useful adversarial self-review within its limits Official acceptance An external reviewer receipt Create a free account to mark your answers and save your progress.