Building Behavioral Evidence: Lab, Field, and Causal Reasoning

From observation to evidence that supports inference

Behavioral economics is not merely a collection of appealing ideas; its value depends on how those ideas are tested. We may observe a pattern in operational data, test a mechanism in a laboratory, or compare an intervention in the field. Each design can answer some questions and not others. Good observation can reveal where a problem occurs, but it does not automatically create a comparison group representing what would have happened without a change. A controlled experiment can strengthen causal inference, but it does not guarantee that the result will recur in every population, organization, or time period.

The central idea: what would have happened without the treatment?

A causal effect concerns the difference between two potential outcomes for the same unit: the outcome with treatment and the outcome that would have occurred without it. We usually cannot observe both at the same moment for the same person, so we need a credible comparison group. The World Bank’s impact-evaluation reference explains that random assignment, when appropriately implemented, helps create a comparison group expected to be similar to the treatment group, making post-intervention differences interpretable as treatment effects under the design’s conditions. This does not eliminate concerns about implementation, spillovers, attrition, or ethics.

Laboratory and field evidence serve different questions

A laboratory can be useful when the goal is to isolate a mechanism: does choice change when framing changes while material outcomes are held constant? Field evidence can be more suitable when asking whether an intervention works inside a real process with all its complexity. Do not treat “field” as a synonym for “better,” or “laboratory” as a synonym for “unrealistic.” Evidence strength depends on the question, measurement, assignment, compliance, sample size, analysis, and clarity about the population of inference. Sometimes the best research program combines several forms of evidence rather than relying on one experiment.

Replication and generalization: do not turn a project rate into a human universal

In Many Labs 2, 15 of 28 replication attempts produced a statistically significant effect in the same direction at p<.05. In a project replicating laboratory experiments in economics, 11 of 18 results replicated with statistical significance in the same direction, and replicated effects were smaller on average than the originals. These numbers describe particular selected sets of studies under particular procedures. Neither 15/28 nor 11/18 should be converted into “the replication rate of psychology” or “the reliability rate of economics.” Their instructional value is to remind us that evidence accumulates and that effect size, context, and design matter.

Internal validity asks whether the design supports attributing the observed difference to the treatment within the study. External validity asks how far we should expect the result to travel beyond the studied sample and conditions. Randomization can strengthen internal validity in context, but it does not automatically make a sample representative of the world. We can therefore have a strong causal experiment in a narrow group and still need further evidence before scaling to different populations, cultures, or channels.

Recap: evidence strength does not come from the word “experiment” alone. We need a suitable comparison and an inference that matches the design. Randomization can support causality but does not automatically solve generalization, and replication results belong to the study sets actually examined.