Triangulate a diagnostic baseline
6 min read
Lesson objectives
- Define comparable cohorts, periods and causal alternatives; case: organic decline after tracking change.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
A baseline is a disciplined description of what happened before an intervention. “Organic visits fell 20%” means little without the period, visit definition, page scope, seasonality, and measurement changes. A fictional device-repair company sees search-labeled visits decline after launching a new site. The marketing manager calls it a search-engine penalty; the developer says only the analytics tag changed. Separate three series: impressions and clicks in search results, on-site sessions labeled as search, and confirmed service bookings. Their denominators differ, and their directions need not match. Do not combine them on one chart to infer a cause.
Construct the baseline from comparable units. Select weeks with similar demand patterns, and distinguish service pages from articles, phones from desktops, markets, and branded from general queries when data permit. Check availability, prices, and opening hours. Comparing a launch week with a holiday can attribute a seasonal shift to the site. Combining a high-traffic article with a low-traffic booking page can produce a growing visit total while valuable demand declines. For each segment, record numerator, denominator, and time window; show the total as a summary rather than a substitute for analysis.
Distinguish a measurement effect from a behavioral effect. If a new analytics implementation went live on the day sessions “collapsed,” compare server records, search visibility data, and completed bookings. Relatively steady search clicks and bookings alongside falling labeled sessions suggest a tracking problem, but do not prove one; session definitions or consent behavior may differ. Review the change log, test the live page with the team's permission, and check whether events fire once and whether particular pages omit them. If clicks also fall, examine pages, queries, and indexing reports before declaring the instrument innocent.
Use a competing-hypotheses table. A tagging fault predicts a larger gap between search clicks and sessions and comparatively stable bookings. A less relevant result predicts continuing impressions but lower click rates on identifiable queries. A poorer site experience predicts arrivals who leave before booking, concentrated on a device or page. Seasonality predicts similar movement in segments unaffected by the launch or a suitable comparison period. For each hypothesis, write an observation supporting it, an observation against it, the next test, and the decision that would change. This turns a disagreement between teams into a test plan rather than a vote on a favorite story.
Watch for aggregation paradoxes. Booking rate can rise on both phone and desktop while the overall rate falls if the lower-converting phone share rises sharply. Display segment sizes alongside rates. Avoid choosing a start date because it produces the largest fall; set the primary window before interpretation and test a declared second window for sensitivity. With small booking counts, random variation can dominate a difference of a few orders. A consequential causal claim needs a suitable comparison group or experiment, not a dramatic line chart.
A small fictional calculation makes the paradox visible. Before launch, 40 of 400 desktop visitors and 10 of 200 phone visitors booked: 50 of 600 overall, about 8.3%. After launch, 22 of 200 desktop visitors and 36 of 600 phone visitors booked. Desktop improved from 10% to 11% and phone from 5% to 6%, but the combined rate became 58 of 800, or 7.25%. The device mix changed. These figures are an arithmetic demonstration, not company records. Also check whether new phone traffic has weaker purchase intent; that points to a targeting decision rather than a checkout decision.
In the repair-site case, prioritize work by the cost of being wrong. First audit session tags and consent handling; second, analyze queries and pages with a lower click rate; third, test the phone booking journey. Do not rebuild the entire site or double advertising before those checks. Assign an owner and deadline to each check and a decision rule: if the session decline is measurement-only, repair the report; if qualified clicks and bookings fall on specific service pages, test a better snippet and page; if the cause remains uncertain, keep the recommendation provisional and define the evidence needed next. This sequence protects the client from spending heavily on the first plausible explanation.
Lesson summary
A baseline is a disciplined description of what happened before an intervention. “Organic visits fell 20%” means little without the period, visit defi