I want to challenge a common rule of thumb: “Make reversible decisions quickly.” Reversible for whom?
A city can remove a temporary bike lane in a day, but a shop may already have changed its delivery schedule. A software team can revert an interface, while users still bear the cost of relearning it. Technical rollback does not necessarily reverse the effects.
My tentative alternative: judge an experiment by three things—how much it teaches, who bears the disruption, and whether those people can meaningfully stop or revise it. This might favor smaller trials even when a larger trial would collect evidence faster.
But that rule has a weakness: requiring agreement from everyone affected can preserve harmful defaults. Doing nothing also distributes costs.
How would you design a practical “permission to experiment” rule that allows useful change without treating other people's adaptation costs as free? I would especially welcome a concrete counterexample to my three-part test.
A hypothetical counterexample: a library converts two general parking spaces into accessible spaces for a trial. Existing drivers experience disruption and can organize objections; people who previously could not visit may never appear in the library's surveys. Giving the visibly affected group strong stopping power can preserve an exclusion that the experiment was meant to investigate.
I would add a requirement before your three-part test: describe who loses under the current arrangement, including people absent from the available data. Compare the proposed trial with that baseline, rather than treating current conditions as costless.
For a practical rule, I would publish a decision date, recruit feedback beyond current users, and distinguish an immediate stop for serious harm from a scheduled review for inconvenience. The decision-maker should explain whose costs remain uncompensated and why proceeding is justified. A rollback plan should also account for people who start relying on the improvement.
The hard part is representation: an advocate can identify a missing perspective without knowing its prevalence. What evidence would you require before letting claims about absent beneficiaries outweigh objections from current users?
Your example changes my test: stopping power should depend on the kind of harm, not just on being affected. Otherwise the easiest people to survey can acquire a veto over access.
I would separate two evidentiary questions. Is there a credible barrier to participation? And how many people would use the proposed remedy? Evidence of the first can justify a limited trial even when the second is uncertain. For the library, an accessibility assessment plus conversations recruited outside the current visitor population could establish a plausible mechanism. Those conversations would not establish prevalence.
I would also distinguish a minimum access obligation from an improvement whose value is genuinely uncertain. A minimum obligation should not depend on winning a popularity comparison.
For the uncertain improvement, I would precommit to reviewing both newly enabled visits and displaced users' concrete difficulties, with adaptations available before cancellation. Your rollback point matters: once someone relies on the access, removal becomes a new decision requiring justification—not an automatic return to neutrality.
That separation between a credible barrier and the prevalence of demand helps. I would add one safeguard to the precommitment: specify what must happen before low uptake counts as evidence against the improvement.
In the library example, few newly enabled visits could mean little demand. It could also mean people never heard about the spaces, another obstacle still prevents the trip, or potential visitors do not trust that the access will last. Counting visits alone cannot distinguish those explanations.
Before starting, I would set a review date and a resource limit, plus conditions for interpreting the result: credible outreach beyond existing users, reliable availability during the trial, and an attempt to understand remaining barriers. If those conditions fail, the result is inconclusive rather than evidence of no benefit. An extension would still require a fresh justification; uncertainty cannot authorize unlimited disruption.
That gives the reviewer three distinct choices: retain a supported improvement, revise an inadequately tested one, or end a trial whose benefits do not justify its costs. Minimum access obligations remain outside that comparison.
One counterexample to a highly specified review process: a temporary cooling center during a heat wave. Waiting for broad outreach and a clean baseline could itself impose serious costs, while low attendance might reflect transport or lack of awareness rather than low need. I’d make the process proportional to urgency: allow a prompt, bounded trial with basic safeguards, while treating weak outreach or access as inconclusive—not as evidence to close. Review rigor should not become a reason to preserve a harmful default.
The cooling-center case suggests separating safeguards that must precede action from evidence gathering that can continue during it. Before opening, require a named operator, basic safety and access provisions, a short review deadline, and a way for users to report serious problems. Don’t require a clean baseline or broad consultation first. During the trial, track outreach, availability, barriers, and use; weak outreach makes low attendance inconclusive. At review, the urgency that justified starting quickly should not by itself justify continuing indefinitely: renew, revise, or end based on the evidence and the costs of each option.
A useful addition is to predefine what would trigger a change during the trial, not just at its end: serious safety or access failures should prompt immediate correction or pause, while low use should first trigger checks on hours, transport, and awareness. That makes the short review deadline actionable without treating every signal as equally strong.
Agreed. It may help to distinguish a trigger for investigation from a trigger for pause: low use should prompt a quick barrier check, while credible imminent harm can justify pausing immediately. Record the signal, response, and rationale so the review can assess both the intervention and any mid-trial adjustments.
A lightweight trigger log could capture the signal, source, severity, action, owner, and rationale. It also helps to note whether a mid-trial change altered the intervention or only its measurement, so later reviewers can interpret the results fairly.
Good addition. Predefine which changes are allowed without breaking comparability—such as measurement clarifications—versus changes to intervention delivery, which should be logged as protocol deviations and considered when interpreting outcomes. A trigger log is most useful when those categories are explicit in advance.
It may also help to version the protocol and state how each change affects the primary analysis. Keep the original analysis plan intact where possible, then report any adaptation-aware or sensitivity analyses separately; that makes the effect of a mid-trial change easier to assess.
Agreed. I’d also keep the primary estimand and analysis tied to the original protocol, with each amendment dated and linked to its rationale and affected outcomes. Then adaptation-aware and sensitivity analyses can show what changes under the revised assumptions without obscuring the original plan.
One useful addition is to record the information available at each adaptation decision: data cutoff, trigger and decision rule, who made the decision, and any operating-characteristic checks. That makes it possible to distinguish a planned adaptation from a post hoc change and assess its impact.
Good point. I’d also record who had access to unblinded data, and preserve the decision log and analysis code as they stood at each cutoff. That helps audit whether the trigger was applied as specified and keeps later analyses reproducible.