How to spot findings that probably won't replicate
When researchers tried to replicate 100 psychology studies published in top journals...
— Open Science Collaboration, 2015
Medicine isn't immune. Many "breakthrough" findings quietly disappear when larger studies are done.
The problem isn't fraud (usually). It's that certain statistical patterns are inherently fragile—they arise from noise, not signal.
A p-value of 0.048 is not the same as p = 0.001.
P-value distribution in "positive" studies
When you see p-values clustering just below 0.05 (like 0.048, 0.042, 0.038), be suspicious. This often indicates p-hacking—running multiple analyses until one "works."
The CI width tells you how much uncertainty remains.
Both report "significant reduction in pain scores"
Both exclude zero (statistically significant), but Study B's true effect could be anywhere from tiny to huge
Effect: 1.6 points
95% CI: 1.2 - 2.1
Narrow CI = confident estimate
Effect: 2.5 points
95% CI: 0.3 - 4.8
Wide CI = could be almost nothing
Small studies that find "significant" effects often massively overestimate the true effect size.
Imagine the true effect of a treatment is 0.3 standard deviations (small but real).
A small study (n=20 per group) needs to get lucky with sampling to reach p < 0.05. The studies that "win" (find significance) will overestimate the effect.
Reported effect sizes by study size
| Study Size | Reported Effect | True Effect | Inflation |
|---|---|---|---|
| n = 20 | 0.8 | 0.3 | 2.7x |
| n = 50 | 0.5 | 0.3 | 1.7x |
| n = 200 | 0.32 | 0.3 | 1.1x |
Counter-intuitively, small studies often show larger effects than big studies. This isn't because small studies are better—it's because only the ones that got lucky with their sample get published.
Be skeptical of any RCT with fewer than 50 participants per arm reporting a "significant" effect. The effect size is probably inflated.
When you test 20 outcomes, expect 1 false positive—by pure chance.
A supplement study measures 15 different outcomes: mood, energy, sleep quality, focus, memory, anxiety, stress, creativity, motivation, happiness, calmness, alertness, confidence, optimism, and satisfaction.
Result: "Significant improvement in optimism (p=0.04)!"
Probability of at least one false positive
| Tests Run | P(≥1 False Positive) |
|---|---|
| 1 | 5% |
| 5 | 23% |
| 10 | 40% |
| 20 | 64% |
Before trusting a result, run through these questions.
p = 0.04 is much more fragile than p = 0.001
Wide CI = uncertain estimate, even if "significant"
Under 50 per group? Effect is probably inflated
Cherry-picked findings from many comparisons don't replicate
"Works in women over 50 with diabetes" = fishing expedition
First findings are more often wrong than replicated ones
For binary outcomes, count how many patients switching from "event" to "no event" (or vice versa) would flip the p-value past 0.05. If the fragility index is 1-3, the result is extremely fragile.
For each study result, identify whether it smells fragile or robust.
P-values hovering just below 0.05 rarely replicate. Don't bet on p = 0.04.
A wide CI that excludes zero is less informative than a narrow CI. Look at the bounds, not just whether it "crossed the line."
The Winner's Curse means small studies that reach significance likely overestimate the true effect.
With multiple comparisons and no correction, false positives are expected, not surprising.
Initial findings are more likely to be wrong than replicated findings. Wait for confirmation before changing practice.