How to spot manipulation before you're fooled
Most published research findings are false. Not because scientists are dishonest, but because the incentives of academic publishing reward positive results, and there are many ways to turn negative data into a "significant" finding.
By the end of this lesson, you'll recognize the most common red flags in methods sections: p-hacking, HARKing, and cherry-picking. These are the tricks that make bad science look good.
P-hacking is running multiple analyses until you find p < 0.05, then reporting only that one.
If you test 20 hypotheses at p = 0.05, you'll get one "significant" result by chance alone. P-hackers exploit this by testing many things but only reporting the hits.
• Testing multiple outcomes, reporting only the significant one
• Adding or removing covariates until p drops below 0.05
• Excluding "outliers" until the result becomes significant
• Stopping data collection when p < 0.05
"After adjusting for multiple covariates..." (Which ones? Why those?)
"In a subgroup analysis..." (Was this planned?)
"After excluding outliers..." (How were outliers defined?)
If the primary endpoint failed but a convenient subgroup "worked," be very skeptical.
HARKing is presenting a post-hoc finding as if it were the original hypothesis.
Researcher designs study to test X. X fails. But they notice Y looks interesting in the data. Paper is written as if they always intended to study Y.
Exploratory findings need confirmation. When presented as confirmatory, they appear more reliable than they are. The false positive rate skyrockets.
"Our primary endpoint (wound healing at 12 weeks) showed no difference. In exploratory analysis, we noted a trend toward reduced infection rates that warrants further study."
"We hypothesized that the intervention would reduce infection rates. Our results confirm this hypothesis (p=0.03)."
• Check trial registrations (clinicaltrials.gov) for original endpoints
• Look for mismatch between stated aims and reported outcomes
• "Secondary endpoint" that gets more attention than the primary
• Very specific hypothesis that perfectly matches the result
If the hypothesis seems suspiciously perfect for the data, it was probably written after seeing the data.
Cherry-picking means reporting only the results that support your conclusion while hiding the rest.
Study measures 10 outcomes. Three show benefit, seven show no difference or harm. Paper reports only the three.
Data collected at 1, 3, 6, and 12 months. Only the 3-month data (where results looked best) makes it into the paper.
Discussion cites 10 studies supporting the conclusion, ignores 15 that contradict it.
• Methods list outcomes not reported in results
• Unusual or oddly specific time points
• Missing data on adverse events or harms
• Supplementary data that contradicts main findings
N=12 and claiming to prove a treatment works? Underpowered studies that find effects are likely false positives.
Suspiciously convenient. Real data rarely lands exactly on the threshold.
Without it, you can't verify the analysis plan wasn't changed after seeing data.
Not automatic disqualification, but raises the bar for scrutiny.
Inventor of device authors the pivotal trial. Consultant for company writes the favorable review.
No complications, no dropouts, no missing data. Real clinical research is messy.
Why not? What are you hiding?
"Death, MI, or need for bandage change" is real. Mixing severe and trivial events inflates rates.
The more a study's results align perfectly with the authors' interests, the more skeptical you should be.
Identify the methodological problem in each scenario.
You can now read a methods section and spot the tricks that make weak evidence look strong. Use this power wisely—most researchers aren't malicious, but incentives shape behavior. Your job is to evaluate the evidence, not the intentions.