Not all evidence is created equal
Someone cites a study to support their argument. Before you even look at the p-value, you need to ask: what kind of study is this?
A well-done RCT and a case series are not the same level of evidence. This lesson teaches you to recognize study designs instantly and know what each can and cannot tell you.
The design of a study determines the ceiling on how confident you can be in its conclusions.
No amount of statistical sophistication can overcome fundamental design limitations.
From strongest to weakest, these are the main study designs you'll encounter.
Random assignment to treatment vs control. The gold standard for establishing causation.
Limitation: Expensive, not always ethical or feasible
Follow groups forward in time based on exposure. Can establish temporal sequence.
Limitation: No randomization, confounding possible
Look backward at existing data. Faster and cheaper than prospective.
Limitation: Limited to available variables, prone to bias
Start with outcome (cases), find matched controls, look back at exposures. Good for rare outcomes.
Limitation: Recall bias, selection of controls is tricky
Snapshot in time. Exposure and outcome measured simultaneously.
Limitation: Cannot establish causation or temporal sequence
Description of outcomes in a group or single patient. Hypothesis-generating only.
Limitation: No comparison group, no control for anything
The design determines what conclusions are possible. An RCT can prove causation. A case series can only suggest it.
The same association found in different study designs means different things.
Always ask: Could there be another explanation the study design can't rule out?
In observational studies, the patients who get a treatment are often systematically different from those who don't.
Example: Patients prescribed statins may also be more likely to exercise, eat well, and see their doctor regularly. Is it the statin, or the lifestyle?
Some terms are designed to obscure the actual study design. Here's how to see through them.
"What about 'prospective database' studies?"
Usually retrospective cohorts using prospectively collected data. The data were collected going forward, but you're analyzing them looking backward. Still not an RCT.
"What about 'real-world evidence'?"
Marketing term. Usually means observational data. Sounds impressive but tells you nothing about the design. Always ask: what is the actual study design?
"What about meta-analyses and systematic reviews?"
These synthesize existing studies. Their quality depends entirely on the underlying study designs. A meta-analysis of garbage is still garbage.
"What about registry studies?"
Large retrospective cohorts. Big N doesn't fix confounding. A million patients with unmeasured confounders is still biased.
When someone uses fancy terminology, translate it back to basics: Is this an RCT or not? If not, what are the potential biases?
Read each description and classify the study design.
Module 1 - Lesson 1 complete
Before trusting any study's conclusions, identify the design. It tells you the ceiling on how confident you can be.