Why your 15-variable model is garbage
You have a dataset. You have a research question. And you have a list of every variable you collected: age, sex, BMI, diabetes, smoking, eGFR, ejection fraction, ASA class, frailty index, prior surgery, insurance status. Why not put them all in the model and let the regression sort out what matters?
Because the model does not know the difference between signal and noise. Give it enough freedom and it will happily fit the random quirks of your particular 200 patients -- quirks that will not exist in the next 200. That is overfitting: the model memorizes the noise in your sample and then fails on new patients.
Overfitting is the single most common modeling sin in the surgical literature. The fix is not a fancier algorithm -- it is discipline: fewer predictors, chosen in advance, and honest validation before you believe a single number the model reports.
The first sanity check for any logistic or Cox model: do you have enough events to support the number of predictors?
For logistic regression and Cox regression, the quantity that limits model complexity is the number of events -- the rarer of the two outcomes -- not the total sample size. A study of 1,000 patients with only 20 deaths has the statistical power of a study with 20 events, no matter how impressive the headline N looks.
The traditional rule of thumb: at least 10 events per candidate predictor (EPV ≥ 10). Below this, coefficients become unstable, confidence intervals are unreliable, and apparent performance is wildly optimistic.
"Number of predictors" really means degrees of freedom estimated, and some predictors cost more than one:
Continuous (linear): 1 degree of freedom.
Categorical with k levels: k − 1 degrees of freedom. A 4-level ASA class costs 3.
Spline / non-linear term: several degrees of freedom (often 3-4) for a single variable.
Interaction term: an additional degree of freedom (more if either term is categorical).
Count degrees of freedom, not the number of lines in your variable list.
| Study | Patients | Events | Predictors (df) | EPV |
|---|---|---|---|---|
| Overfit example | 200 | 30 | 12 | 2.5 |
| Max defensible | 200 | 30 | 3 | 10.0 |
| Comfortable | 200 | 30 | 2 | 15.0 |
For linear regression, the analog uses the total sample size: roughly 10-15 observations per predictor. With 60 patients you can defensibly fit about 4-6 predictors in a linear model, not because of events (there are none) but because each parameter still needs data to estimate it stably.
How you pick variables and how you measure performance determine whether the model means anything.
Stepwise selection (forward, backward, or "screen univariably at p<0.20, then enter the survivors") lets the data choose the model. This is seductive and wrong.
It inflates type-I error (you test the same data many times), produces unstable variable sets (a few resampled patients change which variables "make it"), biases coefficients away from zero, and yields optimistic performance estimates. The resulting "independent predictors" are partly artifacts of your sample.
Specify the predictors before looking at outcomes, based on clinical knowledge and prior literature. Include known confounders regardless of their univariable p-value.
A pre-specified model that respects the EPV budget is more reproducible and more honest than any data-driven search. If you must reduce variables, do it on clinical grounds or with penalization, not p-values.
A principled alternative to stepwise selection. Penalized methods shrink coefficients toward zero to control overfitting.
LASSO can shrink some coefficients exactly to zero (a built-in, less unstable form of selection); ridge shrinks all coefficients but keeps every variable. Both trade a little bias for a large reduction in variance, which usually improves performance on new patients. The penalty strength is tuned by cross-validation.
Performance measured on the same data used to build the model -- the apparent performance -- is systematically too good. The model was fit to those exact patients, so it scores itself on its own homework.
A C-statistic of 0.89 or an R² of 0.7 on the training data proves nothing about new patients. You must estimate optimism using internal validation (bootstrap or cross-validation) or, better, external validation on a separate cohort. Bootstrap-based shrinkage and recalibration pull the optimistic estimates back toward reality before you report them.
A high C-statistic or R² on the training sample is the most over-reported, least meaningful number in prediction papers. Always ask: optimism-corrected or externally validated? If neither, the number is apparent performance and should be treated with suspicion.
Mistakes that produce publishable but unreproducible models.
"Our multivariable model adjusted for 15 covariates." The study had 30 events.
EPV = 30 / 15 = 2.0. This is not multivariable adjustment -- it is noise fitting. The coefficients are unstable, the confidence intervals are misleadingly narrow, and the "adjusted" estimates will not replicate. The maximum defensible number of predictors here is 3 (events / 10).
"We used stepwise selection to identify the independent predictors of complication."
Stepwise selection does not identify independent predictors. It identifies the variables that happened to look good in this sample. The procedure inflates type-I error, biases coefficients, and is unstable -- resample the data and a different set of "independent predictors" emerges. Pre-specified, clinically chosen variables or penalized regression are the defensible alternatives.
"The model showed excellent discrimination (AUC 0.84)." No validation is mentioned.
That is apparent AUC. Reported without internal or external validation, it is the model grading itself on the data it was built from -- guaranteed to be optimistic. The optimism-corrected AUC (via bootstrap) is often substantially lower. Without it, 0.84 means very little.
"We built a prediction model and confirmed its accuracy on the same dataset."
Building and validating on the same data is not validation. It cannot detect overfitting because the model has already seen every patient. Honest options: split-sample (weaker), cross-validation, bootstrap optimism correction, or -- the gold standard -- an external cohort the model never touched during development.
Dichotomizing a continuous predictor "to simplify" (e.g., age > 65 vs ≤ 65) throws away information and statistical power. A 64-year-old and a 90-year-old become identical; the model loses the gradient of risk. It rarely simplifies anything real and usually weakens the model. Keep continuous variables continuous -- and model non-linearity with splines if needed.
Evaluate model complexity, selection, and validation.
Module 5 - Lesson 6 complete
You have completed Module 5: Regression Without Crying. You can now build, critique, and defend regression models -- choosing the right model, interpreting coefficients honestly, and recognizing overfitting before it embarrasses you in peer review.