Can it separate higher-risk from lower-risk patients?
Discrimination includes sensitivity, specificity, and AUC. AUC measures ranking—not accuracy, calibration, or clinical benefit.
Ask: What errors occur at the threshold we would actually use?
Does this AI model deserve your trust?
A high AUC and an impressive abstract are not enough. Use five clinical questions to decide whether a prediction model is informative, transportable, and safe enough to influence care.
Start with the claim
“Our model predicts major amputation with 97% accuracy.”
The outcome occurred in 3% of patients. A model that predicts “no amputation” for everyone is also 97% accurate.
Translation: Accuracy without class balance, a confusion matrix, and a clinically justified threshold tells you almost nothing.
The five questions
Ask these in order. A failure early in the chain weakens everything that follows.
Are its probabilities honest?
Calibration asks whether predicted risks match observed risks. If patients assigned 40% risk experience the outcome 20% of the time, the number is not decision-ready.
Ask: Show me the calibration plot, intercept, and slope—not only AUC.
Did it work on genuinely new patients?
Training performance is not evidence. Internal resampling helps; temporal or external validation tests whether the model survives a new setting, time, or population.
Ask: Was the final model evaluated on data untouched by model development?
Did information leak across the split?
Imputation, feature selection, normalization, duplicate patients, or future information can quietly expose the test set to training.
Ask: Were all preprocessing decisions learned inside the training data only?
Would using it improve a real decision here?
Transportability and usefulness depend on setting, workflow, threshold, harms, and available alternatives. Better prediction does not automatically produce better care.
Ask: What action changes, for whom, and is there evidence of net benefit?
Audit the paper in front of you
Answer all five. “Unclear” counts against the paper because incomplete reporting prevents appraisal.
Standards behind the framework
- TRIPOD+AI for transparent reporting of prediction-model studies.
- PROBAST+AI for quality, risk-of-bias, and applicability assessment.
- CONSORT-AI for trials evaluating AI interventions.