How a model learns to talk -- and why it sometimes lies with total confidence
You ask ChatGPT for a reference supporting carotid stenting in asymptomatic patients. It hands you a clean citation: authors, journal, year, volume, page numbers. It looks exactly like a real Journal of Vascular Surgery paper.
The paper does not exist. The model invented it -- not as a glitch, but as the normal product of how it works.
To trust an LLM appropriately, you have to understand one thing: it is not looking anything up. It is predicting the next word, over and over, very fast. Everything else follows from that.
A language model does not read letters or words. The first thing it does to any input is chop it into tokens -- chunks of text, roughly 3-4 characters each. A token is sometimes a whole word, sometimes a piece of one.
"Endovascular repair of a juxtarenal aneurysm"
Common words ("repair", "of", "a") are single tokens. Rare medical terms ("juxtarenal", "aneurysm") get split into pieces the model has seen before. This is why uncommon drug names and abbreviations are where models stumble most.
Everything the model knows is stored as relationships between tokens, not as facts in a database. There is no row that says "drug X interacts with drug Y." There is only a statistical sense that certain tokens tend to follow certain others. That is the entire substrate.
Given the tokens so far, the model produces a probability for every possible next token in its vocabulary. It picks one, appends it, and runs the whole thing again. An answer is just this loop, hundreds of times.
The model has no idea what "the right answer" is. It has a distribution. Most of the time the highest-probability token is sensible -- because sensible text is what it was trained on. But "most of the time" is not "always," and the model is equally fluent when it is wrong.
This is the key mental model: an LLM is a fluency engine, not a truth engine. It is optimized to produce text that sounds like what a knowledgeable person would write. Sounding right and being right are correlated -- but they are not the same thing, and the model cannot tell which one it is doing.
A model is not born knowing medicine. It is built in stages. Understanding the stages tells you exactly what it can and cannot be trusted to do.
The model is shown enormous amounts of text -- web pages, books, papers, code -- and trained on one task: predict the next token. It does this billions of times, adjusting internal numbers ("weights") until it gets good at it. This is where it absorbs grammar, facts, reasoning patterns, and yes, the structure of a citation.
This is why it can mimic a JVS reference perfectly -- it learned the shape of citations, not which ones are real.Raw pretrained models just continue text. Fine-tuning teaches the model to follow instructions and answer questions, using curated examples of good responses.
This is the difference between autocomplete and an assistant that answers your consult question.Humans rate responses; the model is nudged toward the preferred ones. This makes it more helpful and safer -- but it also trains it to sound confident and agreeable, which can mask uncertainty.
A model rewarded for confident, satisfying answers will rarely volunteer "I don't know" -- even when it should.The knowledge cutoff. Pretraining happens once, on data collected up to a fixed date. After that, the model is frozen. A model with an early-2025 cutoff has never seen the 2026 ESVS guideline update unless you paste it in. It does not know what it does not know -- it will answer about recent trials as confidently as old ones.
The word makes it sound like a malfunction. It is not. It is the model doing exactly what it always does -- predicting plausible tokens -- in a spot where plausible and true diverge.
Exact citations, p-values, dosing for rare drugs, trial enrollment numbers, guideline class levels. The model fills the gap with something that fits the pattern.
A fabricated citation is the canonical example: perfectly formatted, completely invented.
Asking it to make a clinical recommendation without the actual patient data. It will produce a fluent, generic answer that may not fit your patient.
Sounds tailored; is actually average-case.
Mechanisms, definitions, widely-taught management principles, restructuring text you provided. Seen thousands of times in training, so the high-probability answer is usually the correct one.
Still verify anything that drives a decision -- but this is where models are strongest.
Treat every specific, checkable claim -- a citation, a number, a dose -- as unverified until you confirm it from a primary source. The model's confidence carries zero information about whether it's right. Fluency is not evidence.
For each task, decide how much you should trust the model's raw output -- based on what you now know about how it works.
Module 2 - Lesson 1 complete
Now you know why it answers the way it does. Next: how a single question can give different answers each time -- and what "temperature" has to do with it.