Why the same question gives different answers -- and what you can control
You ask the same clinical question twice and get two different answers. A colleague asks it and gets a third. If this were UpToDate, you'd call it a bug. With an LLM, it is working as designed.
Understanding why -- and the few settings that govern it -- is the difference between using these tools deliberately and being surprised by them.
Last lesson: the model predicts a distribution over next tokens. This lesson: how it picks from that distribution, how much text it can hold in mind, and what "reasoning" models actually do differently.
When the model has a probability for each next token, it does not always take the most likely one. It samples -- rolls weighted dice. A setting called temperature controls how adventurous that roll is. Drag it.
At low temperature the model almost always picks the highest-probability token, so answers are consistent and conservative. At high temperature it takes more chances -- more varied, more creative, more likely to drift into error. Most consumer chat tools run at a moderate default you can't see, which is why you get variation you didn't ask for. For clinical fact-finding, you want low temperature behavior: ask for the conservative, standard answer.
This is also why "regenerate" can give a better answer -- you're re-rolling the dice. Useful for brainstorming. A warning sign for facts: if the answer changes meaningfully each time you ask, the model does not actually know, and is sampling from uncertainty.
The model can only "see" a limited amount of text at once -- your prompt, anything you pasted, and the conversation so far. This budget is the context window, measured in tokens. Everything it uses to answer must fit inside it.
The model attends to all of this at once to produce the next token. Two consequences follow.
Anything in the window -- a guideline, a note, a paper -- the model can use directly, even if it was published after the training cutoff. This is the workaround for the frozen-knowledge problem from Lesson 1.
This is the entire trick behind retrieval-based tools, which we cover next lesson.
In a long conversation, the earliest messages can slide out of the window. The model doesn't remember them -- it never saw them for this turn. Details from 40 messages ago may simply be gone.
Long threads quietly lose their own beginning. If something earlier matters, restate it.
It does not actually remember you between sessions. Unless a tool has an explicit memory feature, each new chat starts blank. The sense of continuity inside one conversation is just the transcript riding along in the context window -- not a model that "knows" you.
Newer models (often labeled "reasoning," "thinking," or with names like o-series) do something extra: before giving you a final answer, they generate a long internal scratchpad -- working through the problem step by step -- and only then commit.
Why it helps: each step of "thinking" becomes part of the context the next step attends to. The model is, in effect, giving itself more to work with before it commits -- the same reason that asking any model to "show your reasoning step by step" improves hard answers.
For a quick definition or to clean up text, a standard model is fine. For anything with multiple logical steps -- weighing competing risks, working through eligibility criteria, checking a calculation -- reach for a reasoning model, and ask it to show its work so you can audit the steps.
One caveat: a longer chain of reasoning is more convincing, not necessarily more correct. The model can reason fluently toward a wrong conclusion. The visible steps are useful because you can check them -- not because the model guarantees them.
Each scenario describes how you're using a model. Pick the explanation -- or the fix -- that fits what you now know about sampling, context, and reasoning.
Module 2 - Lesson 2 complete
You now know how a bare model answers. Next: RAG vs. frontier reasoning -- why OpenEvidence behaves differently from ChatGPT, and what a 2026 study found when it pitted them against each other.