* Total Points: 0
Back to Lessons Tokens and Training 0 pts Module 2 · Lesson 1
Introduction

Tokens and Training

How a model learns to talk -- and why it sometimes lies with total confidence

The Problem

You ask ChatGPT for a reference supporting carotid stenting in asymptomatic patients. It hands you a clean citation: authors, journal, year, volume, page numbers. It looks exactly like a real Journal of Vascular Surgery paper.

The paper does not exist. The model invented it -- not as a glitch, but as the normal product of how it works.

To trust an LLM appropriately, you have to understand one thing: it is not looking anything up. It is predicting the next word, over and over, very fast. Everything else follows from that.

Step 1: Text Becomes Tokens

A language model does not read letters or words. The first thing it does to any input is chop it into tokens -- chunks of text, roughly 3-4 characters each. A token is sometimes a whole word, sometimes a piece of one.

Your question, as the model sees it

"Endovascular repair of a juxtarenal aneurysm"

Endo vascular  repair  of  a  jux tar enal  aneur ysm

Common words ("repair", "of", "a") are single tokens. Rare medical terms ("juxtarenal", "aneurysm") get split into pieces the model has seen before. This is why uncommon drug names and abbreviations are where models stumble most.

!

Why this matters clinically

Everything the model knows is stored as relationships between tokens, not as facts in a database. There is no row that says "drug X interacts with drug Y." There is only a statistical sense that certain tokens tend to follow certain others. That is the entire substrate.

Step 2: Predict the Next Token. Repeat.

Given the tokens so far, the model produces a probability for every possible next token in its vocabulary. It picks one, appends it, and runs the whole thing again. An answer is just this loop, hundreds of times.

"The first-line treatment for an acutely ischemic limb is "
heparin
71%
systemic
14%
urgent
9%
aspirin
4%
amputation
2%

The model has no idea what "the right answer" is. It has a distribution. Most of the time the highest-probability token is sensible -- because sensible text is what it was trained on. But "most of the time" is not "always," and the model is equally fluent when it is wrong.

This is the key mental model: an LLM is a fluency engine, not a truth engine. It is optimized to produce text that sounds like what a knowledgeable person would write. Sounding right and being right are correlated -- but they are not the same thing, and the model cannot tell which one it is doing.

Step 3: Where the Knowledge Comes From

A model is not born knowing medicine. It is built in stages. Understanding the stages tells you exactly what it can and cannot be trusted to do.

1

Pretraining: read the internet

The model is shown enormous amounts of text -- web pages, books, papers, code -- and trained on one task: predict the next token. It does this billions of times, adjusting internal numbers ("weights") until it gets good at it. This is where it absorbs grammar, facts, reasoning patterns, and yes, the structure of a citation.

This is why it can mimic a JVS reference perfectly -- it learned the shape of citations, not which ones are real.
2

Fine-tuning: learn to be helpful

Raw pretrained models just continue text. Fine-tuning teaches the model to follow instructions and answer questions, using curated examples of good responses.

This is the difference between autocomplete and an assistant that answers your consult question.
3

Alignment (RLHF): learn what humans prefer

Humans rate responses; the model is nudged toward the preferred ones. This makes it more helpful and safer -- but it also trains it to sound confident and agreeable, which can mask uncertainty.

A model rewarded for confident, satisfying answers will rarely volunteer "I don't know" -- even when it should.

The knowledge cutoff. Pretraining happens once, on data collected up to a fixed date. After that, the model is frozen. A model with an early-2025 cutoff has never seen the 2026 ESVS guideline update unless you paste it in. It does not know what it does not know -- it will answer about recent trials as confidently as old ones.

So What Is a "Hallucination," Really?

The word makes it sound like a malfunction. It is not. It is the model doing exactly what it always does -- predicting plausible tokens -- in a spot where plausible and true diverge.

Highest risk

Specific facts it was never reliably taught

Exact citations, p-values, dosing for rare drugs, trial enrollment numbers, guideline class levels. The model fills the gap with something that fits the pattern.

A fabricated citation is the canonical example: perfectly formatted, completely invented.

Moderate risk

Reasoning over information you didn't give it

Asking it to make a clinical recommendation without the actual patient data. It will produce a fluent, generic answer that may not fit your patient.

Sounds tailored; is actually average-case.

Lowest risk

Common, well-represented knowledge

Mechanisms, definitions, widely-taught management principles, restructuring text you provided. Seen thousands of times in training, so the high-probability answer is usually the correct one.

Still verify anything that drives a decision -- but this is where models are strongest.

1

The one habit that protects you

Treat every specific, checkable claim -- a citation, a number, a dose -- as unverified until you confirm it from a primary source. The model's confidence carries zero information about whether it's right. Fluency is not evidence.

Exercise: Will It Hallucinate?

For each task, decide how much you should trust the model's raw output -- based on what you now know about how it works.

Scenario 1 of 6

Lesson Complete!

0
Total Points Earned
Scenarios (0/6 correct) +0 pts
Lesson Completed +100 pts

Tokens and Training

Module 2 - Lesson 1 complete

Key Takeaways

  • It predicts tokens, it doesn't look things up: an LLM generates one chunk of text at a time, choosing from a probability distribution. There is no database inside.
  • Fluency engine, not truth engine: the model is optimized to sound like a knowledgeable person. Sounding right and being right overlap, but are not identical.
  • Knowledge comes from pretraining: it absorbed facts and patterns by reading text up to a fixed cutoff date. After that, it is frozen and unaware of newer evidence.
  • Alignment makes it confident, not honest: it is trained to give satisfying answers, so it rarely admits uncertainty on its own.
  • Hallucination is normal behavior: fabricated citations and numbers are the model filling a gap with something plausible -- not a bug you can switch off.
  • Verify every checkable specific: citations, doses, p-values, and stats are highest-risk. Confidence tells you nothing about accuracy.

Now you know why it answers the way it does. Next: how a single question can give different answers each time -- and what "temperature" has to do with it.