Why OpenEvidence behaves differently from ChatGPT -- and what a 2026 study found
You have OpenEvidence on your phone. A resident uses ChatGPT. Both answer clinical questions. Both sound authoritative. Are they the same kind of thing? Is one safer?
They are built on fundamentally different strategies. Knowing which is which tells you what each is good for -- and which one to trust for what.
Two strategies: retrieve-then-generate (RAG) and reason-from-knowledge (frontier models). OpenEvidence is mostly the first. ChatGPT, Claude, and Gemini are the second. This lesson is the difference.
Both start with your question. What happens next is completely different.
Remember from Lesson 2: anything in the context window, the model can use -- even content newer than its training cutoff. RAG automates that. It finds the right documents and pastes them in for you, then the model answers grounded in them. That's the whole mechanism. "RAG" = Retrieval-Augmented Generation = search + paste + answer.
Neither is strictly better. They fail in different places -- and the failure modes are the whole point.
Because it answers from retrieved documents, RAG can show you the source, stay current as its library updates, and let you check the original. This is why OpenEvidence feels trustworthy -- it hands you references.
Best when you need a sourced answer you can verify and cite.
If the search pulls the wrong passages -- or misses the best one -- the model answers from poor material. Garbage in, confident garbage out. The retrieval step is a hidden point of failure you can't see.
A citation proves a source exists, not that it was the right source or was used well.
A strong frontier model integrates across vast knowledge, handles messy multi-part questions, weighs competing considerations, and communicates well. It reasons rather than just relaying a passage.
Best for complex, open-ended thinking -- and increasingly competitive on raw medical knowledge.
Without retrieval or web access, it answers from frozen training data and may invent citations (Lesson 1). It can be brilliant and unsourced at the same time.
Powerful reasoning, but you supply the verification.
The clean way to hold it: RAG optimizes for "show me the source." Frontier models optimize for "reason it through." The best clinical setups increasingly combine both -- a strong reasoning model with retrieval attached.
The intuition was that purpose-built clinical RAG tools (OpenEvidence, UpToDate Expert AI) would beat general chatbots on medicine. A 2026 Nature Medicine study tested it head-to-head. The result went the other way.
Two clinical AI tools (OpenEvidence, UpToDate Expert AI) vs. three frontier models (GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6), across medical-knowledge questions, alignment with clinicians, and 100 real de-identified clinical queries.
Schematic of the reported pattern, not exact scores. Frontier models led on completeness, communication quality, and context awareness across all three evaluations.
The authors point to RAG's weak link: retrieval can hurt you if the wrong material is pulled or if the underlying model integrates it poorly. A RAG tool is capped by two things -- the quality of its base model and the quality of its search. Meanwhile frontier base models have improved so fast that reasoning from their own knowledge now often beats relaying a retrieved passage. The specialized wrapper stopped being an advantage.
Don't over-read this. "Frontier models scored higher on benchmarks" is not "trust ChatGPT over your sourced tool for patient care." Benchmarks aren't bedside use, both still err, and the sourced tool still gives you something to verify. The real lesson is the next slide.
The takeaway isn't "pick a winner." It's that the category labels -- "clinical AI tool" vs. "chatbot" -- tell you less than the mechanism does.
Ask of any tool you're handed: Does it retrieve sources, reason from training, or both? How current is what it can see? Can I check where the answer came from? Those three questions predict its behavior better than the brand name or the word "clinical" in its marketing.
The benchmark winner changes every few months. The questions above don't. Learn the mechanism, not the leaderboard -- it's the part that stays true after the next model drops.
For each situation, decide which strategy fits -- and why.
Module 2 - Lesson 3 complete
Next, the lesson that ties it together: a practical framework for trusting, verifying, and safely using these tools at the point of care.