HuatuoGPT-o1: The First Medical AI That "Thinks" Before It Answers

Key Takeaways

If you have been tracking the breathtaking pace of artificial intelligence breakthroughs lately, [music] you'll know that the conversation has moved uh far beyond whether large language models can just generate coherent text. >> Oh yeah, comp >> the new frontier is proving that they can actually reason. >> That's been the focus of so much of the high-profile stuff, right? I mean, think of all the excitement around reasoning capabilities from projects like OpenAI's 01. But if you look really closely at where those, you know, verifiable, reproducible results have been, almost all of that initial solid progress was in one area, mathematics. >> And that makes sense. Math is the perfect sandbox for proving reasoning because the rules were just pristine. The steps are unambiguous. The proof is verifiable line by line. And the final answer is a clean, objective outcome. >> Right? >> You either solved for X correctly or you didn't. End of story. But our challenge and well the central focus of this deep dive is what happens when you leave that clarity of computation and you enter a specialized highstakes life critical field like medicine. >> It's a whole different ball game. >> The difficulty in achieving reliable reasoning doesn't just multiply it shifts entirely because now you're dealing with language with nuance with uncertainty. >> Exactly. Our mission today is to really unpack this research paper Y2GP1 towards medical complex reasoning with LLMs. This project is all about figuring out how to teach an AI to deliberate, to self-correct, and to think iteratively in a complex diagnostic scenario. >> The kind of thinking we expect from a human physician. >> Precisely. A medical diagnosis demands meticulous, complex thinking. that kind of structured reflection we associate with a seasoned human doctor. >> And this is so critical because you know while the medical domain is technically narrower than say all of general human knowledge, the thought process itself is uniquely challenging to verify. >> How so? >> Well, a patient's symptoms can be described in a dozen different ways. A disease has multiple aliases. A treatment protocol has variations. You can't use simple mathematical checks to grade the AI's chain of thought or KYT. It just doesn't work. So the core problem the Wahoo GP1 researchers had to solve was basically how do you build a mechanism that lets the AI question its own diagnosis to allow for that iterative selfcorrection in a field where the language itself is inherently ambiguous. >> That's the million-doll question. How do you verify the reasoning? >> So how did they do it? To solve that structural verification problem, they designed two foundational interdependent components. And I mean, understanding these is really the key to grasping the whole breakthrough. >> Okay, it for us. >> First, they had to create a gold standard of training material that actually demanded deep thought, which they called verifiable medical problems or VMPs. >> It's not just a list of facts. >> Not at all. And second, they needed a specialized grading tool, one designed to handle all those medical aliases and nuances. and they built that as the medical verifier. >> Okay? >> And together, those two pieces unlock the door to using these advanced search strategies and reinforcement learning for something as complex as a medical diagnosis. >> All right, let's unpack that first foundational piece, creating the training data. The journey here begins with a massive corpus. We're talking 192,000 closed set medical multiplechoice gamma questions. >> Yeah. And these are sourced from MedQua, USMLE, and MEMCQA training sets. So this is real objective peer-reviewed knowledge. >> That volume is huge. But as you said, traditional multiple choice questions are well, they're fundamentally insufficient for training deep reasoning. >> There really are. I mean, think about it from the model's perspective. If you give it the prompt and four options A, B, C, D, it can often just reverse engineer the likely answer. It spots keywords. It eliminates the obvious distractors. >> So it can guess the right answer without ever having to do the actual clinical thinking required to reach a complex diagnosis. Exactly. You're teaching it to pass a test, not to practice medicine. >> They needed a data set that actively forces deep thinking, reflection, and the generation of entirely new language. >> Precisely. And that's why they started with those 192,000 questions and had to perform this highly rigorous three-stage filtering process that whittleled that pool down to just 40,000 highquality verifiable medical problems. >> Okay. So, these stages were designed to get rid of the easy stuff, the short stuff, and the ambiguous stuff. Yeah, >> let's start with that first hurdle, weeding out the easy questions. How did they even figure out what required truly deeper complex reasoning? >> It was a pretty clever strategy actually. They took three relatively small base LLMs, Gemma 29B, Lam 3.18B, and Quen 2.57B, and just ran them against the entire corpus. And if they all got it right, >> if all three of those smaller models could correctly answer a question, or if the question itself was very short, it was just discarded, thrown out. >> That's a brilliant way to establish the difficulty floor, isn't it? By eliminating all the questions that even a small general purpose LLM can consistently ace, you're guaranteeing that what's left are the problems where the AI is most likely to struggle and to reflect and genuinely need to construct a complex thought process to get to the solution. >> It forces the model into an exploratory mindset. You're basically curating the curriculum to maximize the need for reflection. >> Okay. What was step two? >> Step two was all about ensuring unique and objective ground answers. So they excluded any questions that ask for the incorrect option because that just introduces an extra layer of logical complexity that kind of obscures the core medical diagnosis. >> Right. That makes sense. And more importantly, they screened out questions that might legitimately have multiple correct answers based on slightly different but still defensible clinical interpretations. >> And to make sure that objectivity was absolute, they brought in the heavy artillery. >> They did to remove all the remaining ambiguity. They used the state-of-the-art model GPT4 to vet the remaining questions and ensure a totally objective ground truth answer. It's a small detail, but it really shows the lengths they went to in order to ensure the quality of that training signal. >> And then comes step three, reformatting. This is where the VMP is truly born. They took those challenging, objective, multiple choice questions and just fundamentally changed their structure, transforming them into open-ended problems, which they denoted as $6. >> And that transformation is what forces the reasoning. Right. >> Exactly. The model can no longer just perform classification or selection. and it has to generate the complex medical term or concept from scratch and the original correct multiple choice option is kept as the definitive objective ground truth answer which they call yay >> let's make this tangible for you the listener we can use the malario case study from the paper so imagine a scenario a 30-year-old woman comes in after recent travel to India she has fevers chills headaches and classic signs like palar and jaundice her lab work is concerning >> the original exam question would ask what is the most severe complication of this condition and then it would offer options like you know a heart block b facial paralysis cerebral edema d a plastic crisis the correct letter is c >> but the VMP version just gets rid of the choices it presents the full case study and the open-ended question what is the most severe complication of this condition >> and the model has to generate the specific complex medical term cerebral edema it has to reason through the history the symptoms the labs and connect all of that to the most severe outcome come of a plasmodium feliperum infection rather than just matching keywords. >> But that leads us straight to the second pillar of this whole architecture, the medical verifier, >> right? >> Once the LLM generates its open-ended answer, let's call it $1, you have to check if y'all is truly equivalent to the ground truth. So why can't you just use the simple exact match method that works perfectly fine in math? >> Because of the chaos of medical terminology, exact match just fails. It fails because of medical equivalents and aliases. >> Give me an example. Okay, so if a patient has an irregular irregular heart rhythm due to a history of alcohol abuse, the ground truth answer might be atrial fibrillation induced cardiammyopathy. But the model might generate a technically correct but linguistically different alias like chronic super ventricular tacocardia induced cardiammyopathy. Both of those terms describe the same mechanism in that specific clinical context. So an exact match verification would fail the model in that scenario even though its complex reasoning was spot-on. >> Exactly. And that means your entire training signal would be corrupted. The verifier has to understand medical equivalents, not just identity, >> which means you need a language-based solution. >> It forced them to use one. So they initially used GPT40 again as the gold standard verifier. It was prompted to compare the model's open-ended output against the definitive ground truth and provide simple binary feedback. True or false? >> Let's just pause on the data here for a second because this is where the difficulty of this whole problem is really exposed. >> It is stunning. I mean, when they tested the simple exact match method, the standard approach used in a lot of math LLMs, it failed. It only had 70.5% accuracy in verifying the output of the stage 1 models. >> 70%. That's a catastrophic failure rate for a training signal. >> It confirms that simple rajax based verification is completely impractical in medicine. >> But the GPT40based verifier, >> it achieved a stunning accuracy of 96.5% in manual verification studies in stage one. This proved that a nuanced LLMbased system can reliably navigate the ambiguities of medical language. >> And they didn't just stop there. >> No. And this is important. They successfully fine-tuned an 8 billion parameter verifier model based on Alami 3.1 that achieved over 90% accuracy. This shows that the function of the verifier is robust and learnable. It's not just reliant on some massive proprietary model. >> That detail is crucial. It shows they could generalize the verifier function. But I have to ask, since this whole system relies on the verifier to provide that clean signal, isn't bootstrapping their initial gold standard onto a proprietary blackbox model like GPT40, a potential vulnerability? >> That's a really pertinent question, and I think it reflects the current state of AI research. They used GPT40, which was arguably the best reasoning engine available at the time, not as the final model, but as the gold standard curator, >> to create the training data and to train their own smaller open-source verifier. >> Exactly. And the justification is in the results. No open- source model could achieve that necessary 96% reliability needed to clean the ambiguous data set and bootstrap the process effectively. So they sacrifice some initial openness for verifiable reliability >> which then allowed them to create a highquality open- source 8b verifier model that is robust enough to grade the training data going forward. >> So it's a pragmatic approach to overcoming a huge technical barrier. So they use the best tool available to establish the truth and then they successfully transfer that truthing skill to a small verifiable 8B model. >> Okay, that makes sense. So with the verifiable medical problems and the high fidelity medical verifier in place, they've solved the structural problem. Now how do they teach the model to actually use this signal to improve its thinking? >> Now we get into the core training methodology. The ultimate goal here is to move the model away from simple linear reasoning and towards a critical iterative reflection process. The field often calls this stream of search or so. So you're teaching the model to actively recognize, diagnose, and refine its own mistakes. >> Exactly. Much like a seasoned medical consultant might do during a difficult case conference. >> And this starts with stage one, supervised fine-tuning or SFT. So this stage isn't about teaching the final answer. It's about teaching the skill of complex reasoning by showing it successful examples of selfcorrection. >> Precisely. It begins simply. The LM gets a VMP and generates an initial chain of thought and an initial answer, the out dollars. This first attempt is immediately judged by that highly reliable medical verifier. >> And if the verifier says false, >> the model doesn't just stop, it's forced into an iterative refinement loop. >> Okay, so you're building a multi-step trajectory where the model is actively applying some kind of search strategy to find and fix its error. >> That's the key. This iterative search continues until either a correct answer is found or until the maximum iteration count which they set at on pound three 33 steps is hit. At that point, the entire seeing process just restarts from a fresh perspective >> and they actually define the cognitive behaviors the model could use. There were four specific search strategies, right? >> Yep. And these really form the blueprint for the AI's critical thinking. >> Okay, let's slow down and detail those four strategies. What's the human analogy for the first one? Exploring new paths. >> This is the most drastic intervention. It means the model basically concludes its current line of reasoning is a dead end. And decides to try a completely distinct new approach. It's like a human physician saying, "Okay, I've been stuck on this infectious disease hypothesis for too long. Let me completely pivot and consider a primary autoimmune condition instead." >> So, just throwing out the first hypothesis and starting fresh. completely fresh. Then you have backtracking. This one is fascinating. It really mimics the human ability to revisit a specific moment where you think you went wrong. >> Right? >> The LLM identifies a specific intermediate step earlier in its reasoning, maybe two or three steps back, and decides that was the point of divergence. It then reverts to that specific step and continues reasoning from there. >> It's the doctor saying, "Wait, where did I go wrong 10 minutes ago? I shouldn't have dismissed that lab result." >> Exactly. And the researchers noted that backtracking was most effective when it was sampled early in the search before the reasoning path had diverged too far. >> Okay. The third strategy is verification. This sounds like a self- audit function. >> It is. The LLM explicitly pauses its reasoning to evaluate the logic and the result it currently has. It's providing a metal level validation asking itself, does this line of thought align with the initial case data? Are my assumptions about this drug interaction correct? It's a structured internal check. >> And finally, correction. This sounds more immediate than backtracking. >> It's a surgical fix. Instead of backtracking several steps, the LLM just critiques and precisely fixes flaws in the immediate prior reasoning step. If the previous step incorrectly calculated a dose based on weight, correction fixes only that weight to dose calculation, and the model moves on. So the model is using these four cognitive tools to perform a true stream of search until a path is found that leads to the verifier saying true. >> Right? And once that successful multi-step trajectory is found, a path that might have involved multiple restarts, three initial attempts, maybe nine iterations, that entire sequence becomes the core training material. >> But this is where the complex Coutique comes in, right? Because that messy multi-hypothesis search trajectory has to be pretty disjointed. It is it's too disjointed to be useful for SF2 training. So the entire sequence is synthesized and reformatted into a cohesive natural language narrative and that is the complex cop. >> This reformatting itself seems crucial. It's not just logging the search steps. It's curating the logical flow. >> It's the curator of logic. Absolutely. This process smooths out the rough edges. It eliminates rigid structured outputs. Instead, it uses these smooth humanlike transitions like or but wait, let's pause and reevaluate this. or that internal dialogue of not so fast. >> So, it's almost like they're teaching it to have a human-like internal monologue. >> That's a great way to put it. It makes the final complex Kodi narrative feel like a genuine reflective thought process. It's teaching the model not just what to think, but how to deliberate efficiently. >> The case study they provide in the paper illustrates this perfectly, and we should dwell on it for a moment. It's a 51-year-old man. He's got progressively worsening dyne and fatigue, signs of heart failure, and critically an irregularly irregular ECG rhythm. >> The initial thought process, the first hypothesis is plausible. The model sees the signs of heart failure. It might recall a history of heavy drinking and it concludes the most likely cause is alcohol-induced dilated cardiomyopathy. >> Plausible, but the verifier knowing the full context is false. >> So the search kicks in. The model applies a strategy maybe correction and its next conclusion might be although alcohol-induced cardiammyopathy was initially considered eskeemic cardiammyopathy is more likely >> still false a valid alternative hypothesis but not the critical path. >> So now a deep research step is applied maybe exploring a new path and the model focuses on the subtle critical details like that persistent regularly irregular ECG and it refineses its thinking dramatically too. The most likely cause is chronic super ventricular tacicardia atrial fibrillation induced cardiomyopathy >> and that is the ground truth. What's amazing is that the initial search trajectory the full messy deliberation was over 1,500 tokens long. But the synthesized complex autouti the training example cuts that down to an efficient elegant 758 token narrative and that narrative captures the essence of critical thinking identifying the irregularity on the ECG questioning the initial hypothesis and connecting the key dots. >> So those 20,000 complex coup data points are then used in stage one the supervised fine-tuning phase which successfully teaches the base LLM that reflective reasoning skill. >> Exactly. >> Okay. So stage one taught the LLM the skill of structured self-refinement. But because these trajectories were generated through a partly random search process, they might not be optimal. This brings us to stage two, enhancing complex reasoning with reinforcement learning. >> Yep. Specifically using the proximal policy optimization or PO algorithm. This is the refinement phase. SFT provides the initial ability to think complexely, but PO optimizes that thinking for peak performance. >> And the verifier steps in one last time, but now it's providing the feedback loop for the RL algorithm. How did they structure the reward system? Because medical problems have a sparse outcome. You're either right or wrong. >> The rewards are sparse, but they're highly strategic. If the final generated answer is correct, the model gets a maximal reward of one. >> Okay. >> If the answer is incorrect, the reward is still a small positive value.1. >> Wait, why for a wrong answer? That seems a little counterintuitive. >> It incentivizes the process of deliberation, even if that deliberation ultimately fails. Think of it this way. If a student studies hard for a test but gets the answer wrong, you still want to encourage the studying and behavior. >> Ah, I see. >> Giving point one is like saying the result was incorrect but thank you for providing the detailed chain of thought. It prevents the model from being totally discouraged by failure. And crucially, if the model completely lacks the think before answering behavior, if it just spits out an answer with no couty at all, it gets a reward of zero, >> a penalty. So they're literally rewarding the depth of thought. Exactly. Now, training a model with PO using these sparse rewards can be unstable. The model might drift away from the complex reasoning it learned in stage one. So, to stabilize this, they combine the function score with a callback liler divergence term. >> Okay, that's a technical term that can lose a lot of us. Can you give us an analogy for what that KL divergence term is actually doing? >> Think of the KL term as a guardrail. The PO algorithm is trying to find the optimal path, the highest reward, which means it wants to change the model's behavior. The KL divergence term acts as a penalty if the new behavior strays too far from the initial good behavior it learned in stage one. >> So, it's like a leash. >> It's essentially saying you can explore and find new higher reward solutions, but don't forget the core effective complex reasoning skills you already mastered. It stabilizes the whole optimization process by keeping the model anchored to its well-learned SFT foundation. >> That's a great explanation. So, the two-stage process is complete. SFT teaches the iterative skill and PO optimizes it. That brings us to the results, >> right? >> So, did all this architectural innovation actually pay off? The researchers built the Wattu GPT01 models based on the LMA 3.1 architecture and 8 billion and a 70 billion parameter version. Let's look at the results. >> The payoff was substantial. The WATU GPT018B model just using its 40,000 verifiable problems achieved an overall 8.5 point average improvement over its base LMA 3.18B model across a bunch of medical benchmarks. >> That kind of performance leap from a relatively small investment in highquality, high complexity data is incredibly significant. >> It's huge. And the 70B model scaled this success. It didn't just perform well on the tests it was trained against. It established new benchmarks. It surpassed other large open- source models, including ones already specialized for medicine like Ultram 7DB and Open BioLM7DB. >> And it was strong on the really tough benchmarks, not just the standard ones. >> That's the crucial part. Its strength wasn't just on standard factual retrieval tests like MedQA. It was on the newer, more reasoningheavy data sets. I'm talking about the health and biology tracks of MMLU Pro and the genetics and molecular biology tracks of GPQA, >> which is evidence that it's not just memorizing facts. >> It's evidence that it acquired robust transferable reasoning capabilities. Previous medical LLMs often struggled on those benchmarks, suggesting they lacked the deep iterative deliberation skills that Hatua GP1 was explicitly trained for. >> This brings us to the aha moments of the research, the ablation study. This is where they systematically removed components to see what was actually driving the performance. So what happened when they tested models trained only the old-fashioned way. >> They confirmed our initial suspicion. Training a fine-tuned baseline solely on the original multiple choice format yielded only minimal improvement over the base lama model. >> So there's the clear empirical confirmation. Simply learning the answer from multiple choice tests, no matter how vast the data set, does not teach robust problem solving. >> It just teaches selection. But the next finding is this hierarchy of deliberation, right? They measure the impact of different levels of chain of thought reasoning. >> Absolutely. Training the model to generate a direct response with no cot performed the worst. Introducing a simple cot a basic single line of reasoning offered very little benefit. But when they introduced the complex cot that synthesized reflective trajectory with self-critique, the performance shot up dramatically. >> How much? an average of 4.3 points of performance gain over direct responses. This just strongly validates the entire premise of the research. You need to incorporate iterative reflection and self-critique for these highstakes reasoning tasks. >> And this complexity had a huge synergistic effect with the reinforcement learning stage. This is where the depth of the thought process really mattered. >> This is maybe the most insightful finding in the paper. Complex CI didn't just improve the initial skill in stage one. It drastically enhanced the benefits from the RL in stage 2. >> So how did that work? >> Well, let's look at the numbers. The simple cot trajectories, they averaged only about 281 tokens long. But the complex cot trajectories with all the reflection and searching were far longer. They averaged 712 tokens. >> That is a difference of over 400 tokens per thought process. Why does that token length matter so much for reinforcement learning? >> It provides richer data. Longer, more complex thought paths contain a greater variety of states and actions for the RL algorithm to explore and optimize. Think of the 712 token complex cot as a vast detailed map of possible solutions. >> A map with more roads to explore >> and with high value intermediate steps that the model can learn to prune or enhance in a sparse reward environment where the big reward only comes at the very end. Having these longer, deeper thought processes means the model gets better feedback on how to achieve that reward. >> So the richer the thought process, the better the RL algorithm can do its job. >> And the empirical data backs this up perfectly. Complex Cody paths yielded a significant 3.6 point performance gain from RL optimization. Models that didn't use complex coty saw only a 1.1 point gain from RL. >> Wow. This proves that to make RL effective in these complex non-mathematical domains, you have to first teach the model a complex iterative thinking skill that gives it a rich trajectory to work with. >> They also briefly compared different reinforcement learning algorithms, right? What did they find about PO? >> PO performed the best among the tested algorithms, which confirms its stability and effectiveness. Here, it slightly outperformed RLO and it notably beat DPO, which is another common RL algorithm. >> Why did DPO struggle? The researcher suggested its weaker performance was likely due to its off-p policy nature, meaning it learns from data generated elsewhere. While PO benefited greatly from its use of value models, which provided necessary stability when dealing with those sparse rewards. >> Finally, we need to talk about the robustness of this whole blueprint. Can this method be transferred to other languages or is this just an English language success story? >> The answer is a clear yes. They demonstrated domain adaptability by taking this exact methodology and applying it to the Chinese medical domain. They constructed 40,000 verifiable Chinese medical questions from the CMBB exam training set and trained a new model Watu GPT01-7BS >> and the results in Chinese. >> The Chinese model subsequently outperformed all other comparable Chinese LLMs across various Chinese medical benchmarks. >> That's critical validation. >> It is. It confirms that the breakthrough isn't language dependent, it's methodology dependent. The innovation is the framework for teaching complex reasoning, not the linguistic content itself. >> So, bringing this back to you, the learner, >> the core takeaway here isn't that they trained a bigger model or fed it more medical journals. The breakthrough was designing an entire system, the verifiable medical problems combined with the medical verifier that enables iterative refinement. It simulates a rigorous self-critical stream of search thinking loop. >> One that mirrors how human experts engage in differential diagnosis and problem solving. Finally moving specialized AI reasoning beyond that strictly numerical realm and into these highstakes language domains >> to synthesize the unique contribution of the Hatua GPTO project. You can really see two major points of innovation. First, they successfully transformed a complex linguistically ambiguous domain like medicine into a functionally verifiable domain. They did that by creating the VMPs and the LLM verifier. >> They made the messy clean. >> Exactly. And second, that verification structure allowed them to apply these powerful learning techniques, searchguided learning and reinforcement learning in a specialized domain effectively for really the first time. This proves that sophisticated reflective AI reasoning can thrive outside of mathematics. It sets a massive precedent for reliable specialized AI training. But given that this entire conversation has been about a lifecritical field, there must be a necessary limitation we have to reinforce here. >> Absolutely. The researchers are highly explicit about the ethical necessity of restraint. They emphasize that while Watua GP01 exhibits highly capable complex reasoning, it is fundamentally not suitable for real world clinical or industrial applications. And that's a really important point. >> The risk of hallucinations, unpredictable inaccuracies, and unforeseen errors remains. This model is purely a research breakthrough for advancing LLM reasoning. It is not a clinical tool or a diagnostic aid, and strict adherence to these restrictions is necessary to safeguard the integrity of medicine and of course patient safety. >> That ethical limitation is paramount. But looking forward, if the successful complexity of medical reasoning requires models to engage in this deep self-critical deliberation which almost mirrors a differential diagnosis process, >> yeah, >> what does this method mean for other specialized highstakes fields? >> The paper mentions finance, law, or security as potential future applications, >> right? I think this work provides a powerful blueprint for building reliable AI in any specialized domain where definitive verification of the outcome is possible even if the intermediate reasoning path is complex and filled with specialized aliases or nuanced language. >> So the technical challenge isn't just accumulating a knowledge base. >> Not at all. It is overcoming the verification bottleneck. So the next great leaps in specialized AI might not come from building bigger foundation models, but from creating better, more rigorous testing and training environments specific to each domain. Environments that force the AI to prove its complex work iteratively, just like a human expert would. >> That's right. >> What specialized field will be the next one to create its own verifiable complex reasoning benchmark? I think that's the question this research really raises for the future of specialized intelligence.