Pydantic Evals in Python: Calibrate LLM Judges with Human Labels

LLM judges are useful only after you check them against humans. Pydantic evals gives you the data set and evaluator structure to do that. You'll understand the pattern behind a pipeline that compares [music] judge ratings to human labels and adjusts the rubric. I'm Professor Py, teaching AI engineering and LLM systems with simple Python. Judging LLM outputs sounds neat until you [music] realize the judge can disagree with real reviewers. That gap is a release risk. Calibrating a judge against human labels turns an opinionated model into a predictable component you can trust. People use this when they need an automated reviewer for moderation, [music] grading, or content quality checks. Success looks like >> [music] >> fewer false positives and negatives when the system goes to production. Intuitively, we want three things. First, a stable set of human-labeled examples. Second, a way to run the judge and capture its [music] verdict and confidence. Third, logic that adjusts the judge so its decisions align with humans. One simple way is to collect [music] cases, run the judge offline, find the boldest mismatches, and [music] then tweak a threshold. That gives you a reproducible gate before release. We will build a tiny evaluation pipeline that stores human labels, replays [music] judge outputs without live APIs, finds disagreements, and then applies a pass threshold to match reviewers. The goal is to make the judge behave like the team expects, not like a bench experiment. Quick note, this code is designed to teach the core idea clearly. Library versions and local setups can vary, so check the official docs if your setup behaves differently. First, we lock the human cases into a Pydantic evals dataset, so everything else runs on the exact same records. This snippet builds a tiny Pydantic evals dataset loaded with human review signals. Skipping this step leaves you blind to whether [music] the judge will even see the same class balance the reviewers did. So, we anchor [music] each case with the human expected output. Collecting them into dataset locks the samples into the evaluation workflow instead of some spreadsheet, meaning later judge runs will target the exact same records. Computing label totals [music] right away surfaces the pass-fail mix, so the printout shows we are testing on an even split before calibrating anything. Now, we [music] run the judge over those stored cases, but we keep it deterministic by replaying recorded outputs instead of hitting an API. This example runs dataset.evaluate_sync, so we can capture how our LLM judge behaves on the stored cases. Letting that judge operate unchecked is how mismatches sneak through release gates. So, mock judge replays the verdict and score tied to >> [music] >> each input's ID. I'm using recorded outputs instead of a live API call here, which keeps the tutorial credential-free while still reflecting the structure of real judge responses. >> [music] >> Report now holds every case result, and compressing it into judge view shows the human label, the judge decision, and the confidence in one short string. Setting max_concurrency equals one keeps the playback deterministic, which matters when you're inspecting disagreements case by case. With raw results in hand, the next need is to find and rank the disagreements, so you inspect the biggest errors first. This code ranks the judge human disagreements, so we inspect the most concerning ones first. The danger is assuming every mismatch is equally bad. A pass label with high judge confidence is a bigger [music] fire than a low score shrug. By filtering report.cases into disagreements, we [music] isolate only the rows where case.output_verdict diverges from the human expected output. Sorting by the stored score pulls the boldest overrules to the top, which tells you exactly where to focus rubric fixes. The printed list now highlights which [music] case and score combination still disagrees, so you see how far the judge strays from the reviewers. After spotting the worst misses, we need a quick fix [music] you can test. Recalibrate the judge with a pass threshold to flip low confidence [music] passes. This snippet adds a simple pass threshold, so we can recalibrate the judge before trusting [music] it. Without a guard, a judge that rubber stamps borderline answers will keep slipping past human expectations. So, calibrated judge flips low confidence passes >> [music] >> back to fails. It reuses the same judge outputs, but overrides the verdict whenever the stored score falls below pass floor, leaving higher confidence calls untouched. Raising pass floor makes the judge stricter and can introduce false negatives, while lowering it lets [music] more borderline responses through. So, you can tune it to match the reviewer's tolerance. The printed calibrated mismatches list going empty shows the threshold eliminated this data set's disagreement. Finally, we summarize whether that tweak actually improved alignment enough to gate a release. This example rolls the raw and calibrated runs into a quick release gate summary. Making decisions on gut feel risks shipping regressions. So, accuracy computes how often each run match the humans. Raw acc reflects the original report.cases, while calibrated acc [music] we uses calibrated report.cases to show the impact of the threshold tweak. We combine [music] the rounded scores and a simple release ready boolean in one printout, [music] which gives a reproducible artifact you can drop into experiment tracking. Seeing the calibrated accuracy jump to the go no go target makes it clear whether the tweak is worth promoting. In a real project, you map the toy data set to a slice of your production logs. For example, recent support conversations or moderation samples [music] that matter to users. Collect those same reviewers, replay judge outputs, [music] and then tune the threshold until the calibrated run matches the reviewers at an acceptable rate. Keep this in mind. It is a reproducible compare and fix pattern. [music] Use it when you automate decisions that humans currently make. One caveat is that threshold [music] tweaks can introduce new errors. So, monitor for shifts in other slices. A practical next step is clear. Tune the pass floor value on a larger validation slice drawn from your production logs and track calibrated accuracy per user cohort. That single parameter will often [music] give the most leverage for aligning the judge with human expectations. [music] If short, practical AI engineering helps, subscribe and watch the AI engineering videos.