📰 Dev.to · Marcus Chen

The 2am call that dropped before the user finished talking, and the week I spent finding out why my tracer never saw it

Position bias in LLM-as-judge flipped 18% of our verdicts

The 1.4 Seconds That Weren't on Any Span

The Retry That Booked Mrs. Alvarez Twice

Bootstrap confidence intervals for your LLM eval metrics

Benchmarking 5 LLM providers on one eval set, no SDK per vendor

temperature=0 didn't make our LLM evals reproducible

Harvesting a regression test set from gateway logs with a plugin

Perplexity held flat after INT4. Task accuracy dropped 7 points.

A 9-point eval gain vanished when we deduped train against test

Our voice agent passed every test and still woke me up at 3am

The 4-layer voice-agent latency stack, traced with OTel spans

Provider drift broke our regression evals. We pinned versions through Bifrost.

Aggregate eval scores hid a 14-point regression in one user segment

Shadow-testing a fine-tuned 8B against gpt-4o-mini through Bifrost

Continuous batching wrecked our p99 latency. Here's the trace.

Virtual keys per tenant: ditching our custom LLM billing layer

LLM-as-judge variance broke our DPO training signal for 3 weeks

Token-level eval harness for tool-calling agents: what we wired up

Prefix caching in vLLM under multi-tenant agent traffic

We Audited Our Agent Tool-Call Traces. Half Our Eval Data Was Garbage.

Why Your LLM Eval Harness Is Lying to You (And How to Fix It)

Measuring AI Gateway Failover: 30 Days of Production Data

What Gemma 4's multi-token prediction head actually means for your eval pipeline