Knowing When the Model Is Actually Right

📰 Medium · LLM

Learn how to evaluate the effectiveness of a model to ensure it's actually working as intended, and why building an evaluation harness is crucial for model improvement.

intermediate Published 30 Apr 2026

Action Steps

Build an evaluation harness to assess model performance before training
Define key metrics to measure model effectiveness
Implement A/B testing to compare model iterations
Use data visualization to understand model performance and identify areas for improvement
Continuously monitor and update the evaluation harness to ensure model performance is meeting requirements

Who Needs to Know This

Data scientists, machine learning engineers, and product managers can benefit from understanding how to evaluate model performance to make informed decisions about model improvements and deployments.

Key Insight

💡 Building an evaluation harness is essential to ensure model performance is meeting requirements and to identify areas for improvement.

Key Takeaways

Learn how to evaluate the effectiveness of a model to ensure it's actually working as intended, and why building an evaluation harness is crucial for model improvement.

Full Article

Title: Knowing When the Model Is Actually Right

URL Source: https://medium.com/@theadityamittal/knowing-when-the-model-is-actually-right-f9d694454337?source=rss------llm-5

Published Time: 2026-04-30T02:39:36Z

Markdown Content:
# Knowing When the Model Is Actually Right | by Aditya Mittal | Apr, 2026 | Medium

[Sitemap](https://medium.com/sitemap/sitemap.xml)

[Open in app](https://play.google.com/store/apps/details?id=com.medium.reader&referrer=utm_source%3DmobileNavBar&source=post_page---top_nav_layout_nav-----------------------------------------)

Sign up

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Fmedium.com%2F%40theadityamittal%2Fknowing-when-the-model-is-actually-right-f9d694454337&source=post_page---top_nav_layout_nav-----------------------global_nav------------------)

[](https://medium.com/?source=post_page---top_nav_layout_nav-----------------------------------------)

Get app

[Write](https://medium.com/m/signin?operation=register&redirect=https%3A%2F%2Fmedium.com%2Fnew-story&source=---top_nav_layout_nav-----------------------new_post_topnav------------------)

[Search](https://medium.com/search?source=post_page---top_nav_layout_nav-----------------------------------------)

Sign up

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Fmedium.com%2F%40theadityamittal%2Fknowing-when-the-model-is-actually-right-f9d694454337&source=post_page---top_nav_layout_nav-----------------------global_nav------------------)

![Image 1](https://miro.medium.com/v2/resize:fill:32:32/1*dmbNkD5D-u45r44go_cf0g.png)

# Knowing When the Model Is Actually Right

[![Image 2: Aditya Mittal](https://miro.medium.com/v2/resize:fill:32:32/1*FCQ4nwEke18rc49ZW5d65Q.jpeg)](https://medium.com/@theadityamittal?source=post_page---byline--f9d694454337---------------------------------------)

[Aditya Mittal](https://medium.com/@theadityamittal?source=post_page---byline--f9d694454337---------------------------------------)

Follow

7 min read

·

Just now

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fvote%2Fp%2Ff9d694454337&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40theadityamittal%2Fknowing-when-the-model-is-actually-right-f9d694454337&user=Aditya+Mittal&userId=d32837efa3a1&source=---header_actions--f9d694454337---------------------clap_footer------------------)

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fbookmark%2Fp%2Ff9d694454337&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40theadityamittal%2Fknowing-when-the-model-is-actually-right-f9d694454337&source=---header_actions--f9d694454337---------------------bookmark_footer------------------)

[Listen](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2Fplans%3Fdimension%3Dpost_audio_button%26postId%3Df9d694454337&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40theadityamittal%2Fknowing-when-the-model-is-actually-right-f9d694454337&source=---header_actions--f9d694454337---------------------post_audio_button------------------)

Share

Press enter or click to view image in full size

![Image 3](https://miro.medium.com/v2/resize:fit:700/1*zZ-L8UFP1thARwurR49RmA.png)

The first thing I built at NovumAI wasn’t the model. It was the eval harness.

That sounds like a contrarian take, but it isn’t. It’s just what the work demanded. NovumAI ships coaching suggestions to sales agents while they’re on live calls — the model hears an utterance and tells the rep what to say next. When I joined, the system existed but the suggestions weren’t great, and the team wanted to make them better. The obvious move was to start training. The problem was that I had no way to tell whether anything I changed was actually an improvement.

That’s a worse problem than it sounds. Without an eval, every change becomes a coin flip. You ship something, the team thinks it feels better, you ship something else, somebody thinks it feels worse, and after a few weeks of this you’

Read full article → ← Back to Reads

Knowing When the Model Is Actually Right

Key Takeaways

Full Article

Related Videos