Better Models: Worse Tools
📰 Simon Willison's Blog
Newer LLM models can be worse at using certain tools due to overfitting to specific edit tools, making it challenging for third-party coding harnesses to implement compatible tools
Action Steps
- Investigate the tool usage patterns of newer LLM models, such as Opus 4.8 and Sonnet 5
- Compare the performance of different edit tools, such as Claude's search and replace and OpenAI's apply_patch mechanism
- Consider implementing multiple edit tools in third-party coding harnesses to ensure compatibility with various LLM models
- Test and evaluate the effectiveness of different tool implementations with newer LLM models
- Collaborate with LLM developers and researchers to better understand the implications of model updates on tool usage and compatibility
Who Needs to Know This
Developers and researchers working with LLMs and coding harnesses, such as Pi, may need to adapt their tools to accommodate the evolving capabilities of newer models
Key Insight
💡 Newer LLM models may require adapted tools to ensure compatibility and effective usage
Share This
Newer #LLM models can be worse at using certain tools due to overfitting, making it challenging for third-party coding harnesses #AI #coding
Key Takeaways
Newer LLM models can be worse at using certain tools due to overfitting to specific edit tools, making it challenging for third-party coding harnesses to implement compatible tools
Full Article
Title: Better Models: Worse Tools
URL Source: https://simonwillison.net/2026/Jul/4/better-models-worse-tools/
Published Time: Sat, 04 Jul 2026 22:54:55 GMT
Markdown Content:
# [Simon Willison’s Weblog](https://simonwillison.net/)
[Subscribe](https://simonwillison.net/about/#subscribe)
**Sponsored by:** Sonar — Gartner just named Sonar a Leader in the 2026 Magic Quadrant™ for Technical Debt Management Tools. Read the report and learn how to measure and remediate technical debt across your codebase. [Get the report](https://fandf.co/3STP9vD)
4th July 2026 - Link Blog
**[Better Models: Worse Tools](https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/)**. Armin reports on a weird problem he ran into while hacking on Pi:
> The short version is that newer Claude models sometimes call Pi’s edit tool with extra, invented fields in the nested `edits[]` array. And not Haiku or some small model: Opus 4.8. The edit itself is usually correct but the arguments do not match the schema as the model invents made-up keys and Pi thus rejects the tool call and asks to try again.
>
>
> That alone is not too surprising as models emit malformed tool calls sometimes. Particularly small ones. What surprised me is that this is getting worse with newer Anthropic models as both Opus 4.8 and Sonnet 5 show it but none of the older models. In other words, the SOTA models of the family are worse at this specific tool schema than their older siblings.
Armin theorizes that this is because more recent Anthropic models have been specifically trained (presumably via Reinforcement Learning) to better use the edit tools that are baked into Claude Code. This has the unfortunate effect that other coding harnesses, such as Pi, may find that their own custom edit tools are more likely to be used incorrectly.
Claude's edit tool [uses search and replace](https://platform.claude.com/docs/en/agents-and-tools/tool-use/text-editor-tool#str-replace). OpenAI's Codex [uses an apply_patch mechanism instead](https://developers.openai.com/api/docs/guides/tools-apply-patch), and OpenAI have talked in the past about how their models are trained to use that tool effectively.
Does this mean third-party coding harnesses like Pi should implement multiple edit tools just so they can use the one with the best performance for the underlying model the user has selected?
Posted [4th July 2026](https://simonwillison.net/2026/Jul/4/) at 10:53 pm
## Recent articles
* [Have your agent record video demos of its work with shot-scraper video](https://simonwillison.net/2026/Jun/30/shot-scraper-video/) - 30th June 2026
* [Porting the Moebius 0.2B image inpainting model to run in the browser with Claude Code](https://simonwillison.net/2026/Jun/22/porting-moebius/) - 22nd June 2026
* [sqlite-utils 4.0rc1 adds migrations and nested transactions](https://simonwillison.net/2026/Jun/21/sqlite-utils-40rc1/) - 21st June 2026
This is a **link post** by Simon Willison, posted on [4th July 2026](https://simonwillison.net/2026/Jul/4/).
[armin-ronacher 24](https://simonwillison.net/tags/armin-ronacher/)[ai 2,102](https://simonwillison.net/tags/ai/)[openai 426](https://simonwillison.net/tags/openai/)[generative-ai 1,859](https://simonwillison.net/tags/generative-ai/)[llms 1,826](https://simonwillison.net/tags/llms/)[anthropic 304](https://simonwillison.net/tags/anthropic/)[llm-tool-use 72](https://simonwillison.net/tags/llm-tool-use/)[coding-agents 218](https://simonwillison.net/tags/coding-agents/)[pi 5](https://simonwillison.net/tags/pi/)
### Monthly briefing
Sponsor me for **$10/month** and get a curated email digest of the month's most important LLM developments.
Pay me to send you less!
[Sponsor & subscribe](https://github.com/sponsors/simonw/)
* [Disclosures](https://simonwillison.net/about/#disclosures)
* [Colophon](https://simonwillison.net/about/#about-site)
* ©
* [2002](https://simonwillison.net/2002/)
* [2003](https://simonwillison.net/2003/)
* [2004](http
URL Source: https://simonwillison.net/2026/Jul/4/better-models-worse-tools/
Published Time: Sat, 04 Jul 2026 22:54:55 GMT
Markdown Content:
# [Simon Willison’s Weblog](https://simonwillison.net/)
[Subscribe](https://simonwillison.net/about/#subscribe)
**Sponsored by:** Sonar — Gartner just named Sonar a Leader in the 2026 Magic Quadrant™ for Technical Debt Management Tools. Read the report and learn how to measure and remediate technical debt across your codebase. [Get the report](https://fandf.co/3STP9vD)
4th July 2026 - Link Blog
**[Better Models: Worse Tools](https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/)**. Armin reports on a weird problem he ran into while hacking on Pi:
> The short version is that newer Claude models sometimes call Pi’s edit tool with extra, invented fields in the nested `edits[]` array. And not Haiku or some small model: Opus 4.8. The edit itself is usually correct but the arguments do not match the schema as the model invents made-up keys and Pi thus rejects the tool call and asks to try again.
>
>
> That alone is not too surprising as models emit malformed tool calls sometimes. Particularly small ones. What surprised me is that this is getting worse with newer Anthropic models as both Opus 4.8 and Sonnet 5 show it but none of the older models. In other words, the SOTA models of the family are worse at this specific tool schema than their older siblings.
Armin theorizes that this is because more recent Anthropic models have been specifically trained (presumably via Reinforcement Learning) to better use the edit tools that are baked into Claude Code. This has the unfortunate effect that other coding harnesses, such as Pi, may find that their own custom edit tools are more likely to be used incorrectly.
Claude's edit tool [uses search and replace](https://platform.claude.com/docs/en/agents-and-tools/tool-use/text-editor-tool#str-replace). OpenAI's Codex [uses an apply_patch mechanism instead](https://developers.openai.com/api/docs/guides/tools-apply-patch), and OpenAI have talked in the past about how their models are trained to use that tool effectively.
Does this mean third-party coding harnesses like Pi should implement multiple edit tools just so they can use the one with the best performance for the underlying model the user has selected?
Posted [4th July 2026](https://simonwillison.net/2026/Jul/4/) at 10:53 pm
## Recent articles
* [Have your agent record video demos of its work with shot-scraper video](https://simonwillison.net/2026/Jun/30/shot-scraper-video/) - 30th June 2026
* [Porting the Moebius 0.2B image inpainting model to run in the browser with Claude Code](https://simonwillison.net/2026/Jun/22/porting-moebius/) - 22nd June 2026
* [sqlite-utils 4.0rc1 adds migrations and nested transactions](https://simonwillison.net/2026/Jun/21/sqlite-utils-40rc1/) - 21st June 2026
This is a **link post** by Simon Willison, posted on [4th July 2026](https://simonwillison.net/2026/Jul/4/).
[armin-ronacher 24](https://simonwillison.net/tags/armin-ronacher/)[ai 2,102](https://simonwillison.net/tags/ai/)[openai 426](https://simonwillison.net/tags/openai/)[generative-ai 1,859](https://simonwillison.net/tags/generative-ai/)[llms 1,826](https://simonwillison.net/tags/llms/)[anthropic 304](https://simonwillison.net/tags/anthropic/)[llm-tool-use 72](https://simonwillison.net/tags/llm-tool-use/)[coding-agents 218](https://simonwillison.net/tags/coding-agents/)[pi 5](https://simonwillison.net/tags/pi/)
### Monthly briefing
Sponsor me for **$10/month** and get a curated email digest of the month's most important LLM developments.
Pay me to send you less!
[Sponsor & subscribe](https://github.com/sponsors/simonw/)
* [Disclosures](https://simonwillison.net/about/#disclosures)
* [Colophon](https://simonwillison.net/about/#about-site)
* ©
* [2002](https://simonwillison.net/2002/)
* [2003](https://simonwillison.net/2003/)
* [2004](http
DeepCamp AI