Agentic test processes, LLM benchmarks, and other notes on agentic coding fr
📰 Hacker News (AI)
Learn how to leverage agentic coding and LLMs for testing and debugging, and understand the importance of human oversight in AI-assisted development
Action Steps
- Use LLMs like Codex to automate testing and debugging
- Implement a data-driven approach to find and fix bugs
- Create a pipeline from support ticket to pull request
- Review and verify AI-generated tests and results
- Consider using agentic loops to improve testing efficiency
Who Needs to Know This
Developers and testers can benefit from using agentic coding and LLMs to automate testing and debugging, but must also be aware of the potential pitfalls and need for human oversight
Key Insight
💡 Agentic coding and LLMs can automate testing and debugging, but human review and verification are crucial to ensure accuracy and quality
Share This
🤖 Leveraging agentic coding and LLMs for testing and debugging can improve efficiency, but don't forget the importance of human oversight! 💡
Key Takeaways
Learn how to leverage agentic coding and LLMs for testing and debugging, and understand the importance of human oversight in AI-assisted development
Full Article
Title: Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island
URL Source: https://danluu.com/ai-coding/
Published Time: Wed, 08 Jul 2026 04:00:32 GMT
Markdown Content:
**Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island** | [Patreon](https://patreon.com/danluu)
* * *
I've been using AI fairly heavily since last November and the whole thing is a funny experience. An agent will do something that, if a human did it, you'd immediately fire them. My reaction, of course, is to act as if this is great and spin up a thousand agents so they can do even more of that.
Mid-last year, I had GPT (maybe 5.0 or 5.1) try to find the source of a bug. Naturally, this code didn't have tests and `git bisect` wouldn't work, and it was a UI interaction bug for which I'm not even really qualified to write a test for, so I asked Codex to bisect between dates X and Y to find the commit that introduced this bug. Codex immediately told me the offending commit was after this date range (which couldn't possibly be correct). On telling Codex this was wrong, it then told me some commit that was obviously also not the offending commit once or twice. On telling it those were wrong, it then told me the offending commit was some plausible looking commit. When I asked it to prove or disprove its theory, it told me that it wrote a test and confirmed that the alleged commit was the breaking commit.
I then asked it to show me by making a video with the full developer end-to-end stack in the normal browser test environment. It claimed that it didn't have permissions to do that (which was a lie), but it could make video of the execution of the repro before and after the commit in playwright with the appropriate test code. The video was convincing and showed the feature working properly before the commit and failing to work after the commit. Something about this didn't feel right, so I tried reproducing the issue by hand before and after the commit and found out that the whole thing was a fabrication. The video made it look like Codex had reproduced the bug, but it was an artificial browser environment that was designed to create a fake repro, not the real environment.
Like I said, because this was non-ironically such a great experience, I immediately thought to myself, "how can I get more of this?" and started using agents more and more heavily until I was using coding agents heavily mid-late last year.
Since this post covers a relatively disparate set of topics, here's a brief outline.
* [Testing background](https://danluu.com/ai-coding/#testing-background)
* [Some details on testing](https://danluu.com/ai-coding/#some-details-on-testing)
* [Caveman mode](https://danluu.com/ai-coding/#caveman-mode)
* [LLM variance](https://danluu.com/ai-coding/#llm-variance)
* [Misc](https://danluu.com/ai-coding/#misc)
* [Agentic loops and writing this post](https://danluu.com/ai-coding/#appendix-agentic-loops-and-writing-this-post)
* [Some reasons people talk past each other](https://danluu.com/ai-coding/#appendix-people-talking-past-each-other)
### Testing background
LLMs are highly leveraged when it comes to testing. In terms of the amount of effort it takes, it's easier than ever to hit a particular quality bar and yet, software seems to be lower quality than ever. A decade ago, we looked at [the bugs I ran into in an arbitrary week](https://danluu.com/everything-is-broken/). There were quite a few bugs then and I run into more bugs now, but I don't think this has to be the case.
For one thing, after a bug has been shipped, it's easier than it's ever been to use a data-driven approach to find and fix the bug. Just for example, at work, I tried creating a pipeline that goes from support ticket (chat or email) to pull request (PR). As far as I can tell, this works ok. Since I work for a company that has a traditional workflow, all of these fixes get reviewed by a human
URL Source: https://danluu.com/ai-coding/
Published Time: Wed, 08 Jul 2026 04:00:32 GMT
Markdown Content:
**Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island** | [Patreon](https://patreon.com/danluu)
* * *
I've been using AI fairly heavily since last November and the whole thing is a funny experience. An agent will do something that, if a human did it, you'd immediately fire them. My reaction, of course, is to act as if this is great and spin up a thousand agents so they can do even more of that.
Mid-last year, I had GPT (maybe 5.0 or 5.1) try to find the source of a bug. Naturally, this code didn't have tests and `git bisect` wouldn't work, and it was a UI interaction bug for which I'm not even really qualified to write a test for, so I asked Codex to bisect between dates X and Y to find the commit that introduced this bug. Codex immediately told me the offending commit was after this date range (which couldn't possibly be correct). On telling Codex this was wrong, it then told me some commit that was obviously also not the offending commit once or twice. On telling it those were wrong, it then told me the offending commit was some plausible looking commit. When I asked it to prove or disprove its theory, it told me that it wrote a test and confirmed that the alleged commit was the breaking commit.
I then asked it to show me by making a video with the full developer end-to-end stack in the normal browser test environment. It claimed that it didn't have permissions to do that (which was a lie), but it could make video of the execution of the repro before and after the commit in playwright with the appropriate test code. The video was convincing and showed the feature working properly before the commit and failing to work after the commit. Something about this didn't feel right, so I tried reproducing the issue by hand before and after the commit and found out that the whole thing was a fabrication. The video made it look like Codex had reproduced the bug, but it was an artificial browser environment that was designed to create a fake repro, not the real environment.
Like I said, because this was non-ironically such a great experience, I immediately thought to myself, "how can I get more of this?" and started using agents more and more heavily until I was using coding agents heavily mid-late last year.
Since this post covers a relatively disparate set of topics, here's a brief outline.
* [Testing background](https://danluu.com/ai-coding/#testing-background)
* [Some details on testing](https://danluu.com/ai-coding/#some-details-on-testing)
* [Caveman mode](https://danluu.com/ai-coding/#caveman-mode)
* [LLM variance](https://danluu.com/ai-coding/#llm-variance)
* [Misc](https://danluu.com/ai-coding/#misc)
* [Agentic loops and writing this post](https://danluu.com/ai-coding/#appendix-agentic-loops-and-writing-this-post)
* [Some reasons people talk past each other](https://danluu.com/ai-coding/#appendix-people-talking-past-each-other)
### Testing background
LLMs are highly leveraged when it comes to testing. In terms of the amount of effort it takes, it's easier than ever to hit a particular quality bar and yet, software seems to be lower quality than ever. A decade ago, we looked at [the bugs I ran into in an arbitrary week](https://danluu.com/everything-is-broken/). There were quite a few bugs then and I run into more bugs now, but I don't think this has to be the case.
For one thing, after a bug has been shipped, it's easier than it's ever been to use a data-driven approach to find and fix the bug. Just for example, at work, I tried creating a pipeline that goes from support ticket (chat or email) to pull request (PR). As far as I can tell, this works ok. Since I work for a company that has a traditional workflow, all of these fixes get reviewed by a human
DeepCamp AI