Introducing HackerRank ASTRA: A Benchmark for Real-World AI Coding | Rafik Matta, EM, HackerRank

Key Takeaways

You've just seen some of the amazing AI based features we're releasing across our products. Well, I'm excited to talk to you about something else we've been working on. There's been a lot of rumors floating around our industry from some prominent figures that AI is about to take our jobs. Well, at Hackerank, we believe that the future is human plus AI. We're already experts at assessing and educating developers, and we figured our next challenge should be to assess AI models for the real world. What do I mean by that? Well, we've noticed there's consistently been a gap between how models perform on current benchmarks versus how they perform in the real world and on the job. That's why we decided to build our own benchmark. And I'd love to introduce to you Hacker Inank Astra. Hackerank Astra's initial release consists of 65 project problems. These problems are complex. They haven't yet been seen in the training data of any of the models and they represent true real world end-to-end engineering tasks a developer would see in their day-to-day job. We focused on assessing correctness and consistency. Come check it out at hackerank.com/ai/astra. Our leaderboard is live and we're adding models as they drop. But we're not stopping there. AI agents are quickly becoming the interface between models and the real world. We're already successfully using them in our product and engineering practice and we're finding them really powerful for our research efforts as well. Let's check out some of the cool stuff we're doing with Hacker and Castro. We figured that the best way for us to evaluate models in the real world is to simulate the real world. So, we decided to develop an AI product manager and an AI developer. We get them to talk to each other and try to solve real problems. Let's see what that looks like. Going to select the problem. Well, this one's a good one. I'm going to select the model. This model's for the software developer. Once I start the session, the product manager is going to pose a problem to the developer. It's poorly posed, so they're going to keep chatting with each other for a while. Eventually, the developer is probably going to start building something. In this case, it looks like they're building something for the back end in fast API. For this problem, it's a real-time analytics problem. So, it needs to have some event streaming in place. They're going to take some time and they're going to develop this code. And eventually once they're done, they're going to run some tests. There we go. The tests passed and then the product manager is going to offer some feedback. They're going to keep going back and forth for a while until they finish the session. So, let's see what that looks like when the session ends. Once the session ends, we generate a report for how well the the software developer did. On top of correctness and consistency, we've added three more dimensions around efficiency, cost, and communication. That way, you get a well-rounded picture of how the developer did. For all you model builders out there, we're releasing an Aantic benchmark with these two agents you just saw, a full suite of real world problems, and a fully automated evaluation harness. And we're just getting started. Come check us out on our hugging face space. We want you to help us evolve Hacker and Castra. Thank you. Back to you, VC.