The BEST Local LLM for OpenCode? Find it with opencode-benchmark-dashboard

Key Takeaways

Hi everyone. In this video, we'll talk about Open Code Benchmark dashboard, which is a tool that I developed. And yes, it is useful mainly if you want to compare large language models performance on your computer. And yes, the current large language models that runs on your computer are very powerful. And they can also buy things for you on Amazon with the right integrations. And yeah, I I don't have a very fast computer, so it took 23 minutes and 33 seconds to do this operation, so to search something on Amazon and add it to to the cart. Uh but yeah, the point of this video is um an evolution of a video that I already made on my channel, which is this one. Uh this video was about best performance and trade-off between accuracy and speed. But I did this test with uh normal LLMs. Uh this time, I'm using Open Code. And Open Code it's a a more advanced way to use large language models. And yeah, with Open Code uh if sometimes the model fails, you can retry and it is done automatically. Uh because uh Open Code um doesn't just answer to your questions, to your prompt, but uh it uh runs uh and execute the commands uh on your computer. So, this is the the project. And um so, why to use this? Obviously, uh to to find the best trade-off for your your your use case and your hardware on your computer. And you will generate a chart like this. So, these are online models. And these are uh local models in my hardware, which is just a CPU. It isn't uh very fast. And here, you have uh fast and accurate models. So, in the in the first first quarter, you find the most interesting uh uh LLMs. And yeah, you decide the prompts to to test the LLMs. And um another important thing is that often people talk about token per second. But I don't think it is a very relevant metric because some models reason a lot. So, it means that they spend a lot of time thinking and not finding the solution. Uh so, if you find models which have uh less reasoning, you also improve your speed, your token per second. Because they reach the solution faster. So, with this tool also, you see useful tokens, not just token per seconds. So, um yeah, there are scenarios where small LLMs uh can fix themselves using the commands on your computer. So, they could find faster the solution for your tests. Uh okay, uh how can use it? You download this on your computer and you will find the link in the description. So, you install the dependencies for uh Bun Node.js runtime. You should have already um Open Code configured on your system. And with this tool, you can um use local LLMs, but also remote LLMs from Open Code or from other providers. So, you can also compare different providers with your um prompts. And you run like this. So, Bun run answer the name of the model, which is a an online model, this one, but it depends on your uh Open Code models command. So, here you have the list of the available models in your system. And you can also configure it in this path. So, in dot config open code open code dot json. Here, you can put your OpenAI compatible models here. Uh so, you run it with answer and it will generate um the the answer. So, in prompt answers, you can see that you have the answer. And then uh you can evaluate the answer. And um the evaluator model can also be another one. So, it isn't important. Under config benchmark.json, you can replace the evaluator model. And this evaluator model decides if the answer is correct or not. So, it's it's more advanced that just um see exactly the the answer. It can also decide that the answer is correct also if the wording is different uh to reply. And you can also run the test for a single um test. So, with minus t, you can decide a single test just to rerun that. And you can also evaluate again just um a a test. And with Bun run dashboard, uh after you finish the execution of answer and evaluation, you can uh run Bun run dashboard and you get something like this one. So, it's an anti interactive dashboard dashboard with all your tests. Uh you can also filter if you want to see just some models, not every model. And this is uh my situation. So, here uh I have my personal tests. So, about coding, about data extraction, knowledge, reasoning, and so on. And here uh it's all the models that I tested. The models with llama.cpp are my local models that I use. And I also have Open Code models, which are the ones that Open Code provides for free. And I also tested some models on OpenRouter, which is another integration of a remote provider for LLMs. So, you can also see if remote providers perform in the same way or you can compare local models with remote models. So, like uh Nemotron. This Nemotron is from OpenRouter. And uh this Nemotron is from my computer. But in my computer, I use a quantized version. So, as you can see, um as you can see, the results are a little bit different. And the remote version also is faster. But the remote version is also less quantized. So, we have some more green uh results. And yeah, this could also be useful to see if you can use a quantized version or not for your use case. And this is um my situation. So, these are the remote models that I test. And at the top for accuracy and uh speed, I have Step Fun Step 3.5 Flash, which is uh a very good model that you can test now on OpenRouter. And the other one is Open Code GPT-5 Nano. It also performed quite well on my tests. And yeah, but these are online in my computer. The best result I had is this one with Lama CPP. When 3.5 35B A3B. So three three billion parameters active. It was the most accurate on my computer and it also quite fast. And as you can see I tested other models that are bigger, slower, but they do not perform as good as this new model from When. Other good models that I mentioned in the past are this one. So Namondron 3 Nano 30B A3B. And GPT OSS 20B. These are also quite good and quite accurate. And if I want something more for data extraction and yeah, less accurate but also quite acceptable, it's this one. Young Code 4B and this is the quantization Q4KM. So this is the situation in in my computer. As you can see I have many more models that I test. And also this one BU 30B A3B. It's good. Yeah, it depends on your use case. Some models are good for agentic coding. And yeah, data extraction. But if you need more knowledge of the world you have to use bigger models that it's it's normal. And this one is Namondron 30B A3B. Okay, this is the project. So Open Code Benchmark dashboard. You find the link in the description. And let me know in the comments which model do you use for local AI. That's it. See you in another video. Bye.