MCP is "dead" - or is it?

Key Takeaways

Today, I want to talk about how I'm saving anywhere from like 60% to 90% of my tokens for tool calling. And the solution is actually dead simple, and honestly, you can implement it anywhere. The two methods I'm going to mention are actually live in Anything LLM today, but like I said, if you have a favorite tool, you can do this, or you can actually ask the maintainers of whatever tool you use to implement this. It will work anywhere and kind of any place that uses tools, or especially MCPs, this will help a ton. And if you're not a technical person by any means, you just use these tools, I urge you to stick around because I'm not going to do a super deep tech dive on this. I'm going to link technical stuff in the description if you are interested, but in general, this is just an idea to help people understand tools. What even are tools? And why does the thing that I'm doing save so many tokens? If you haven't been on the channel before, my name's Timothy Carambat, I'm the founder of Anything LLM. It's this all-in-one open-source tool to basically give you kind of like a ChatGPT-like experience entirely on your device. We focused primarily on local models. So, all of my kind of concerns are about optimizing for local models. I honestly believe that local models can give you a cloud-like experience on consumer hardware. So, that's like my whole focus, and also where this kind of idea even comes from. When you use a tool like Cursor or Claude Code or Code X or whatever it is that you use, and it reads a file or searches the web or does literally anything, that is a tool. That is also technically now an agent. An agent is an LLM that can call tools. So, that's all that is, by the way, if that was confusing ever. But since tools make models useful, one would naturally assume, why not just have infinite tools? Because then my model can do infinite tasks. You would actually be mostly right, but that is actually the problem that I'm talking about today. Because every single tool is tokens. It's as if you typed it in the prompt yourself. So, you actually can't have infinite tools because it just takes up infinite context, which we can't have with our current kind of model architecture. So, like take this example. If my prompt was, "Make that an issue on GitHub." My model is going to look at, I don't know, let's just say I have 400 tools available. Because it's my model, I want it to be able to do anything that I ask. It will look and hopefully properly identify that the GitHub issue creation tool is the tool it needs. Now, that's great because it created my tool. The issue, however, is that all 399 tools it didn't call just got wasted. Totally wasted. For local models, this means worse performance and a slower time to first token. So, basically, worse performance to the regular person. On cloud, it's actually even worse because you will not only get a performance hit, but you also just lit money on fire for no reason. Because with cloud, you pay for both input and output tokens. So, it's actually a double whammy on cloud. In late 2024, Anthropic came out with this open-source framework, essentially, that was going to make, or hopefully was supposed to make, building tools so that regular people could pull them in and get access to doing these cool tasks with their models called Model Context Protocol, or MCP. You probably heard about it. It was all the rage for quite some time. And candidly, I think we're actually going to circle back to MCP just from how things have been panning out with the skills whole push, which if you're not familiar with skills, bless your heart. But the idea with MCP was have a common, unified interface so that developers can quickly build tools and give those to anybody else. And then tools like LM Studio or Anything LLM or Ollama can use these so that we aren't rebuilding the wheel every single time someone wants to read from their Gmail. And I will be the first to say, when this came out, I was its number one hater. I still don't think this is the best solution, but it is widely adopted, and it's actually become a huge token waster now. Not as much as skills, but certainly still a problem. And if you're wondering my thoughts about skills themselves and how they basically waste hundreds of thousands of tokens and why they will never really work for anything except tools like Claude Code, I actually have a video on my channel about this. So, I'll link that. One of my main gripes about skills that's actually pertinent to this video is reliable execution. In tools like Claude Code or Code X or Open Claw, basically everything's a prompt. The good news about this is that if you need an output like, I don't know, you want it to send an email, but then maybe every now and then you also want to do like 12 other things before that email, a tool like that uses this skills paradigm is super flexible. It's basically a hammer and every problem is a nail. The problem is that there's no guardrails to have the model consistently do the same thing every single time. You can prompt it, but that doesn't actually guarantee any level of success at all. And there's a reason for that. Models themselves, local and cloud alike, are essentially a black box. And if you use a black box to basically build a black box, you may get the right output purely just on the model's intelligence alone. But today's execution is not tomorrow's execution. Like imagine if you had a calculator where on days that end in Y, but only on the third day of the month, 2 + 2 is 9. Like that is not useful at all. But that is actually something that can happen if you use tools like skills in Claude Code and Code X and all these other tools because the model provider can change the rules at any time. There's a concept called quantization, which basically makes the model slightly dumber, but then also makes the model easier to serve to hundreds of thousands of people at the same time. There's also a thing called hyperparameters, which are basically the rules or the config for the model for everyone. You normally can't change these, especially in tools like Claude Code. So, the model maker can just edit these at any time, and it can help save them money. Maybe they want different inference speeds. Maybe they want to get you off of an old model and on to the newest model. Well, then good, we'll just quantize the old model, make it too dumb to where you have to use the new model. To drive this point home, if you go on Reddit and just search, "Did Claude get dumber?" you will always find posts of people saying that Claude has gotten dumber. This is not collective hysteria for the most part, and this is Anthropic or the model provider tuning those little details to save on money, to serve faster results, to do all this stuff. This is something that's out of your control totally. Compare this to something like a code-written skill, which would be an MCP or something that you write yourself. This is a tool that is dictated by code. The only LLM process is right here, where you ask the model, "If you're going to call this tool, it has to look like this every single time. If it doesn't, the tool is not going to work." So, the models are typically very good at this, and it will usually make something like JSON. The benefit of this is that if you have a reliable input and you send that through your code, which you have written or someone else wrote, where an input always equals an output, then you will get the same thing every single time. The issue is that your code, unless you tell it to, cannot do any of these side objectives. So, you basically have a less flexible tool, technically, but in my opinion, this is what most people want from a tool. I don't want to ask a question, and today it cost me 100 tokens, and tomorrow it cost me 10,000, and then the next day cost me 500. It would be great to know that if I do this, it will cost within a percent of this many tokens cuz that's how many the input params are. And if it goes into my code, I am going to get the answer I want. This idea of like MCPs and code-written tools are super token efficient. They're just not flexible, and that is why so many people have moved to skills or attempted to move to skills, and they're now only finding out that this is not going to work. An example of this not working is basically this Reddit post that I think is hilarious. It was posted 3 days ago, and it says, "It cost you 2% of your session usage to say hello to Claude." Now, why is that? That is because everything is just a prompt. And if everything is just a prompt, you're sending everything with your question, which is why the bloat is so bad. Okay, so clearly skills are not the way we should go about doing tool calling. Maybe we'll go back to MCP. Well, this is the first solution that will help you save a ton of tokens, and there's only one other provider out there who I've even seen do this, but let's get to the issue. So, I have two popular MCP tools defined right here, GitHub and Atlassian. Together, there are 114 tools. Just from me loading these, if I put this in Claude Code, if I use these two MCPs in Claude Code, I am sending 114 tool calls to Claude Code every single prompt. Every one. And that may not sound like a huge thing cuz you have a million tokens of context until you actually look at the tokens or estimate tokens of what this is going to cost. This would mean that every single chat, whether I use these tools or not, I am sending 34,000 tokens worth of potentially useless tool calls. These tokens count against your usage, and if you use the API, you're literally paying for them. This is a huge amount of waste. So, let's talk about people who are doing this right. In LM Studio, you can load in MCPs. And if you look on the sidebar, you can see these little checkboxes. If you uncheck a box, that tool no longer goes to your model. This is hugely useful if you know that you will only ever want to read information from GitHub. If you were doing this with any other tool, being able to gradually turn on and off into the tools that you know are useless, that is a huge token savings right out of the gate. In fact, in Anything LLM, we even tell you if an MCP is going to be a problem by estimating the amount of tokens with your model and suggesting, "Hey, you should turn off some tools because you're just going to be wasting money or time or inference." Aside from LM Studio and Anything LLM, I do not know a single other tool that is adding this very simple UI kind of feedback to save people tons of money on tokens. And the reason for this is that MCP itself does not have this. The client itself has to go out of their way to support this. But there is one additional thing, and this is really the kind of impetus of this video. Anything LLM has added a process I'd like to call intelligent tool selection. Sounds fancy, it's super easy, and you can go and ask or build this in your favorite tool or ask the maintainer to add this as a feature. And it is where you will save thousands on tokens. Intelligent tool selection is dead simple. It is basically a preprocessing step that adds a tiny bit of overhead that saves you thousands on tokens. So, for example, if I were to say, "Make a GitHub issue from that email from Tom," right? We may have a ton of tools available. Searching the web, Notion, iMessage, who knows? I have GitHub and I have Gmail. From those two tools, they both have a ton of other tools inside of them. I only need creating an issue and reading an email from Tom. So, why don't I just filter everything else out and then give my LLM a token window that basically says, "Hey, here's my prompt and the history, but there's only really two tools that you should call. So, just call those." This will save you literally thousands on tokens. In the description, I will link the exact place that we do this in Anything LLM's code base so that you can give it to whoever maintains your tool of choice, but the idea is just reranking. Reranking is a specialized super tiny model that basically takes your prompt and then takes the function name and the description of what it's supposed to do and simply just grabs the top 15 or 20. So, for example, this text right here is straight-up out of the server logs for Anything LLM. I had 114 tools. It took me roughly 2.2 seconds. I'm using a local model, but I went from 18,000 tokens to 1,800. I saved 90% on just tokens. And this is configurable in Anything LLM, so you can mess with it. But this process right here, I have no idea why this is not the industry standard. If you have better hardware, the reranking will run faster. Uh if you have less tools in the window alone, so if you're I I had 114, but if you have like 50, you will rerank to the top 15 much faster. But the end result is that the models that end up with your LLM to actually do the thing that you want it to do, that is now smaller in tokens. So, on local, you get better performance and faster times, and on cloud, you save money. And from our benchmarks, 99% of the time, the tool or tools, if you're doing a complex multi-stage one, are always in the window. So, the model is going to call them. And this happens, by the way, every single chat. What is relevant for the previous prompt does not mean that that's the only tools available for all future prompts. This actually happens continuously, and it does improve in speed over time. Now, I am sure that this is not a novel thing that just I have invented. We just simply built a process around it. That being said, I'm really surprised that you don't see this in a lot of other tools, especially tools like Claude Code, where this would benefit so many people. But either way, I hope those two tips, basically being able to filter tools out of the MCP, if you have an MCP at all, and then just in general, intelligent tool selection or reranking or tool reranking or whatever you want to call it. Those two tips alone are going to save you 60 to 90%. Depends how many tools are in your window. If you have a ton, then you save more. If you have less, then it has less of an impact. But the end idea is that you stop wasting tokens. LLMs do not have infinite context. Every token has a cost, sometimes literally. This is what I was hoping to solve with these two approaches. I'll try to link everything that I cited or anything that's relevant in the description, but let me know what you think of this. Uh I may have missed something crazy. And if you have a unique approach, I want to hear about it. But in general, I think we need to all start being a little bit more mindful about what it means to add context into a chat. If you enjoyed this video or you found this kind of approach actually genuinely useful, then you can subscribe. Uh that would help me a lot, but in general, I just appreciate you listening. Thanks.