SKILL.MD is Not Enough

Key Takeaways

Hello community. So great that you are back. Everybody is talking about skills, strategic skill libraries and everything. So let's have a look at the latest research. And the title says it all. A skill markdown is not enough anymore. Now this is now part two. Part one you find here in this particular video where we look at simple multiskll MD configurations. But this is here what I would like to present to you today. At the end of this video, you will completely understand this. We have a look at knowledge sources. So in my video yesterday, we looked here textbooks and question answer at Mathematic Corpora. And I showed you there as a new policy optimization. And we have here a new training time AI optimization where we have an internalization into the model weight. And we hope to outperform the human intelligence with this kind of additional self-training and self-improving AI knowledge. Now the new stuff today is here in green the first one and then the third one. So two additional paper and a short outlook how it will evolve further. Okay. First study this is here March 12, 2026. This is Yara East China Normal University, Shankai Innovation Institute, University of Science and Technology of China. And they go now for automating the skill acquisition through large scale mining of open-source agendic repos, a framework for multi- aent procedural knowledge extraction. And guess what? If you see repos, yeah, they go for the GitHubs. So they ask a simple question. Hey, instead of making our LLMs bigger, what if we just taught them new tricks by reading here the I don't know top 10,000 GitHub repos or top 100,000 GitHub repos in your particular domain. And we built then new intelligent agents here with I don't know 50 beautiful skills that we extract from GitHub. We have a codebase and we can immediately verify this. And if you want to integrate it, we have here a reinforcement learning with a verifiable reward structure. Now they use this now here in a new pipeline. I'm going to show you to extract here all the procedural skills and coded now as structured skill MD files, markdown files that you know. And this hopefully will augment the agent without touching a single model weight. So you don't have to do any supervised fine-tuning or any reinforcement learning of your model. You just go here in the in context learning in your context window. You put here multiple skills. Is this working? Now they tell us here and they had a case study here on aum explain agent and code to video and they demonstrate it has a 40% gain in the knowledge transfer efficiency. Okay. So let's have a look. Now the deeper point here is rather simple. Now it [snorts] implicitly augugust that an open source software ecosystem like GitHub or you have heard that openi is maybe trying to duplicate here the effort of a new GitHub has already incorpored enormous quantities of domain expert procedural knowledge. So why not use this? Why not mine this resources systematically, safely and at the scale that is really here interesting? We create new engineering challenges here that are now open to the eye research community. Let's have a look now. Yesterday in this video here, this is the thumbnail of my video here and this is here especially where we had a deep dive. You see our source were a chemistry textbook or a textbook on theical physics or whatever. Yeah. And then we had a huge realm that extracted now particular skills. But since this was here I already put in realm. So you understand it was text base but we had here also some graphs or some interpretation here visual explanations. Great. So now we take here now the additional step and now we say okay so we have the verbal the textual the linguistic explanation maybe a graph explanation and now all that we need for the verifiable reward function of our reinforcement learning is we have to have the code and if you go now let's say one top 10,00 GitHub repos in your domain let's say theoretical chemistry we extract now the skills from the code repos then we have a beautiful code implementation of I don't know the top 100 skills that you have to have if you do something in chemistry. Now just want to show you they give us here some beautiful examples. Uh they are basing their examples here on many I was not familiar with this. This is a community maintained Python library for creating mathematical animations and I just want to show you the two examples that they list here in this particular paper. So just to show you the theorem explain agent here this is toward a video based multimodal explanation for LLMs with Ethereum understanding. So what we have if you look at this you see here rendered videos examples. So depending where you are working with here for example the gradient no here you have a beautiful the mathematics then here you have a visualization or with brownian motion here you get immediately [clears throat] understanding of what is happening here. So you have a beautiful abstract then they explain to you how it works pure with the theorem a planner agent a code agent and then they render here the video beautiful you can run here the experimental results then you have a human evaluation network great and you even have some performance benchmarks interesting if you look at this you see here for the visual consistency a set 3.5 was really excellent here and if you go for accuracy adapted. Gemini 2.0 was great. So you see exactly where we're moving. Yeah. And then you have different videos that are now generated and that explain now complex mathematical complex physics or whatever you have now here as a video that is generated here by this AI system. And the second example they show us is code to video. So this is a codecentric paradigm for an educational video generation. And you have everything code AR of PDF the data set and they also show you here no code to video. So you do have here beautiful code but you want to make it explainable. You want humans understand it too and not just if you are the best coder in the world but can you give me here a visual interpretation what is happening introduction to neural networks or whatever you like. how diffusion models perform on this video. And you have here VO3 van code to video and they want to show you here that their code to video application here is one of the best visualization of complex topics. You can go for space filling curves. You see here how the system interprets this and generates here creative visual explanations. Abstract is available the explanation. Wow, this is okay. How it works beautifully. So those are the methodologies that they authors show you in the study and they apply here their methodology on those two apps and you have all the details in the original paper. So now you might ask hey how do you do this? What is the methodological framework here for the skill extraction of the GitHub repos? Now they show us in this paper the systematic acquisition of skills from the GitHub repos. They have a multi-stage pipeline that transform here the monolithic code base into modeler skill markdown artifacts. So they have three stages. The first is a repo structural analysis by an LLM. Guess what you know this. Then interestingly the semantic skill identification on this codebase and then the translation. So we will focus here on the second point. This is the most interesting point. The authors do the semantic skill identification through a dense retrieval. So to identify here the latent skills they go here of a two-stage ranking problem combining now a dense retrieval and a cross encoder refinement. And this is something that we already know. Now the dense retrieval stage is simple. You have an extraction agent that encodes the task description and the code modules into a dense mathematical vector representation using here our old friends a by encoded structure from the transformer and then we have a binary ranking stage. So now we use now a cross encoder ranker and then you have a re-ranker if you want that performs now the fine grained relevance assessment by jointly encoding here the task model pairs and producing here relevance scores. So what is the best GitHub repo for a particular subtask? If you want to see this here in detail and you want to have a GitHub repo this is the only paper that I found. This is by Edinburg Naper University the school of computing and they go here in detail from the retrieval to the ranking a two-stage neural framework for automated skill extraction. So if you have a better paper you found a better paper please leave just a note here in the comments to this video otherwise you can go with this one. So the mathematical optimization is interesting because here the orus use the dense retrieval we have our bi encoder we have our cross encoder but what is the main task the main task is an alignment finding the right module for my task description in chemistry in physics among thousand and thousands of existing files of marketplaces and whatsoever. Now, if you just remember my last video yesterday, the authors there used here also a reinforcement learning and new policy optimization, but their challenge was different. They wanted to go for a composition. They wanted to figure out how to combine the skill A and a skill B to maximize here the difficulty bonus without breaking here the logical validity. They wanted here to build more complex training data sets so we can train our future AI system on training complexity that go beyond the human knowledge. So in the video yesterday I showed you how to build a self-arning AI that is not integrating here the human knowledge anymore in the loop but a self-arning AI that goes to higher and higher complexities. And I showed you how to build here exactly the flow structure of this. Now of course think about we also need to import external data. And I got some comments here from my viewers that say what about rag? It is easy. It is beautiful. It is a complimentary layer. So when rack says more or less hey here is the information about a particular module or particular fact or knowledge X the skill just says here skill MD says and here is a proven workflow for doing X on the one hand we have the information about X on the other hand the skills give us now the real implementation if we're doing X so we have the code like say from GitHub like I showed you in this video or in my yesterday video if you want to go for theoretical textbooks and have here a detailed explanation here in English or whatever. By the way, did you notice that we are already multimodal because I was already talking about vision language mon so let's do this now professional now luckily just 2 days ago March 12th 2026 we have here from Hong Kong University and university of science and technology a new publication about X skill so we have now skills in multimodal agents and we have exactly the same topic about continual learning. So maybe even a selfarning algorithm. But now we find a new term in the title from experience. And I thought this is interesting. Why we have suddenly experience and skills in multimodal agents? What do we need in the multimodal world that we don't have in the pure textual world? This is the website and this is the code for the GitHub if you want. Now let's start with a very simple example. Imagine you have a brilliant detective who has to solve a complex case like tracking a stolen painting. Yeah. So the idea of the paper is now that this detective keeps now two kind of notes. First a case book of experiences. So what is this? These are short tactical notes like when the painting is photographed at an angle always request a frontal scan before comparing the signatures. Simple trivial I know. And second, our classical procedural manual of skills. The structured workflows like four an art identification case. Step one, step two, step three. Our workflows both are distilled generalizable elements of wisdom. Okay. So let's do this now. Simple example. Now we map it over to a mathematical framework. In order to do this, we have to understand what is the main idea of this preprint. Now I was playing again with the free claw visualization that we have now. So I hope this works. Now if we simplify this, we have two phases. Phase one is the accumulation where we have rollouts where we have skill fragments that we detect where we have then a skill manager and we build a skill library. The second phase is the inference run. So let's have a look at this. You see that here in the phase one in the pure accumulation you are familiar with this. No skill fragments detected. We have a skill manager that builds skills out of the insights from our roll out and the cross roll out. And this is similar here in this video where I showed you how to build multiskll configuration. We had also sequential rollout structure. Great. Now during the accumulation phase, the agent simply runs multiple attempts on training tasks. Sometimes it fails, sometimes it succeeds. and an intelligent knowledge manager and AI system extracts here both types of wisdom our skills and our experience that we have. So we end up with an experience bank that we call E for experience and a skill library K. So skill library you're familiar with the new thing is exactly the experience bank. Why? Because if we go now for the interference run what's happening during the inference when a new task arrives the system now decomposes my complex task into let's say four simpler subtask with a lower complexity retrieves relevant experiences and skills adapts them to the current visual context and injects them as a nonprescriptive guidance and then the agent the our AI agent can then choose choose to follow it, to adapt it, to override it, whatever. So you see we have if you want from the experience bank, we have a strategic knowledge and the skill bank is more or less here the real code base how we have to um construct here the particular specific workflow for this job. Okay, let's have you an example. This is a screenshot here from the paper and they said okay what we have we have a tool set you know python code we have web search image search and we can visit something then we get this image and the quer is now what is the prototype of the two mascots in the corner of the picture and if you look at this you say I have no idea I don't see anything so they now simulate here the gantic reasoning without their new methodology no step one I can see the mascots in the upper left corner oh my god Yeah, here they are. The objects are too small. I could use zoom to zoom in first. Okay. And then step two, the muscles are present in the image. I can tell that the prototype is a bird. Okay. Failed for not converting a normal viewing angle. If you look at this, the agent identifies this as a bird. And you see it goes on. It just fails here to understand exactly what is happening. it has not the right idea how to handle this on a strategic level. No, but then we have now the agendic reasoning with excill. So what we have we have here our recall of the previous experiences no of the self experiencing here in the interaction with the environment. So our experience two is when failed to identify something check if the image needs to be converted or experience three if unseen visual knowledge needed use image search. So with those information we link now to skill fragments like the tool template 2 image.rotate or image.crop and we have a workflow crop and then search. So you see it adapts here to the current task tool template one when the image needs to be converted rotated for a certain angle and we have identified our workflow and now since we have here the experience the skill fragment and a task workflow definition they do it and they succeed. Okay. So this is the example by the artist. This is if you look at this you say hey I'm familiar with this. No because exactly this previous experience what they call an experience we already encountered this and this is here also some people put this in the skill MDs. Are you familiar with this? I showed you in this video when we looked at Entropic's financial service plugins and we looked at the competitive analysis skill MD file. You saw that entropic optimizing this here on real world cases found out that it had to place some instruction like if prompt lists seven competitors include all seven competitors not five or six. And you see this is more or less if you want also a pre- idea of this experience base. No when failed to identify check if the image needs to be converted. So those are not explicit uh workflows or some templates that we have but let's call it a strategic idea how to handle if we encounter a problem. And here we have also instruction that are kind of similar. So trying to sum this up, what can we say about Xkill? They are enabling a training free, you remember this is your in context learning a knowledge accumulation from visual tool interactions because they tell us here in the very first sentence they tell us here the multimodal agents we have today. This is here March 12, 2026 can now tackle complex reasoning task with diverse tools. Yet they still suffer from inefficient tool use and inflexible orchestration in open-ended settings. So if you think that our multimodal agents are already perfect today, so sorry to tell you the authors disagree. So this framework is simple. As I told you, we have a phase one and a phase two. And this is it. So we have our roll out. We have here the syncing process for all our different tools that we have. We have a tool template extraction, the workflow extraction that we find and we build here our skill library with our markdown file. This is the classical way. What is new is that during the roll out you have also inside here what is working, what is not working, what methods [clears throat] can we have, what experiences did we have last time maybe so that now we have a faster learning process. So you have insights here from comparison from the last time or from your cross roll out critique. So what you build? You build now an experience manager very similar to the skill manager you know and more or less you do the same with the experience items. No similarity filter you don't want a double quantity filter judgment. What is the best experience here? What is the complexity of experience? And then you build here right next to your skill library an experience bank they call it. Beautiful. So when now the infer it's time here to run a particular task. So we have the inference run solving now the task with the experience and the skills. This is here a straightforward thing. And of course since we have here directly here a feedback we can have a skill adaptation. We have a system prompt injection. And therefore we can optimize here our task. So if you want it also has here the genome of a self-arning agentic system. Beautiful. Now for the mathematical framework you're not going to believe it if you've seen my last video. Again we go for a partially observable mock of decision process. So everything that I showed you in my last video the mathematics more or less accounts exactly here. They have a different nomenclature. to have different elements, different norm vectors, but it is more or less absolutely identical. So therefore, let's have a look at the results. And the results are interesting. Now they go with a Gemini 2.5 Pro and Gemini 3 flash GBD5. And you see immediately Xkill outperforms every single time. No. Okay, not here, but otherwise Xcale is just gorgeous. Now this is exactly [clears throat] what you expect. However, there's some negative inside I would call it. Yeah, because look, those were all our proprietary models, Gemini 3 or GBD5, the huge mile. Let's have a look at a visual toolbench benchmark. Let's go here with pass it for. So, here we have an X skill. Let's say Shaminai 64. How does it compare to an open source model 64 visual toolbench posit 4 20 and this is a 235 billion vision language model Q and3. So the distance that we have in our vision language model with even here 235B is 60 compared to 20. Now you say okay if you go with a 32B I think this is interesting that here this new method X skill is almost on par you go from 20 to 19. So you say okay so here you see that this particular training of this vision language well the 235B did not really was not really optimized for this particular task. No because the jump from 19 to 20 and the M size here goes up. Wow. So this is here strange but it tells us here open source models currently for the visual task are not as competitive as you look for Gemini. Now you can see okay GBD5 here has also only 37. Yeah I know but in general this is a huge difference between propriatory and open source. Okay, now there's another insight I find fascinating and it's buried here in the table four and we are talking here about tool usage distribution and we go again with the visual toolbench here beautiful so we have the code search here and yeah visit never mind what I'm interested is this one how important are experiences what if you say listen I just go with my skill MD file I don't care about this new insight from this strange scientific papers. Leave me alone. If you would do this and let's just look at code. So skills alone produce almost no change in the tool selection distribution success. Without the tools you have 66 and with skill only activated you have six. Yeah, also almost 66. So just add here the skills is not really helping the model in its performance for the tool usage distribution on the benchmark tests. But just look what happens if we do now and we have now the experience our experience bank included. So if you want our strategic knowledge here of the skill utilization we jump now to 74. So this is not important. So skills are not really this performant for the systems. But my goodness, we missed out here on the experiences. So experiences not skills are responsible for the behavioral shift of our systems and the experience and code here. Let's call it a tactical knowledge. No. So for this type of problem visual problem that I have that I give here to my Gemini this type of tool or tool combination is the most effective. This is the content now of my experience. Absolutely fascinating and this is experience only. And then if you combine now skill and experience or this complete new methodology you see you jump up but the jump is not really so significant. So this experience has an extreme important role to play. Okay, just wanted to show you. So therefore, and I don't know if this is really generalization, but let's say h skills are therefore the correctness mechanism. How to perform a job, how to perform a workflow. And what we're missing up until now experiences are now the strategy mechanism that tell us exactly here on the strategic option that we have and what is this best strategic option and these two together attack you different failure modes that we encounter and are not really interchangeable. So if you see an entropic financial skill MDs where they put a little bit of these experiences or at least the the the failure mode uh stopping here into the skill h this is a much nicer combination and it turns out that we discovered it right now going now from textual LLMs to vision language model well where you see immediately multimodal this is what we need. Yeah, if you're not familiar with the different benchmark here, short summary for you and the tool definitions. I showed you this here in a screenshot. We have the web search tool, the image search tool, the visit tool, and the code interpreter. And those are the different details and the parameters that they used in their models. What I really find interesting, they are really transparent and they give you here also the prompts in the annex of the paper. Have a look. Here you have the system prompt for the multi-tool agent search operation. Then we have the generate a raw skill prompt. Then we have a merge skill prompt. I think this is interesting. No, because each part of the new skill you ask is this part better? Is this part redundant or too specific? Is it complimentary? Is it generally different? What to do? How to cope with this? And then of course the skill manage prompt and they refine here the skill markdown file to remove here the redundancy to generalize your specific cases and improve here the structure of your skill MD file. So hey why not copy this and try it out if you can optimize your own skill MD file for your particular job. Beautiful. Now just some personal insight at the end of this video. I [snorts] think it's really interesting this distinction that we have now because skills MDs if you think about it are task level specific if we break down a complex problem into multiple task or subtask and we need skill for this subtask. So we have defined workflows, we have maybe defined templates how to solve this. But what we were missing are those experience, those tactical knowhows. No, and they operate now on an action level, not on a task level. They are of course conditional and you have multiple tactical options by this experience. But it is important that you don't start here the complete search process but you already have this experience knowh how maybe from your past experiences or you just imported here from I don't know a friend this has some interesting grounding no if you think about skills now they seem analogous to procedural schemas in cognitive science no structured plans for clauses of situation while our experience ences are kind of the episodic prototypes. No, compressed generalizable traces of specific encounters when we encountered problems and we found options how to solve this and then how to use our skills and to use our tools and everything. So it is interesting we go now from the skill term we widen it here going multimodal here to the experiences and the question is what is the future and I showed you this at the very beginning of this video. So here you have our three papers the GitHub repo introduced here at the beginning of this video the Matt corpora the paper two I showed you yesterday in my video. Then we had a look here at the X skills here at the middle of this video and you already see now we have a continued loop accumulate here from the own interactions. We have your skill library. We have our experience bank here and this helps us. So if you think about what is the next step this is here code. This is here text. This is here visual and everything else. So you see we just add more multimodalities maybe we add audio maybe we add video for robotics. So we will convert to something here our world mono our evolution agents selfmining if we have a robotic system from all kind of interaction if the robot is traveling around in our ecosystem in our environment ontological and specific optimization paths I showed you here yesterday but I think in general it is about self evolving systems skills improve from whole episode. It's like in the human case. No, a baby that is experiencing its world. I think something similar will happen to the robotic system. They will become self- evvolving system. They just need a little bit help with the cold start. But otherwise, I think they will become self-evolving. And all of this is kind of converging toward uh oh great wait a second a composable governable but a selfimproving skill ecosystem and I think talking now about an ecosystem of skills and experiences and agents and tool use and MCP and rack system you see it is all melting together. it is optimizing all the different parts are optimized here in their own elements and taken together their interactions. So therefore I think an absolute fascinating topic if we want to go from a simple skill MD for let's say an experience bank and you want to experience this for your task in your domain knowledge if this really improves the performance of your AI system. This is it for today. I hope there was some new information for you. you had a little bit of fun with this video. It would be great to see you in my next one.