Understand DSPy: Programming AI Pipelines

Key Takeaways

hello Community attending a conference is great because you notice what people ask you so let's talk about here DSP if you are not familiar with this DSP is a self-improving machine Learning System and I got a lot of question how it is compared to in context learning plus with our many shot in context learning capabilities of our new llm with with a 1 million token context length so here we go if you want to learn about ICL plus I have a video on this but dpy let's focus on this now dpy today has got a new name interpretation today it is called a declarative self-improving language program pythonic and you might say what so let's start from the beginning DSP originated here from DSP in 2022 and the idea was simply the DSP stands for demonstrate search and predict and it was at the time a framework that relies on passing natural language text in sophisticated pipelines between a language model and a retriever model so you see here we focus language model retriever model and the building and optimization of pipeline between those models this is the origin and this is the key for you to understand DSP later on because of course it evolved over the years if you want to understand demonstrate search predict in a second let's try this one this DSP framework was used in in context learning so you see here this is has its origin in ICF at the time in 2022 they started here with an input question to be solved by the llm and a twoot training example so we have a question and a twoot example where we have a question and the answer that we like and another question another answer that we like and this was our training set and then our DSP introduced the number of composable functions that bootstrap Now new training examples and this process they call demonstrate and what they done is simply by decomposing here the query our question here into if it was a complex query into three smaller less complex subqueries and for those subqueries they got information from a different knowledge corporate they did the search operation eqution and then for the simpler questions and all the information on the simpler questions they started to generate or predict now a more complex output so you see this is the idea of DSP that later moved on to become DS Pi but this gives us an Insight you should always be remembered when we talk about DS Pi at the same time you see here that Omar so the identical person from DSP also did some research on C Bear this was simply here you remember birth in the 2020s and then two years later you have here the publication from Stanford University also with Omar where they were talking about efficient retrieval system so I told you it is about the language model and the retriever model and the information retrieval between those two models this is where thepie evolved originated now Co Bear version two is something beautiful we use today for my green grasshoppers here I explain what is co bear especially version two you have your query and your text passages they're independently encoded here with the classical bird and the output embeddings and coding each token are projected here to a lower dimension and during the offline indexing every passage D in the Corpus is encoded into a set of vectors and this vectors are stored and at the search time when we get our question the query q is encoded also in a multiv vector representation and its similarity to our passage D is computed as the summation of VAR side maximum similarity operation namely the largest for example cosine similarity between each query token embedding and all passage token embeddings plus there's a clever idea utilizing here our residual compression technique where each token embedding Vector is encoded by referencing a set of precomputed centroids th storing only the deviation from this precomputed centroids so there are a lot of clever tricks to use here in cool Bear version 2o now there's now a beautiful difference with 5cl plus our many shot in context learning when we have 1 million token context length up to 10 million token context length with the new Gemini programs llms we have here simply one prompt but thepi is now a machine learning framework it is a machine Learning System and with every machine Learning System that you know when we finetune an llm you know what we have we have a loss function we have an Optimizer for example an atom Optimizer and we then let the system run compile and get trained so all our weights weight tensors are now optimized for particular task and the same we have here DSP has its training set its development set and the test set for its performance evaluation and and this is a crucial word a self-improving machine Learning System so whatever the system is self-improving the system has to run one cycle analyze what have I done try to improve myself run another time look what have I done is it better worse self-improve again run the complete system and you understand that this is completely different here from a simple many shot prompt example in in context learning therefore dspi as it is today we have a declarative self-improving program in Python pytorch and this program more or less lives exactly between our language model and our retriever models and now this in between we call a pipeline beautiful so DP machine learning framework you have a training set development set test set and it is a self-improving machine Learning System and now you understand when we talk about dpy is a programming model designed to streamline here the development of the optimization of pipelines language model pipelines for complex task and we have more or less four elements in our program structure we have signature mod modules T parameters and compiler now signature if I tell you that this is here a declarative program structure you know signatures abstract here the input output behavior of our modules that live inside of the pipelines so signatures input output they just tell you what to do models are now critical elements here because they replace you to traditional prompt templates that you might know from Lang chain with functions but this functions are parameterizable functions so you see here modules submodules different modules combined and you get already the idea and then we have in thep t promas a very strange word substituted with an Optimizer an optimization tool to automat the prompting process within our pipelines similar to an atom Optimizer if you want an analog on and then of course you have to compile all of this into machine readable code executable code and this is done by a compiler and an optimizes here our pipelines based on a matric you can Define matric can be accuracy a specific metric that you need for your task whatever and this now enhances you the quality and the efficiency of the program let's go for a very simple example now we build here or we test here a simple pipeline configuration and as I told you the as p is here in the interface between language model and retriever model so what do you think we go here with a rag functionality retrieve augment and generate here from external data sources now remember in pie we have now two things we have our pipeline in this pipeline we can build the pipeline the way we want it we can have a single model of structure we can use multiple modules we can have a multimod structure we can have sub modules in our multimod structure it's like a Lego you can build whatever you want in your pipeline but remember those modules they they are here with parametric function the io process in each and every model and of course the complete pipeline that is buil of our modules so always remember that we optimize later on when we do the compilation here and we do the runtime we optimize the structure of our configuration of modules plus we optimize here the specific par parameter the specific parametric function of the pipeline which you could also call here the prompt structure beautiful easy example we have in the spine here a specific module it is called retrieve and this module has been built for you and as you can see here you say from the asy retrieve and then you go to specific Vector databases milus chroma DB or Asia C nitive search functionalities you activate whatever you want so we have here some pre-build systems for us where we can retrieve here our external information and if we build now the pipeline for a simple example we have here the class rack this is of course a DSP module we initialize it and we have here two subm modes that we need of course we need the DSP retrieve functionality because we want to retrieve this either from a simple data set or from some external data on databases Vector stores whatsoever and then we want to give the system kind of an intelligence so the second sub model we need is a chain of sord so this simply tells the system hey sync step by step with all the information you retrieved and try to make sense out of it that's all there is and then you just generate or predict the the answer given you have not the context and a specific question great after we build our rack pipeline now comes the interesting part because we have done a programmable structure we can now run here this and optimize automatically here our rack pipeline so what we need for this we need a training set training set development set a test set so we know we understand where we want to go if I say we want to go we have to do some validation on a specific metric that is particular chosen for this task so you go for example with accuracy and if we have the data we know what we want we know have the question and the answers we have a metric for the validation then we just need here a DSP compiler and this compiler includes here a number of tailor prompters or as I told you optimizers for your new DP program and this here is simple to an ADM Optimizer when we find you our LMS so we let this run and you see now why it's a machine learning system because it is quite similar if you find you now our llm but remember we have now the structural component of our Pipeline and the parametric component of each module and the complete pipeline oh sorry yeah for those optimizers into original literature I just found here yeah you can start with 10 examples but having as the or right 50 to 100 examples or even better 300 to 500 examples this will help you a lot of to increase the performance of your pipeline structure and the prompt engineering so 300 to 500 examples keep this in mind now pipelines you can build like Legos why not do AA module one AA module 2 and AA module 3 and maybe you go here to database 1 database 2 and database 3 and then you just build another module and this is already pre-built for you you go with a majority vote module so if the result of the first retrieval is 17 the result of the second is 24 and the result of the is again 17 you know what the majority vote mod is it says hey we have two * 17 so I suppose this is the correct version and the chop is done however you know of course that sometimes model generate the wrong result multiple times the wrong result so this is in no way helpful at all to make it even simpler for building here your pipeline you go here with the parametric knowledge of our large language model one to the parametri knowledge of the large language Model N plus one use three different llms use J GPT CLA and use gemini or whatever you want and then you do a majority vote on this you see your possibility to build your pipelines are almost end lines you can build your own modu in code if you want it all depends on your specific task in your specific domain beautiful yeah if you want a little bit more advanced they have pre-built modules here for example for react so where you have a tool use so they can go out and do something maybe they have a python environment to calculate something or it is simply that they can go and connect to some external data server or you go here with a self-reflective rag where the rag does its work and then comes the self-reflective part of the thing where the rag thinks hm is this correct and the rack itself tries to make sense of this what it retrieve this kind of information and then you can add a chain of sword with a hint and I thought okay I know what is chain of sord but what is with a hint now this is easy I show you here the code so you have here classic question and answer data set where you have your question where you have your answer and then if you activate here this particular chain of sort module with a hint you're not going to believe this so when you say hey what is the color of the sky this is your question and you provide an additional hint to the system and the hint is it is what you often see during a sunny day if you look upwards towards the sky and you say wow this is so intelligent beautiful so you see lot of modules for you to do you can build your own Stanford NLP DSP and beautiful GitHub repo for you again this chain of s teaches the LM to sync step by step this is all there is to it just go step by step you can type this down or you use this module in this module here the aspy module extended his signature to include additional reasoning steps like your doain in physics or mathematics you need some logical structure in your argumentation the way you do it is here's the class definition of the chain of sort and you see here reasoning let's think step by step in order to uncover something for myab us a very simple example when would you use this pie how would you use it for what job would you use it now let's say we have our new GPD 5 system our vision language super model finally out and I want now to find here the best prompt for scientific summarization of scientific texts of hundreds of pages but I want to do this in the style that I like in the way that I like that a summarization is done on the domain knowledge that is included in my system and I don't know the behavior of gb5 I have no idea about it so I take here for example here my old gbd4 prompt plus 20 summarization example of gbd4 so this shows exactly I have here the complete text and then you have here my summarization that I like so here I give it the system 20 examples here the DSP data set and then have given it here original prompt older example and say now start this system self-improve here yourself with the new operational Lodge language model this is now GPD 5 evaluate this llms of course you need to have a system that you know that you trust we use your gbd4 turbo the classical system now you define a metric for your evaluation I can say hey it's a more dense more concise more interdisciplinary focused summary given here with new insights in physics and Mathematics whatever you want and then simply you let it run for 100 turns so the system builds now on your gbd4 prompt does some alteration based on its parametric knowledge comes up with a new prompt structure with a new sub prompt structure modifies it and after 100 runs you know that you have achieved now with this particular prompt number 100 for example the best evaluation result given your particular metric on your particular domain so I have now after paying I don't know how much money to gb5 but for 100 runs I have now my new optimized gbd 5 prompt for my summarization in my style on my multidisciplinary scientific text documents so you see gb5 will have new insights new options available I could try this out myself but I don't want to wait so I let the system do the self learning I just have to pay for using your GPD 5 100 runs and then hopefully the system found not just a local maximum but a global maximum for the pr structure now let's go back to ICL plus you remember that I told you now with this 1 million and 10 million token length we can add a lot of many short examples and in my video showed you hey we can up add to 500 examples here and then let's stay with this I have here my old gp4 prompt in pink and then you can say hey in this prompt for ICL plus hey gbd given that I have showed you now 500 many shot examples here in my preferred summarization style and I've given you my old gbd4 my best gbd4 prompt for summarization your task is now to find the best new prompt for yourself for and now if you want this is not the metric for a better and a more concise scientific summarization of complex interdisciplinary texts doc doents and then I could add try three times to find a better prompt and show me the result and you see you could do this in if you type this yourself or you just let the system run and experience this itself it kind of depends on the money you have available so you see this is quite a dynamic machine Learning System and please do not say the DS is only a prompt engineering tool it is much more than this so to show you here for my green gr help us where is the self-improving happening and I go here with three steps and at first we have here the DSP modules as I told you they are parameterizable they can learn and adapt their behavior over time they're learning of course to the use of demonstrations as I've showed you here in the original DSP idea of 2020 examples of input and desired outputs that the system collects and analyzes and learns and then iteratively processes this demonstration now the program dspi modules can then refine the strategy for prompting or for fine tuning you don't just have to prompt here you can also change here the fine tuning you find different parameters that you try out in your Loop that that you do the evaluation and you apply reasoning techniques and whatever so those modules operate G and learn here from the demonstration and this modularity of the DSP pipeline is one of the beautiful things here but of course we can also have here the dynamic optimization here with our teleprompters remember simple idea just optimization tools to automate enhancement of our pipelines between our language model and our retrieval models or even more complicated pipeline structures they evaluate different configuration and strategies by simulating your pipeline of a small set of training examples so whenever you let DSP learn and have then inherent knowledge about the configuration of the modules and sub modules in your pipeline the better the system will be and of course the last one that compiler driven Improvement the better your compil yes you know it if you have a metric accuracy efficiency whatever will be it will be here the perfect compilation so if you now get a little bit interest and said hey I would like to try this out now that I understand where it comes from for what it was designed how it evolved over the last years where the Au think in Stanford now that it is now a self-improving machine Learning System so can can I use it now for my task for my work for my company let's go there and you see what optimizes are currently available for f short learning for instruction automization for finetuning yes you let the system run I don't know 100 different fine-tuning cycles and then you find out where the best hyperparameters are available or for program transformation those are available today for you or of course and they also answer here into official documentation and here you have the link where which Optimizer to use in what particular place and they say if you have very little data 10 example you use this particular Optimizer if you have 50 example per task we recommended use this optimizers and you have more than 300 example use this Optimizer so I think it's quite easy then of course here the modules I have showed you almost all we have our predi we have our chain of Sword our program of Sword for code our react yeah and then we have a multi-chain comparison you remember Lang chain and all of this now you buil multi-chain comparison chain of s and I've also shown you here the majority vote system now you're familiar with this and for the last part I have to show you here the signatures what signatures do we have well input output if your task is question answering you're not going to believe your signature is simple question to answer for sentiment classification sentence to sentiment or paragraph to sentiment or whatever you have and you have a document summarization task document to summary or you define it whatever you want if you have multiple input output feeli like for rack answering you have a context you have a particular question and it will generate your answer or if you have multiple choice answering with reasoning you have questions you have three choices you have to reasoning part and then a selection part so you see as easy as could be why not give it a try build your pipeline for your particular task and I hope with this very short introduction to dpy you are able to build your perfect dream system that will do a selfimprovement of your complete pipelines that lead into or exit your large language model to improve the performance of your llm and of your complete AI system I hope it was a little bit interesting I hope it was a little bit fascinating and it would be great to see you in my next video