[LLM NEWS] KANs, Gemma 10M Context, OpenAI Updates?, Automatic Prompt Engineering, Tokenizer Arena

Key Takeaways

hey everyone so we have a new episode here of llm news and I'm going to go through some of the stories that caught my attention and try to provide a little bit of my own takeaways and insights as well so the first news we have here is from open ey so they're doing uh live streaming at 10: a.m. BT Monday May 13 so next week Monday and they have some updates on chat GPT and gp4 updates I know the community has been waiting for some updates from open AI on their models and you know there's a lot of different uh needs right now with these models but there's also a lot of ideas into how chat GPT can also be improved so we are expecting to see a lot of really good news with these new updates that are coming and you can find the link here and it's they have a add to calendar button if you want to keep track of what they're talking about I would say it has been some time since open a has made any really big updates and I think we might get a big update we are not too sure and they have also changed their website a bit as well and so everyone is curious about what exactly openi is working on and what they're going to release now there was an interesting news that came out um May the 8 here like two days ago and it was talking a little bit about how open AI you know is reading a search product so there's a lot of speculation about potentially open I releasing some features that enable better search we know that openi with their chpt product they already have some form of search but I think search is such a huge market and if they're going to go there we might see some kind of new features or even a completely new product we we just don't know yet these are all rumors we'll see what open a announces on Monday there there might be significant updates with your models we might see I don't know a newer model for instance which just don't know right anytime open ey does uh live streams there's usually something really big that they're announcing so we're expecting something big either way but we just don't know if it's on the product side or if it's going to be on the on the tooling side of things next up we have anthropic and Tropic released this new feature with their Cloud models and basically what they're providing here is a new way to generate production ready prompts in the console itself so they have a playground and you can go and use it and you can automatically do this prompt engineering I like this idea of prompt generation because you know there's a lot of like best practices that actually go into building really good prompts and to actually scaling these models and making them more performance on the task that you're working on so automating this process I think it has a lot of really good use and so I'm excited about this particular feature and basically what they're saying is that they're using you know prompt engineering techniques like Channel thought reasoning to create more effective precise and reliable prompts now something interesting that grabbed my attention here is the use of Chain of Thought we use Chain of Thought for all the use cases that we work on pretty much all of them that we work with clients on and and and this is how we improve these language models on a variety of T whether it's Health where it's legal any any domain that we work on we're either using some version of Chain of Thought or some variant of it in combination with some more advanced prompting techniques so that's interesting that they're using it for automating of promp engineering and they have some kind of uh testimonial here and you can also try it so I've actually tried it here and if you go to their console you can generate a prompt you can see that it's saying the experimental prompt generator can turn a task description into a high quality prompt for best results be sure to describe your task in much detail as possible so you're really just describing the task and what you expect as output and how it should be formatted you are still doing a bit of prompt Engineering in a way but the heavy lifting is going to be done by this um automated tool right and so I think it could be useful it can speed up the development of like your use cases and that's the thing right when you're building an MVP you want to like speed up or optimize some parts of or some components of this process right and I think the prompt engineering actually takes a lot of work this is why we actually teach I do a lot of training about this and Consulting because I do see that a lot of like engineers and developers struggle to really optimize their prompts and design really good prompts or there's always like some best practices that they're missing or something they're doing wrong and you're not getting the desired results so the idea of automating this is really cool and I really applaud anthropic for releasing this and for us to try it out so what we're going to do is we're going to test it out uh summarize the document these are kind of common tasks but you can put or explain your own task as well so I'm just going to try this one say summarize documents into 10 bullet points Max I'm going to generate the promp and then it goes into automating this promp I remember chat GPD actually had a tool for this or some plug-in for this where you know I gave it some prompt and it automatically gave me or suggested or optimized my prompt but usually the prompts that were optimized were not so great there was a lot of like biases in those prompt designs so that's something to keep in mind when you're using a tool like this keep in mind that this particular tool is probably optimizing for the Tropic models right the cloud models instead of you know in general all the other models right the open source models and so forth so that's kind of my assumption I don't know if this is true but let just keep that in mind okay so you can see here it gives you an actual prompt it's a lot longer right and it's using these kind of variables these kind of notations right I believe this is like XML which is more particular to Cloud models and then it gives more specifics about the specific thing that we want which is the 10 bullet points and then it says is here present your final bullet point summary inside summary tags so it's giving very specific details about what the prompt should look like and what the output is that you want from the large language model right so best practices are you want to provide you know structured very structured prompt I can see that it actually na nails that you want to be as specific as possible I didn't really read everything here but it looks like it's actually capturing the main point which is the 10 bullet points and then it's also doing this extra bit which is probably a suggestion or a more common way to do summarization and it says present your final bullet points some are inside and then put them inside the summary tag so you want to always structure things correctly and and I can see that's a good best practice anyway so I'm going to start editing here and then you can see it takes us right into the playground I can start to kind of test this right you can see here it has this variable I guess if I run it it ask me for that document I can just you know a document actually let me just take this one here this will be a better one to do copy this one and then paste it here I'm going to run this all right so this is your prompt and then this is your response here okay here we go one two three I notice the formatting is not so great here but that's fine um okay so open is develop a new feature site sources the feature will allow okay some versions may include relevant open a Under Pressure this one there has been speculation of op search plans okay jbd can results or certain queries for being users but the feature is limited and can have hiccups open it de cin to commment in the details of the search it looks pretty good actually and I could understand what this news is about based on this summary but you can get more specific right like it doesn't end here now that you have something to work with you can keep optimizing and that's how I would use it and this is how we actually optimize you keep iterating on your prompt you know be more specific about how you want it maybe I don't wanted this way maybe I want longer sentences that's something that you would add to the prompt itself so just some tips there in other news we have this new paper that came out which is consistency llm so we know a lot of developers and researchers are working on making these models more efficient especially at inference time right because we do want to put the systems into production at some point so how do we actually improve uh you know the the the inference side of things and so this is some work that I've been or an area that I've been trying tracking for some time there has been a lot of interesting ideas and I think it's still very much very research heavy and there's a lot of ideas right uh changing the architecture right adding components to the architecture and so forth this one in particular is trying to using something called parallel decoding so is trying to do more right with the resources that you have and without you know affecting the generation quality essential right and show that they can achieve 2.4x to 3.4x improvements in generation speed while desing the generation quality you always want to pay attention to generation quality when you're looking at this type of work and this goes not only for like the inference papers it also goes for other papers that are proposing different architectures or alternative architectures right you always want to check the generation quality or maybe some Advanced prompting technique as well or some kind of new data set right you always want to check whether those models are producing generation quality otherwise they're just SOL in one problem and you know creating others so that's the consistency llm is trying to perform parallel decoding by mapping randomly initialized and token sequences to the same result yielded by Auto regressive decoding in as few steps as possible so it's it's basically trying to uh do more with you know those different steps that it's supposed to do in this next token prediction kind of approach so I believe this is a has a lot lot of potential but we'll see um how how how it develops next up we have the tokenizer arena so one thing that I do when I'm developing applications with llms I do rely on a few tools obviously I need a good playground to quickly iterate over prompts and my use cases but I also end up using other tools in combination you know when I'm trying to develop my use cases so sometimes I would use something to manage those proms something to log the results of those prompts right some observability solutions some tracing solution and so forth but the tokenizer is something that a lot of people don't talk about and tokenizer is really important right because these models are tokenizing information and you need to understand how those tokenizers work that are that are backing these models right because that does affect the results and performance that you're getting from these systems you may be optimizing your prompts for instance for a use case and using some temperature value right some setting like that and you notice that you're not getting good results or the mold doesn't understand as specific uh term or concept and you try and you try you don't get good results and the model is just not understanding sometime it's because the model is not tokenizing the information correctly and so that's something to always keep in mind and you can see here as an example right to when I put here hello I am Elvis and I'm not an llm you can see how the llm is not being tokenized correctly here and that will lead to some issue with this particular you know prompt right like it will not understand that concept and so I need to deal with it a certain way different models providers use different tokenization as well you it's an area that's very interesting and doesn't get a lot of attention so I like to see tools like this that gives us a closer look or in-depth look at how these models behave and how they perform so take a look at the tokenizer it's really good work and it's available in hug and face spaces here up next we have Gemma with a 10 million context window so we again see that there's a lot of efforts for efficient models inference time we also see like long contexts llms as well starting to become a trend because we have seen Gemini 1.5 Pro and what it's capable of doing already and a lot of people are interested and also doing that with open-source models so gemi is one of those models right on Google how do we take a model like that and improve or increase the context window all the way to 10 million with again the most important part that it performs really well on the use cases the generation quality has to be there otherwise this doesn't make any sense right what are you optimizing for optimizing for something but you're you end up them all that's not it's useless so that's something that I always want to keep track of now they do provide some implementation details here which I really like I always like when someone goes into like the technical details of what they're doing this is not a paper but I this could easily be a paper and it's using against some of the more recent developments like recurrent attention and infinite attention by the way this one might do a paper overview on this one to get into those details using some recent ideas to improve the performance of such a system right in terms of how much memory it's going to use and so forth right the complexity of that how computationally intensive that system is so it's just barring ideas and kind of combining things and doing experimentation and so forth which a lot of the work that you see today is going to involve some of that right like borrowing ideas and kind of adapting them and so forth so it's good to understand specifics about you know some of the areas and some of the ideas that are being proposed recently right so when you see like a new technique like new attention mechanism something it's good to understand it from a developer point of view because you can borrow ideas from that and kind of combine it with other older ideas as well um I really like this idea by the way this idea of like grow length where you gradually increase context science from 30 2K all the way to 10 million and the idea here is that you are basically trying to simplify at the beginning right in the beginning layer simplify what's being learned and then all of that is leverage as you go deeper into the layer so you that's being explained here I think that makes a lot of sense in terms of learning and so on and making the learning effective and and efficient as well so take a look at this for more details next up what we have here there are two posts and both of these posts actually are from folks that are working deeply with large language model so I have a lot of respect for people that are very transparent about the work they do and you know documented online share about their learnings and you know working with companies and working on different applications with large language models and I usually kind of monitor this and I think this is a great space like what I'm doing right here to highlight some of these areas because it's a conversation it's a discussion right how to do better evaluation for instance right people have different experiences right with these ideas and tools so for instance here what's mentioned is the importance of evaluating your evaluation pipeline so you can build an evaluation pipeline which is a lot of work already right having the right metrics your cability Solutions and so forth but are you evaluating those right and you spending a lot of resources and time and that actually takes a lot of resources and time and expertise and so that's something that you should be looking at now something interesting that chip mentioned here right the link will be in the description to this uh post but it says many I estimate 60 to 70% UI to evaluate AI responses with the common criteria being conciseness relevance coherence faithfulness Etc I find as judge very promising and expect to see more of this approach in the future now we use AI as a judge or llm as a judge in this case because we're using llms uh for a lot of use cases and we build evaluation pipelines and we have to evaluate how those evaluation pipelines that are powered by these language models as well we have to do the evaluation um and we're getting good results we're it's very promising I was one of those persons that was very skeptical about these language models being used to evaluate the results of other models and so forth right but I I think where it gets really interesting is in that the reasoning capabilities of these systems are so good today and they keep improving and I think they will just improve more um that makes them so fit for this type of task because you're measuring different aspects of the outputs of the systems right and everyone will have a subjective rubric a subjective way of measuring how good your systems are right so we have the standard metrics like Precision recall accuracy and these different ones that you use for different tasks but again at the end of the day these models are producing a generating you know these long sequences right that might have some quality issue and the way you want to assess the quality of it might go you might go into details into how to do that right you might have your own subjective way on how to do that so you kind of need a system in a way to do that and I think this is what's really promising about this area and again we're doing a lot of experimentation getting good results other clients that we work with also get good results with this so it's a very promising area I was at first very skeptical but because I've done the experiments I know that there's a lot of potential here and I'm very optimistic about it as we increase or improve the capabilities of the systems right evaluation very important and keep a an eye on that so when you see people talking about valuation just pay good attention to what they're doing there's a lot of comments about other folks as well with their experiences and so this is why I like this type of post there's another one as well from Jason Leo so he talks a lot about like his experiences I think he does a lot of Consulting if I'm not mistaken and he talks shares a lot about working with large language models he's also the creator of instructor which is one tool that we love to use as well for structured outputs which is another aspect of working with these large language models but he also talks about evaluation right being very picky doing quantitative analysis right measuring your systems have a really good systematic way of evaluating your systems right the importance of good observability and so forth right asking really detailed questions it's not about your subjective feeling about how a system performs not just about the qualitative part that's really important too but you need to have a system that's reliable to measure how good your use cases are right so and that gives you hints and ideas into what are the questions that really matter for your application and so forth so he goes into a lot of details I think it's like really random thoughts and I guess he's going to write like a detailed blog post about this I wish he does because I think all of this that he's talking about gather with chip is really important which is the importance of evaluation and asking the right questions to this systems because you may be optimizing your system for the wrong reasons and the wrong metrics so it's really important to ask those detailed questions and something he mentions here is like the use of embeddings no I have a lot to say about embeddings this is something we talk about in our course as well and you know choosing the right embeddings you know are you going to find T embeddings when to find tun embeddings what vectors are solution to use what are the features that you want to use right when to use one over the other you know and and so forth right so there's a lot of talk about this and and for me it really depends on the use case I going to depend on the use case and the resources that you have something he mentioned as well as elastic search well I work for elastic and I know a lot about elastic search and let me tell you elastic search has really powerful features right keyword search is not to be ignored in fact most of the systems that we build that are rack systems today are a combination or hybrid search which is like keyword and semantic search so don't sleep on that and there are tools like elastic search that already have this enabled the last paper I want to cover here is col of Arnold networks can for short now I haven't read the paper this is something I'm going to do but the idea here is that they're proposing some kind of alternatives to multi-layer perception multi-layer perceptions is one of the key components of the modern deep neural networks that we use today they state that while MLPs have fixed activation functions on node neurons cans have learnable activation functions on edges right referred to as a weights so you can see here kind of the changes and the equations and so forth again I haven't really read the paper in detail so this is something that I plan to do and potentially do a paper summary I have so many papers that I have been requested to summarize so all of that is kind of work in progress but something that I do always when I see an alternative method I want to see exactly what are they solving is this just an alternative method and to show the theory or some idea like that or does it actually have potential that's something that I'm was looking at and we know that with deep learning today say that scaling really matters so they have some analysis on scaling laws and so forth uh but I really like one chart that they presented here which is something that you should pay attention to and this is again coming from you know what is the actual practical Insight of this paper right I'm almost I was asking that I'm a researcher but I I love to also look at how practical this actually is and can I actually adopt this somewhere so there's a I think towards the end of the paper there's a nice little chart here let me see here on page 33 I believe and it tells you know should I use cans or MLPs and so you can see if you're going for accuracy and so far there's like a nice flowchart here if you're going for interpretability or efficiency which we have discussed a bit here as well in this particular episode of the LM news so you know use this to kind of give yourself some guidance look at the results as well um what areas are they evaluating the system on right what is the kind of memory usage and so forth all of these things are things that you will look at to make a decision whether this is something you should be using no one thing I've seen is that there is a lot of experimentation on this already I have seen some implementations on GitHub there's a lot of like interesting ideas already and this just came out a couple of days ago that bit for this episode of the llm news I hope you got some insights here I hope there was something interesting for you here I'll keep doing this I receive really good feedback from the first episode and I will love to do more of these but again your feedback really matters cuz that's the only way I can improve these um again leave a like if you enjoyed this video And subscribe to the channel because it tells me if you really find this information useful have a good one