Open-source AI models are surpassing closed source (fast) | AI/ML Monthly

Key Takeaways

hello Daniel Burke here machine learning engineer and in this video we're going to do a video walkr of my newsletter AI machine learning monthly which covers the latest and greatest but not always the latest in the world of machine learning and AI with a specific focus on open source so if you want to link to the text version it'll be below if you want time stamps they're also there too let's dive right into it I've got a few things that I've been working on here the first is a guide to bounding box formats and how to draw them next is a reveal of a project that's coming up building a custom object detection model with hugging face Transformers a small app to calculate how much memory and machine learning model needs uh a blog post on beginner's guide to embedded machine learning in other words how to get machine learning models to run on device and there's a video version of ml monthly January 2025 which was from last month so be sure to check that out if you haven't seen it so the first is uh a guide to bounding box formats and how to draw them on learn. I want to release more tutorials and blog posts on here so this is the one that I published a few weeks ago but essentially what it does is it guides you through different bounding box formats and so uh we have xyxy XY wh cxcy wh where they come from where to use them and what different Frameworks use which so pytorch hugging face Transformers YOLO type models or Google Gemini and the goal of this blog post is to introduce the different formats and how to draw them on an image so say we're got this image of someone putting a piece of trash in the bin and then we want to draw a bounding box around the bin because we're building an app like trash ify which I'll show you in a second but we have this box that this one was drawn nicely and then this is the same box but drawn in the wrong format so there's nothing worse than having a good object detection model but then drawing boxes in the wrong format so if you want to learn more about object detection in different bounding box formats check out this blog post that leads into the work in progress uh on learn huggingface face.com not a affiliated with hugging face just a tutorial website where I'm working on different uh projects or open source tutorials and this one is creating an open- Source object detection model or fine-tuning an open source object detection model for your own use case it goes along with the text classification project that is live this one is the code is all done uh but I'm just adding all the extra materials that go along with it and if we go to the demo this is what we're going to do we start with data we find a model and we train it and we finish with a demo which is trashy which is an machine learning model with the goal of encouraging people to clean up their local area so given an image like this it will detect hand trash and Bin and if it gets those three things you get a plus one so you can imagine this could run on a iPhone or a smartphone detect these three items in the camera frame and then if it detects a hand trash and Bin you get plus one point to a GL Global leaderboard of people in different areas and you can see who has a high score four picking up the most trash in their local area so if we try it on a custom image let's see this one well this is in the test set you'll have to believe me here that this came from the test set so if we predict on this and there we go so this is without nms filtering which is non-max suppression and then if we have here we have nms suppression so we learned this in the project is plus one because we have a hand we have a trash and we have bin and then we have the ml model memory calculator so uh what if you're like me and you'd like to run open source models and I have a RTX 490 and that computer over there which has 24 gigs of vram I want to see which models are capable of running on there so 7B tends to be about the or 7 billion parameters tends to be about the sweet spot for me um but there are larger models becoming more and more available these days and of course smaller models so with the help of Claude I made a simple app to help you calculate how many uh or how much memory you'll need for uh your machine learning model so if we have 20 million parameters we got plenty of room there but different levels of model there so go and check that out and then if you want to learn all about embedded machine learning how different models get run on device why Tesla uh cars might want to use models on device or do use models on device rather than the cloud go and check that out an in-depth blog post about embedded machine learning anyway that is my work for the past month uh work in progress but we'll get updates on that soon so let's jump into Daniel's open source AI resources of the month and so the first one here we have segment anything with text using SATA now I don't know if that's pronounced but it's like s A2 V8 so this combined Sam 2 which is segment anit in from meta which is the goal of um drawing boxes or not boxes sorry like pixel by pixel segmenting different items in images and videos that was what was the upgrade with Sam 2 um but the team from bite dance have released uh Sava 2 we've got a demo that I've like all good machine learning cooking shows prepared earlier so if we install uh upgraded version of Transformers flash attention I am running the bigger model so I do have an a100 running in here we have 40 GB of vram and the model I'm using the 8B version so 8 billion parameters requires 27 GB to run so I've already loaded that and say we have this image here and at nutrify we like to do uh food intelligence so I'd like to segment all the different food objects here so if we keep going this was my lunch the other day I've already done it the beautiful thing is you you can prompt with text so if I go the main subject it doesn't give me that good results because if you look at that image there really is there a main subject or not I'll leave that up to you to decide but then when we get more specific with food so if we have this food list here a portion of dried meat sticks there fruit salad with watermelon and cantaloupe and salad with red onion bokini rocket leaves cucumber slic we have that over there so if we run that and then we plot The Masks how cool is that so we get it segmented and then of course because we've got segmentations we can turn those into boxes this is the type of thing you'll learn in the uh blog post that I showed before we now have bounding boxes around our food items just from text so that's the power of the sa 2v model which comes in uh four different sizes 1B 8B 4B 26b and that's where you can use the ml model memory calculator to figure out if your Hardware can run it now a really cool thing that found is because it has been trained with segment anything V2 plus an llm it has semantic understanding and so let me get specific with this if we didn't want the mask around the plate or the Bowl here we just specifically wanted the food items let's see if we can do that via prompting what I'll do is I've got another the same list here this list but I've replicated it but now I've specified at the end only no plate only no no bowl only no glass bowl no Fork so if I change this to food list only and we run the prediction it's pretty quick because it's only using the Sam 2 decoder and then if we have a look at our visualization look at that they are some great looking boxes or sorry uh Mass so there's a portion of dried meat a little bit pixelated but that's okay we can see that it's it's fairly Pixel Perfect now not again actually not Pixel Perfect but it for our use case it certainly serves its purpose and then if we want boxes our boxes should be now much tighter around the food there we go so that is really cool so a big shout out to the bite Den team for publishing that there is also a research paper that goes along with it and of course uh a GitHub somewhere the link yeah the link is on the hugging face um Reaper if you want to see that and then if we go to the keynote this is an example of segment the main subject uh so if there is clearly a main subject you can just get it masked out um and then we've just seen the example there of if you get specific of just cementing the target thing that you want there's the hardboiled eggs and what I found really cool in the paper is the data annotation pipeline so this is getting to be more and more of a theme is creating large scale synthetic data sets for your specific use case and they start by creating an object level expression uh leveraging a large VM M so intern VL 276b then they create a scene level expression and then finally a video level expression and they merge that all into one so check out the paper a bit more there if you want more information but otherwise if you want to try the models out they are available on hugging face next up we have Google's release of Sig lip 2 so I've got another demo here that I've tried out um so if you put an input image well let's start with sigp so sigp 1 is probably my most used computer model outside of uh my own models that we built for NFI uh because it marries up language and image and that's what sigp stands for sigmoid language image pre-training and so sigp 1 was already Incredible sigp 2 has improved sigp 1 across almost every Benchmark well across every Benchmark so if you put in this input image here and then a list of text descriptions sigl 2 is designed to match those uh image with text and the more specific the text or the better match the text the higher score Sig lip will assign to that text input so if we have a plate of food a plate of breakfast food a plate of eggs on toast with sides and then a plate of food with eggs on toast grilled sausages smashed avocado and cabbage red cabbage wilted spinach mushrooms and tomato chutney oh my goodness so this is a breakfast that I had the other day at a company uh cafe nearby where I live called Good Company shout out if you're on the Brisbane uh north side of brisban Epic Cafe okay and what we can see here is both sigp 1 and sigp 2 did an incredible job of matching the most specific text now the thing is all of those text inputs could describe this image but they are ranging from simple to more detailed and here's the thing if you want to get really specific you'll get a higher score but if you just stay nice and simple you'll still get a score but in comparison to the text that better suits it it is a lower score so that is the job of sigli zero shot image classification and you'll also notice that in many open source VMS uh sigp not sigl 1 sigp 2 is brand new sigp 1 is the vision encoder for many VM so Vision encode llm encodes to text we marry them up Sig I'm assuming Sig lip 2 is going to be uh a dropin replacement for all of those models open source models that are using sigp one as a vision encoder so if you want to see the blog post there's a blog post here on hugging face goes through the architecture if you want to see um the paper that's also available here um that's on hugging face but the paper's also on archive and if you want a demo you can try out the demo here on Google's hugging face sigp 1 versus sigp 2 as you'll see the more specific the text to the image the higher the score so all three of these could describe this image but the most specific one gets the most or the highest score which is generally what you want and then we have siget 2 all the model weights are available in open open clip as well so whole bunch of different variants there so big shout out to Google there thank you for open sourcing siget too up next we have Ovis 2 and Ovis stands for open Vision so this is an open- Source VM so if we jump into here there are six different variants ranging from 1 billion parameters to 34 billion parameters there is a GitHub available with all of the code there and this is the architecture I was just talking about so what over 2 does is it combines and uh text and then they pass those embeddings into an llm to produce an output so if you have uh image input we embed the image and you have text input you embed the text combine that in the embedding space of an llm and then produce text as an output so um if we want to go to the demo overs 2 16b so if we upload an image there we go we can take this from trashy um please write caption for this image and then we can let that run it's also capable of not only captioning images but uh thinking about different step-by-step processes so there's a lot more advanced use cases here but there we go recycling in Action a handful of paper waste ready for disposal so there we go nice and simple caption but all of these models are open source I believe they use uh Apple's aim V2 backbone yeah there we go so previously in Ovis 1.6 was Sig lip one so if we go to obis 1.6 and we jump into here we go uh used sigp 400 mil so sigp version one with a paired llm so this new version uses Apple's recently released a V2 so if we look up that we go to Apple here is a vision encoder which scor incredibly well on a lot of different benchmarks uh I am excited to see if they substituted this one for siget 2 and what the performance look like there and if we check out the performance uh overis 2 Series performs uh the best on the par curve of all of the open compass benchmarks even the 34 billion parameter model performs nearly on par with the 72 or 78 parameter model so really cool release there um from aidc I so thank you very much for that on to the next So speaking of Open Source VMS we're continuing the trend here with Microsoft launching 54 modim moim modal and 54 mini so mentioned in last month's machine learning monthly January 20125 we had 54 which is a 14 billion parameter llm so text only whereas uh 54 multimodal and 54 mini 54 mini is text only again but a smaller size 54 multimodal can do everything so speech text audio if we go here mini instruct we go to Microsoft's page 54 collection multimodal instruct beautiful so now we have language vision and speech and what really impressed me is that not only is it capable of 23 different languages it's that if you uh it's got a bunch of different use cases but if we go to the speech recognition and this this is where I'm I'm starting to get really impressed by a lot of models been so good at speech recognition is 54 multimodal instruct uh performs the best or has the lowest uh error rate so lower is better in this graph across many different three different benchmarks except for this one here where Gemini 1.5 Pro outperforms it slightly but in terms of word error rate so versus even models like whisper V3 from open AI which is a a model specifically trained to turn speech into text so that's really cool but then if we go down here I've taken the example I'll let you read through all the benchmarks if you want um It's A 5.6 parameter or 5.6 billion parameter model but I've taken this audio processing code and again like all good machine learning cooking shows have got an example already so if we have transcribed the audio to text and then translate the audio to French there's an example um audio file here that they've used use SE tags as a separator between the original transcript and the translation so I recorded my own audio file I'm going to play that uh over the top of this text here and we'll see if that works you can see if how well the transcription is hello Daniel Burke here I am testing the new F4 multimodal model from Microsoft and it is capable of taking in audio inputs and turning them into text it's cap of taking image inputs and turning or returning text from the image input and it is also capable of just straight text to text it's also quite small so it can run on many devices locally and it competes with models that are much larger than it and are also only available through uh a closed Source API so a really cool open source release from Microsoft here so basically instead of uh the only thing that really got wrong was how to spell my last name but that's uh this is probably a more common spelling of Burke then b o u r ke and then I don't speak French but we have the transcription here straight afterwards and that's just from a prompt so audio file in and then transcription and translation out that is awesome so if you speak French please uh tell me how this translation is compared to the text input here and then uh not only do we have the multimodal model which is 5.6 billion parameters if we go back we also have 54 mini instruct which is a text only llm but that is 3.84 billion parameters and it performs well above its weight class so even at the smaller size so these models are 2x bigger on the right hand side here we have a 63.5% performance or average score sorry across all these benchmarks and then the only model only two models to outperform it or three here are quen so 7B so that's twice the size uh GPT 40 mini outperforms it as well but that's a closed SCE model and then Gemma 2 which is 9 billion parameters so this is 3.8 so 2 and 1/2 times bigger is only slightly higher than it again so incredible release from Microsoft there if you're interested in small uh llms or small multimodal llms including speech now which is awesome um check out the 54 versions you have a blog post technical report and a cookbook uh on the Microsoft hugging face page and they are all MIT licensed now so many open VMS coming out uh there how do you keep track of them well that's where the open vlm leaderboard comes in which Aggregates 31 different benchmarks I'll let you go through these in your own time but if you're searching for an open source VM model that you can use in your own workflow um for example I'm really impressed by the overs 2 models so I'm looking to use them in nutrify workflows and if you want to you can sort by date here of the last most recently evaluated models so there we go Gemini 2.0 pro which was recently released we have an EV validate of a few days ago um and you can sort by average score you can sort by model size so if you're like me you have uh one GPU and you want to run it locally you might look for a smaller model but what really impresses me here is that the open models are there is now many open models that are outperforming even the Clos Source models so Claude 3.7 CET which was recently released as well um doesn't score as high across all of these benchmarks average as overis 234b even overs 26b is performing better or even 8B is performing better than Claude 3.7 CET so that is really cool to see is that open source models are getting better and better so go and check that out you can even if we see which ones at the moment step 10 um but generally I've found the ones with the weights listed are open source so that's the open V M leaderboard from open compass up next we have small VM V2 so now small VM which was also mentioned in last month's machine learning monthly I love how fast the open source space is improving now has video understanding in built into the model so it comes in three different sizes 2.2 B 500 mil and 256 mil and it's also mlx ready so if we go mlx GitHub mlx is Apple's machine learning framework work or one of them for running on uh Apple silicon so this is really it's really helpful for running if you want to run it locally on your Mac um or other Apple device so if we go down uh the model is very performant for its size and there's also a 500 mil video instruct version and the cool thing is that the 500 mil video instruct version is now running open on an iPhone sorry running not open on an iPhone so if we have a look at what's going on how many plants do you see and it's going to answer the question this is all running on device so that is really cool to see that's what I'm really excited about is is models like this running on device so it's just with you all the time and we'll see we'll see a model like that shortly again as well so here left or right cup for a special so there we go beautiful now of course that's been sped up but this is of course a V1 version so it's going to improve over time if you want to use that uh or try it out there's a form there to fill out and then you can also use the model for uh video highlight generator so it's got a plugin for VLC media and then you can pass it in I think it's yeah long form video up to one plus hour and automatically extracts the most significant moments so that that at the moment that's really important to me because my father is in hospital at the moment and he's getting taken care of by some wonderful nurses however their their main job is to give him care right but the other day my mom and I went up and we found out that he had a rough night but we didn't get much information cuz they of course were prioritizing providing the care rather than writing it down so it would have been really nice for us to have that information and so I'm thinking uh in that case of course you would it's a healthc care space so you'd need privacy concerns but this small blm2 could run locally on the hospital record footage of um throughout the night surveillance footage and uh write a report every day of when something happened um so for example at 3:00 a.m. woke up and had to be delivered Care by the nurses that's so rather than the nurses spending their time their precious time writing and doing paperwork they can provide the care so this is just uh I think over the next couple of years video inference is going to be incredibly useful so you can all the demo codee there there's also an opportunity to fine-tune the model so the 500 mil variant parameter variant uh you can probably perform full fine tuning so you can see the notebook there but otherwise um there's a blog post there's the collection for small VM uh there's a demo here which is describe this image because it's got multimodal and much much more of course you can sign up to try the app out later on too and up next we have have continuing the trend of Open Source VMS we have IBM release Granite 3.2 models under Apache 2.0 and so we're now entering the era of having models that are designed for purpose-built business use cases so that's what Granite is designed for uh business use cases engineered from the ground up to ensure trust and scalability so the models are available as Apache 2.0 on hugging face and what I love about the uh IBM um Granite series is that they've built out documentation quite substantially for how to use the models so if you want to perform multimodal rag you can process documents and they've got tutorials on how to do that so this is really important I think for any future if anyone wants to open source their model documentation on how to use it is probably just as important as the model itself so if we I shouldn't have closed those if we look at 8B instruct which is a pure text only model in many different languages uh we have thinking capabilities summarization text classification extraction Q&A Rag and a few more here but then we also have the vision oh wait that's the guardian so if we look at the vision model there we go so it's a compact and efficient Vision language model which has around 3 billion parameters and it's specifically designed for business use cases so extracting information from tables charts infographics plots diagrams and more so so if we have a look at the benchmarks where uh it's a business use case so document vqa document Vision qu question answer chart QA so questions from charts text of eqa questions from text Granite Vision uh performs the best across almost all of these benchmarks it does slightly fall on the mmu which is general knowledge but if you're mostly using a model to extract Knowledge from your own data it doesn't necessarily need general knowledge so of course take all benchmarks of the grain of salt always test it on your own um data but there is a paper associated with that I found that a good read of how they used synthetic if we look here sin have they got this here it's not mentioned here but it is mentioned up here synthetic data yeah there we go so creating synthetic data specific capabilities including document understanding tasks so uh there is also a GitHub beautiful open source release from IBM for the granite model so if you're using uh llms or want to use llms or VMS for a business use case be go and sure uh be sure to go and check out the granite series up next we have an open source release from The Arc Institute which is Evo 2 which now instead of predicting language tokens uh the next token in a series like the cat will jump over the fence uh Evo 2 is trained on 100,000 species worth of DNA data so llms are starting to introduce or starting to enter the biological field and so I believe it is yeah 9.3 trillion nucleotides so base pairs which are ACG andt like in DNA and so what this makes this model capable of is that it can accurately identify disease causing mutations in human genes so that is a really cool use case and the one that they h highlighted here was for example in test with the variance of the breast cancer uh Associated Gene brca1 Evo 2 achieved over 90% accuracy in predicting which mutations are benign versus potentially potentially pathogenic so my blessed grandmother had surgery on last Friday for breast cancer and so having this potential use case for something in the future to screen someone uh and go hey this um this sequence that we found may be uh benign or it may be potentially pathogenic we could probably investigate that further is really close to home for me at the moment so uh incredible release here from the ark Institute and then there's two papers associated with it we have the uh biology focused one which is genome modeling and design across all domains of life with Evo 2 and then we have the machine learning Focus one which is they talk about how they designed the algorithm which which was striped hyena 2 and they used that for uh next token prediction rather than the dense Transformer because what they wanted to do with I believe does it have it here million a sequence yeah there we go so the model can process genetic sequences of up to 1 million tokens at once so this was really important to be able to go over if we have a gene that ranges from uh I'm not sure the exact range in a human genome but I believe it's probably from like 100 nuclear tiddes to maybe even Millions long so you want it to be able to process a long sequence at a time and so what they did is they created stripe tyena 2 which is much more efficient as the training sequence increases so the dense Transformer scales here quadratically with um sequence length and so if you want to train it in less time you want to make a more efficient model so that's how they created this that's a machine learning Focus paper finally there was also a GitHub so if you're in the uh biological field and you want possibly the most state-of-the-art performing model for genome modeling at the moment go and check out Evo 2 and now we have hugging face release open deep research so over the past couple of weeks Google and open Ai and of course perplexity have their own version of Deep research which is essentially uh like you tell an agent or an llm to go and find information on the internet searching all through those links taking steps to sort through that information and then returning back a report and it may take a bit of time to do that but the beautiful thing is is that it does it uh automatically that's the whole goal anyway so if you go into the blog post open source de deep research uh performs very well for an open source uh model open source version and they re replicated the workflow in about 24 hours so that is incredible work for such a short time frame but if we go down here the way I imagine it is that is's an llm in the pilot seat and what it does is a code agent so the llm you give it an instruction say as I've given one here what are all the ingredients nutrition information in Domino's Pizza in Australia and I know one second let me move my head so you can see that uh this is the instruction I gave it and it's going to generate code steps so if we look through here this is running live at the moment I just put this in it takes a few minutes per step or thereabouts and and this is what happens is it will let the agents Express the action it wants to take in code so what this means is that you have an llm and you go hey do this thing for me and then it's going to go oh I need to use this code step to take the next step and then I use this code function to do the next step such as search the web click on pages read information and so what this paper showed um which is what they Ed to implement their system was that code actions require 30% fewer steps than Json so really cool read through here of how um agents are definitely evolving they aren't perfect but they are getting better and better so read the blog post announcement there this is all built on top of hugging fa's open- Source um Library small agents which is to designed to help you build agents which combines an llm with tools and then this is the Deep research GitHub so you can see the workflow that they went through to reproduce these steps and then of course here's the demo which I've gone what are all the ingredients nutrition information in Domino's Pizza in Australia so it's got a web search here it's search for Domino's Australia menu nutrition then it's visited a page here Domino's is this a actual web page let's find out there we go that's exactly what we wanted and what's it done next so it's visit page gone there page down thought scroll down page down visiting URL contrl F on a page this is really cool so I'm going to let this run for a bit we may check back on it in a few minutes we may not and see where it ends up but that is a really cool open- Source version of an agent and this field is just evolving very quickly so it's awesome to see um such cool projects being replicated within a day up next speaking of the small agents Library there is also a course to go along with it so if you want to learn more about Open Source agents or just agents in general check out the hugging face uh open source course for agents it's got three sections there so far so introduction to agents Frameworks to use and the small agents framework so the small agents framework is what the open deep research oh no it's got an error Okay so not perfect yet but it looks like we can start to see the steps that it was taking and that was similar to oh wait there we go it worked a few popular pizzas Margarita Pizza here's the nutrition information meat lovers Pizza beoni Pizza so that's that's pretty cool pretty cool to go from a text prom to uh information backed with their official nutrition page does this it works Margarita is that correct so classic crust does it say anything here one piece 833 calories what do we get okay maybe it's not perfect 116 calories eight servings per Pizza okay so maybe that's times eight anyway we're not going to fact check that for now but that's just to show the workflow and that the field is rapidly evolving if you want to learn about AI agents check out the open source hugging face AI agents course now up next we have an Open Source 1 .6 billion parameter Transformer model for text to speech I had to think about that then I near said speech to text um but they release two versions of it which is a hybrid and a Transformer based the hybrid is more efficient because what they're trying to do is make it faster than real time which means um if you have one second of speech it can be generated faster than 1 second so yeah the hybrid version I'll let you read the blog post for more information on the specific architecture but the important thing to take away is that it's faster and more efficient or slightly faster than a Transformer there's a architecture and that's the goal here is to um the real time Factor so if you're using an RTX 490 locally I'm lucky enough to have one of those over there it can generate speech faster um than real time so that's beautiful that's where we want now it is hard to evaluate text to speech models so they've got some examples here of audio generated by the model so let's just have a listen I don't really care what you call me I've been a silent spectator watching species evolve Empires rise and fall but always remember I am mighty and enduring respect me and I'll nurture you ignore me and you shall face the consequences so that was pretty good but if we try another one I don't really care what you call me I've been a silent spectator watching species evolve Empires rise and fall but always remember I am mighty and enduring so to me the zos one sounded a bit more natural versus the 11 Labs would sound a bit more robotic I won't go through all of them you can of course go and try them out for yourself but the models are available on hugging face under the Apache 2.0 license and then if you want the demo we can try some information here so okay so I've got some text here let's see how this sounds and now this is with the hybrid model I've turned off all the emotions uh haven't really played around with the setting this is just the default version I guess yo my name is Daniel Burke and I am a machine learning engineer at nutrify and I live in Brisbane Australia okay um I think cocko 82 mil from last episode was a little bit better than that I'm not sure where that voice came from but um yeah if you want to open source uh text to speech model go and check that out up next we have ai2 release Almo for iOS free to use on device llm and so Alo stands for open language mixture of experts so I'll let you read through the blog post if you want to check it out the takeaway is that it's now very possible to run large language models well this is a relatively small one compared to other big models but it is an llm running on your device without an internet connection so no API calls it's just running locally on iPhone and iPad so if we go down here they reveal how they made it and it's also an open source model which is available on hugging face under Apache 2.0 so it is open source and then the app is available on the App Store as a free download and even more impressive is that the app code itself is available on GitHub so if you want to build uh an app that runs locally on device with an llm an open source one you can see exactly how that's built so really cool and I tried it on my own phone which is an iPhone 15 Pro and uh as you can see airplane mode is turned on so I asked for a simple hummus recipe I did speed this up by 3x to just for brevity but it was pretty quick like faster than what I could read and then I said I was allergic to tahini which I'm not I was just trying to test it out and it it gave a good fix so this is really cool to see of um like where the future is going and more and more powerful models running on device this is the trend that I I'm loving throughout this year so far and I think models like this should potentially replace the assistant that's on most iPhones beginning with s I'm not going to say that out loud up next we have local Ai and their GitHub repository of many different open-source tools so uh local AI is an organization and they focus on creating efficient open-source AI software that runs on your device so this is a background removal one um for OBS so I'm using obs s to record this right now if I didn't have a green screen you can use um OBS to replace your background so it would just be a cut cut out of me but I kind of like my nerdy computer background so check that out if you're using OBS and you want to see uh how or just all the code really of how an on device AI system runs and then we also have local vocal which is speech AI assistant in OBS so you can run OB open AI whisper in real time to create speech captions appearing on your OBS recording so that is really cool as well and then that's not all there is a lot more here but awesome collection of different software that you can run AI um on your own device next up we have open rhf so rhf stands for reinforcement learning from Human feedback it's how you align large language models with human preferred outputs and starting to be the same in computer vision as well so if we go through here there's a whole bunch of different methods or reinforcement learning algorithms that are implemented in open rlf such as reinforce Plus+ or po which is proximal policy optimization and then GPR or grpo sorry which is what deep seek R1 used to uh improve their model and then there is also a great blog post that I found and that is unraveling rlf and its variance progress in Practical engineering insights so this goes through uh a bunch of different RL methods and nonr Aral methods such as direct policy optimization and explains how they work and the different advantages pros and cons of each so really cool library to see uh if you want to use some reinforcement learning on your own models go and check it out the documentation is also very comprehensive as well so quick start a whole bunch of different yeah it's built on top of Ray so it's very performance efficient so go and check that out open RL HF on GitHub up next we have a series of case studies so these are examples where a company has taken something and they've built it into the real world and then they've written about how they did it so first up we have fine grain so they share how they create a prompt high resolution image segmentation so we saw a prompt segmentation model at the start of the video and that is sar2 VAR but this is fine grain have deployed this in an API that you can use so we have the segment segment anything model they actually didn't use that one they used a different architecture so how we built our own box promp segmentor so they use grounding Dino as part of the pipeline which is grounding Doo to text boxes based on text and yeah they use the MV4 net MV4 MVA sorry MVA net architecture so if we go to this there's the archive paper for that and that's a really cool uh example of how research gets translated into production and so someone has done a comparison here we're trying to segment the chair Photoshop still has all the background canva has a few missing artifacts down here and the fine grain object cutter does a really nice job of capturing all the details probably not 100% Pixel Perfect but it's doing this with machine learning so that is incredible and their goal was to uh get rid of this or improve this cropping with things on the edge so they discuss how they did it they create five different channels of input and after the release you can see the improved model so fine grain they create Pixel Perfect Image editing API and then you can even check uh or try out the demo and this is a really cool use case or cool example of how building an open- Source demo on hugging face has turned into a company so they got a lot of feedback uh from people using the space and tried to improve the product here so there we go what should we erase erase the soap there a pretty good job there and what else erase the Potted Plant wow that is cool it even does it in the mirror okay that's really cool so go and check out uh object F grain object eraser and if you want to learn more on how these different types of models get built go and check out their blog post up next we have the makers of the Zed code editor open source Zeta so this is a great CA case study of oh where's blog blog and then we have Zed so Zed now predicts your next edit with Zeta so this is a really cool example of taking an open source model which is quen 2.5 Koda 7B which is available on hugging face and they fine-tuned it using their own data set so in one way they you supervise uh fine-tuning with UNS sloth and Laura or low rank optimization or low low rank adaptation and but they found that direct preference optimization with just 150 carefully selected examples were significantly enough to improve zeta's behavior and then there's another uh so that was just making the model better but there's two parts to making a product right you have to make the model good to begin with but then you have to serve that model so Zed's priority is fast inference or fast the app should just feel fast so what they did was they set aggressive performance targets and the predictions are delivered in under 200 milliseconds for the median case and under 500 millisecs for the 90th percentile so really cool example of how they used um speculative decoding to where the model uh writes like you have a smaller model writing uh draft and then you go over the top and they also served it um with Cloud flare workers to make sure that when someone pings a model so they send information from their Editor to the cloud it selects the closest data center because when you're moving information on the internet it is the speed of light but that still takes time to move around so you need to ideally send information to the closest compute Cloud to you geographically and then of course the Zeta model is also available um the one that they fine-tuned from Quin 2.5 coder 7B is available for testing on hugging face so really cool use case to see an open source model getting deployed into a product um thanks to fine-tuning and serving so go and check that out from The Zed team up next we have LM Studio introducing speculative decoding and then a couple of bonus extra resources on the topic of speculative decoding from Google so uh LM studio is an app for running lmm on on your own device so if we go here you can run it llms on your Mac so type to them natively there's a whole bunch of different open source variants um there's also an SDK available now but this is a great case study on how they made their models faster so speculative decoding is a technique that can speed up token Generation by 1.5 to 3x in some cases so that is really cool so it works by using a draft model first quickly predicting the next few tokens as a draft generation and then right afterwards because you've got the draft you can go over it uh with a bigger model and they either get confirmed or rejected by the main model so the main model being a bigger model so kind of like in uh if you were writing an essay you might write the first draft and that takes you uh you you type it out really quickly CU you're just going over but then you go back through it again and you edit it and you make it better so performance stats when you have a main model that is 32 bill um and you have a draft model that is much smaller so 0.5 billion parameters uh it goes from 7.3 tokens per second to 17 tokens per second or close to 18 tokens per second so a speed up of 2.43 and then we have some other stats here but if you want to read about how they did it please go through the blog post really cool example of how research goes so this paper is what introduced speculative Dakota so 2022 or 2023 and they show how they imp improve the inference of a transformer model here so that goes to show like it takes a little while for it often takes a little while for research to go into a product that you can use and then if you want to learn more about speculative decoding there is a specific use case here on the Google blog post which is called speculative rag so retrieval augmented generation speeds up Rag and then uh there's a great breakdown of different speculative decoding techniques and how they're being used see much faster generation there it's almost probably two and a half times faster of where of how specul to decoding works and where it's been used so go and check those out on the Google Blog and big shout out to LM studio for shipping an amazing new feature and now we're going to do a quick fire around on these releases cuz they are generally from big companies and they have enough uh broadcast reach themselves so I wanted to focus on all of the open source stuff uh a bit more here from smaller players but Google have released Google Gemini 2.0 uh flashlight and pro and Flash so three variant and from my personal experience they all perform quite well uh especially um at their price point I think they're about 1,000x or maybe 10,000x Gemini flash is 10,000x cheaper than that may be too high I'll put it on screen versus GPT 4.5 um a great blog post here from Phil Schmid on how to get structured outputs from PDFs with Gemini 2.0 and then there's another um Gemini oh sorry imagen 3 release in the Gemini API so imen 3 generates the best images I've ever seen so I've got this workflow here example so if you were to wanted to create synthetic data for training your own models uh you could have an input image this is a real image we saw it before up here when we segmented the eggs but then I use Gemini 2.0 to caption this image so with a detailed description and we have a generated image so that's pretty good for a generated image and then we could feed that now we have two samples for the price of one and we could also write in the prompt to uh reorder the order of some of these or make it on a different background make it on a different plate and so all of a sudden we can improve our data augmentation Pipeline with new samples entirely new samples so rather than just augmenting this we just create a new sample and tweak the text so if we go back to where we were at so incredible Fleet of releases from Google there uh we have anthropic release Claude 3.7 as well as Claude code so Claude 3.7 is their latest um Flagship model Sonet and it performs from my testing Claude is is incredible it's probably my favorite writing llm uh and also very good at code and then Claude code is a a gentic coding tool so it lives in your terminal and it can take steps but there's a video um you can see the example here in the video example xai introduced grock 3 which is we're starting to get to that point where llms are all sort of converging on being really good so there's probably five or six l that are GPT 40 and above now and for the most impressing thing to me for grock 3 is how quickly they got to uh state-of-the-art results so I believe the xai team was only formed about 18 months ago from the time of recording this and so they've gone from um starting a new team to having the best or most performing llm in the world according to the benchmarks in about 18 months so that is really cool open AI released GPT 4.5 um the most impressive release there for me was about a 2X reduction in hallucination on the simple QA Benchmark so instead of GPT 40 which was 61.8% hallucination GPT 4.5 apparently is 36% 37% yeah so the downside is of this model is that it's quite expensive yeah so if we look at the pricing of GPD 4.5 it's $75 per 1 million input tokens but if we go to Gemini pricing and if we look into this for example yeah Gemini 2.0 flashlight probably not as good a quality model or Gemini 2.0 flash which is a great quality model from my experience is 0.10 so if we do 75 75 / by 0.1 750 times more expensive on GPT 4.5 on import tokens than Gemini 2.0 flash so you got to think about that is it worth the 750 uh X increase potentially for your use case not for me personally and then to finish it off we have ai2 release scholar QA so this is a really cool example of uh a rag app across a whole bunch of different research so if we go to um the demo we can go um how does mag need help sleep what this is going to do is it's going to search over I believe it is we're just going to leave that run in the background 8 million academic papers from various fields of study and they've created this as an index using vasper and they've used a combination of bm25 which is an indexing algorithm as well as dense embeddings and then if we go to here it's going to search through or create a a workflow of finding different papers which relate to the query that I put in which was how does magnesium help sleep and then it's going to aggregate all of those sources and give me an answer with link backs to the research paper and now as we can see a few minutes later it's generated an answer with references so biological mechanisms of magnesium in sleep regulation effects of magnesium on sleep parameters uh clinical evidence for magnesium sleep benefits of course all of these are cited and it is used I believe claude5 or 3.5 CET to basically take all of the information from the research papers and create it into this beautiful display and then if we wanted to jump back in and check all the sources we can check the papers here so really cool for doing some scientific research here and just aggregating it and then of course if there is a chance of hallucination we can go back into the original paper and see if that information is correct so a huge month uh for February 2 if you'd like to see uh anything in a specific in a future episode leave a comment below but otherwise I'll see you next month