OpenAI's NEW API? Tools to build AI Agent SWARMS?? | Unscripted Coding

welcome everyone to unscripted coding today open AI announced a bunch of new features for developers that's you and me um in this 19-minute video on X I will fully admit I I wasn't going to watch through these because generally they don't tell me a whole lot and I thought we could together look at the documentation and figure out what's going on now we don't have to go into a documentation just yet uh the post does highlight five features for us the first three look like new tools uh web search File search computer you sound like things that Google and anthropic uh have had with their apis largely searching the web searching files on your computer and finally computer use is a cool one where you could send a screenshot uh for the llm it responds which coordinates to click with mouse or what to type in and it will very slowly for example sign up for X if I tell it to by clicking this seeing what's next then the next then so on the two that I find more interesting and you could see from the time stamps that they spent a whole 6 minutes or so on responses and another 9 Minutes on agents those are the two big ones uh I want to dive into deeper now I guess the first question is what is well let's dive into responses first and the first question there is what is it well so far most of us have used open AI as an API with chat completions um we send part of a conversation or a whole conversation and it predicts the next piece so for example um you know the the AI would say something I would say something it would say something we keep this all logged and then I send it all together and it predicts the next phrase that's really where chat completion comes in um all the end points there are to basically manage this conversation responses seems to be advancing this giving it the use of tools giving it more features and probably cleaning up the API a little now what this comparison doesn't talk about is that open AI also had another similar API endpoint called assistance and this never picked up even I barely used it because it was hard to use it didn't make much sense basically you were creating threads and and it somehow you know you have to manage it even though open AI kind of manages it it was difficult and I never really saw the benefit over chat completions um other than tool use which is huge I think responses API is a pun intended response to this it looks like it packs some of the features most of the features of assistance but makes it way way way easier so what does that mean well in this comparison it sounds like you could do a lot on chat completions text generation audio Vision structured outputs function calling and actually web search is surprising to me here um but the the first five of these are all really just carrying the conversation where responses is adding is web search File search um computer use and probably additional tools in the future that require different styles of inputs and outputs uh they clarify that chat completion is going to stay along um of course because so many apps are built on top of it they can't possibly get rid of it but um let's just take a look here now I copied this code into vs code already and we're just going to run it to show you that it can can um it it it it can carry a basic conversation so let's just run it and it's going to take a second I'm sure and hopefully it spits a one- sentence story back out to us now very good um one sentence respons a unicorn yes so that's basic we could have done this not with responses but with check completion instead the benefit here is it looks like we're going to be able to input a whole bunch of different things so what I want to look for is this tools um and luckily we can do this tools and over here we have um the type and input so I'm going to copy this again and just find my screen here and paste so we're going to get a response we're going to print out the output what's a positive news story from today that's a good one and I'll tell you very quickly if it's something that happened today hopefully you'll know it and we're just watching down here for the response and March 12th 2025 technically it's the 11th but depending on your time zone could be March 12th um Oshkosh Chamber of Commerce I I have no clue where that is but uh this looks like a okay um I'll accept that as true for now or you know what I can copy this come back here and see if this is a yeah this is a legitimate February 24th is there a March 12th day in here okay so not great but it is a real article it is a real response it is Rel relatively recent so I'm not going to fail it just yet um the other one I wanted to bring up is file search so I think we're going to have to open a new tab here but you can see that we can store Vector IDs and Maxum results um largely that means that we're going to store our information with open Ai and this is something I haven't seen yet but how it worked in assistance API is you can upload a bunch of files they persist within open Ai and you're just referencing uh I I think an ID number of all your batch files together I suspect this is the same in um in responses here so what we can do is we can uh create certain things so what is deep research by open AI we have a file search and no that's not quite right here we go create a vector store and upload a file so we can create a file bunch of different things um and you can see the client is still open AI files create purpose assistance so this is exactly the same process as before and what happens is you can get a result at the very end now I'm not going to go through this um right now but within open ai's playground you can open up an assistant you can create it within a web interface and upload files to it but the big problem is there should be file size limitations you can't just upload say your entire um entire uh knowledge base from your business you know millions of files millions of customer records whatever so um here it looks like well this is quite generous this is quite quite a bit bigger than I remembered it uh total size of 100 gigs for all files um total of 10,000 individual files and each file can be a Max of 512 uh that's kind of interesting um oh I suppose okay here's where I'm going with this there's 512 megabytes so let's say half a gig 10K files at Max you can have five 5K uh uh 5K G okay never mind um Mind Playing Tricks here so 10,000 files each at half a gig 5,000 gigs okay yeah that's okay uh doubting myself it is starting to get late at night so this is possible you're very likely to be able to store some significant information in here so their Vector store has gotten much better which means that they're storing all this data and they're presumably fetching it very very effective very effectively um it's going to be helpful to see how this this works but uh again I don't have a vector store ready to go as long as I understand what's going on this is help helpful for me uh this is way easier than assistance you're just going to upload it from playground and then you're going to get the the ID place it here and now you can just call it as a tool very very simple um function don't really care uh that that's just typical calling a function we've seen that in the past uh I am just curious what this ah I see reasoning um you can adjust how much time uh the 031 the O Series of models will will spend so that that's help as well I don't I think this exists in chat as well okay so we have gotten a good sense of what's happening with um responses here actually let's take a quick look the response looks exactly looks just like before delete a model response let's input items okay okay um so when I was doing propep work for this before uh the the list of items was somewhere else so it's handy to have uh here as well I think what's going to be really useful for me is this web search one so uh you know what let's try that one more time because I think this is the biggest one I am going to to use and I'm just going to say um what is the F case for AI IP ownership and you know this might be something in its knowledge base now but I suspect it will have to search the internet to to respond to what it is this is a really important and interesting case if you are looking at can you own things that AI produc uses and here uh that looks like a pretty good response but there is no link like before so who knows if it fetched off the internet but this will be one to play around with the next one I wanted to talk through is the agents SDK this is one where I I really do think we're just going to talk through rather than play with code because um the use of it is probably quite limited this is very early but the idea is almost like Lang chain in which you want to use and take advantage of different models capabilities um if you look here open AI has summarized it very quickly you want to use the 40 Mini model for something low latency if you need responses very quickly so say you're responding to a user in real time or you know generating a conversation say an audio conversation uh you need something that that runs very very quickly low latency uh responses will be Dumber because it is low latency now if you scale it all the way to a top 01 and 03 mini are not fast models because they have that thinking mechanism so it makes sense that they're best for long-term planning hard tasks and reasoning what I think is useful to understand about this agents adk uh SDK is that once you start building this out you can create a number of different agents uh and each of these ass uh agents will have their own Persona uh they could pretend to do different things they can be given different tasks but they can be different different models uh different um running different tools and the idea is having multiple of these agents interacting with one another is going to be helpful [Music] in achieving an objective and I'm being very vague here I'm just going to install it on the back and see if we can run something um because not a whole lot of people are using agents effectively we're still not in that area where a lot of people are are thinking about how to use agents but if I think about my classic example of a very dumb obvious way to use it is to give AIS different personas and have a conversation watch them converse with each other to get to a end result and one example I have of this in real life of courses I am in meetings all the time with people of different backgrounds different professions and so if we have say a business objective uh I might bring a legal opinion and someone else has a perspective as a finance or or accounting background somebody has a marketing PR background and say we're dealing with a big um PR disaster finance will be going how cheap can we make this I will be trying to cover uh all sorts of legal liabilities and the pr person's talking about how to spin this properly and so the final draft of a document shouldn't be written by any one of us now if you ask an AI to to do that you might be able to have it kind of Juggle the three but it's way it's way more interesting to watch multiple agents kind of build out their thinking process watch them duke it out and argue with another one another until they actually produce a document now that's just one way another way is Lang chain has been doing I don't think they call it agents um I don't think Lang chain calls them agents but uh you can have multi-step pieces so for example uh we go back to Which models are good here um maybe we use 01 and 03 to plan the task and then we need the tools from 4.5 but we want to do a series of checks so once 01 and 03 plan out the the plan of attack maybe 40 mini can look at a whole lot of documents very very quickly just to pick out the the date from each one and then finally you have uh 4.5 code up a chart uh a interactive chart to to display the information you found from pulling out all the dates I'm just making this up as you go um so certainly the agents SDK is going to allow you to choose models tools knowledge separately but what's interesting as well is that you could also they are thinking about how do you build applications and guard rails is something that really I have not looked into or thought about at all um and so they're really trying to build out something where we could create the whole chain very effectively um since I have uh installed this let's try and do this hello world example here um I think we're going to need to pass in an API key here and I will just comment this out and let's fingers cross see if this works and maybe not API key um don't want to mess with my environment variables just yet so I think we're going to leave it at that um these are all interesting uh none of these are very complete so uh just a few days ago there was a Chinese AI launch called Manis Ai and had a lot of these agentic functions so I feel like open AI push these out quite quickly the only one foreseeable that I would be using quite quickly are web search and file search uh I might be using the responses API just because file search is required but uh I'm glad to see that chat completions has a web search so I don't have to migrate over just yet uh this is the big one for me web search uh we don't I need to use it all the time and I don't have great options there's lexity sonar uh there are tools like uh Gemini's grounding um and more is better um just going to see how all of these perform okay I'm just going to end it here I've been rambling for long enough and thanks for watching bye