OpenAI GPT-4o mini vision capabilities using API - Multimodal Development with OpenAI API

Key Takeaways

welcome to the fifth video of the multimodel development with open course GPT 40 and recently announced GPT 4 mini have Vision capabilities which means that these models can take image as input and answer questions about them in this particular video we'll learn how to use these Vision capabilities using Python and open AI API if you have missed the previous videos I've posted the links to the entire course in the description below so let's get started to start with I've activated my virtual environment and typed in Jupiter lab to open my Jupiter ID it opens up in the browser here where in the projects directory I've created a new python notebook now there are two ways to send an image to open a let's look at them one by one in the first method we send an image to open a via an image URL my messages object is a list here which contains one dictionary this dictionary has two keys role and content we've already seen the role as system and user in this particular case we are passing in the role as user the value for content is interesting it is a list which contains two dictionaries the first dictionary has two keys type and text the second dictionary again has two keys type and image URL so we can understand type is of text and text contains our input question and in the second dictionary type is an image URL and the actual image URL again contains a key value pair where key is URL and the value is an URL which contains an image here is that image it is a receipt and we are asking it the question what items are present in this receipt based on this we get a response from openi which matches my image another way of sending an image to open a is by converting that image to base 64 format I've written a small function here which takes in an image as input reads it and converts it into base 64 format the messages list is almost the same as is the first method the only change here is the value of the URL in this particular case instead of the image URL we are specifying that image is of type PNG and it is in base 64 format and then we append the base 64 encoded image that we received from this particular method I'll show you the image that we are using here this is a different image and hence when we ask the question what items are present in this receipt we get a different response which is total of $77 which matches this particular receipt so these were the two ways in which you can use the vision capabilities in the new model in the next video we'll learn how to use function calling to enhance our project please like the video and subscribe to the channel thanks for tuning in see you in the next one