Microsoft OmniParser: Best AI Screen Parser to Control Computer?

Key Takeaways

now we have Omni passer from Microsoft if there is a user task a restaurant in Johannesburg with vegan option then you can see it's passing the screen with each individual elements then it's taking action by clicking restaurants now it's identifying the elements and typing the city name Johannesburg then it's clicking the link and off that it's choosing vegan option by identifying and finally we got the answer toss complete this is using Omni Parts the model released by Microsoft which is capable of extracting every elements from a screenshots including its exact position if you see the performance for omnip posa with GPT 4 V it's comparatively much better than the general GPT 4V version even when you compare that with other models with Omni paa this looks much better in element extraction from a screenshot in this we'll be seeing how you can run Omni passor in the code format in a notebook format and also in gradi your user interface here is a user interface I'm going to take the screenshot of this page uploading that here and then clicking submit in omony parer here you can see the output it's able to identify every single element from that screenshot that includes the exact position that is super cool by the end of this video you will learn how you can run this Omni parsa in your computer with GPU and having the ability to run a demo like this going through omnip Passa research paper the code released in GitHub and it contains thousands of stars in just few days and all the models available in hugging face I'm going to take you through step by step but before that I regularly create videos in regards to Artificial Intelligence on my YouTube channel so do subscribe and click the Bell icon to stay tuned make sure you click the like button so this video can be helpful for many others like you first step get clone get t.com Omer and then click enter now navigate to the Omer folder now we are inside that folder next pip install hyphen R requirements.txt and then click enter this installed all the required packages next I have a bash script using this you are able to download the Omni parel model from hugging face I will put all the code in the description below so I'm going to run this code I've saved that in download. sh file so in your terminal Bash download Dosh and then click enter this will automatically go through individual link and then download the model now the model weights got downloaded in appropriate folder and it converted that to best. PT that's a required file so everything is taken care of by just running that one script next just type Python gradior demo. and then click enter this will automatically run the gradio interface as a side note I'm using this configuration which with RTX a6000 graphic card and a virtual CPU and here you can see the gradio URL is ready for us to view so I can just open this and here is the user interface so here I'm uploading a screenshot and then clicking submit now it's processing the request now we clearly identify the different elements in the screenshot and you have the list of past elements there are two more things which I want to show you which is how you can run this Omni passor in the code format and also in Notebook so in regards to code you have three different steps first step is to import libraries Second Step configuration and third step to pass an image so now I'm going to show you how you can code so step number one importing libraries so these are the libraries we are importing now step number two configuration so I'm going to add few configuration setting the device Cuda and then loading the model this is where the model got downloaded you can see that in the folder weights I can detect and here you got the models and and assigning that model to the device that is the Cuda device then we are loading the caption model processor so here we are using icon dedi that is the YOLO model and for caption we are using Florence 2 then we are loading the image from the path so if I open this image so this is the image and we are going to pause elements from this image so that's what's here we opening it converting it then giving some settings so I'm keeping all these default now final step is paring that image so step number three get labeled image and results so in the step number three we are using two different function check OCR box and get label image using those two function and we are passing all the configuration here whatever we defined at the top then using that we are getting the decoded image and then finally saving it that's it only three steps importing libraries configuration and running the model now I'm going to run this code in your terminal Python app.py and then click enter this will automatically load the model run it it's using exactly the image which we have mentioned before and parsing the result it's saving that result in a file and printing out the elements detected so if I open the saved image so here is the labeled image labeling every single element from both of these tabs this is super cool and also I can see it's very accurate which means we can EXT this further to take action using all these elements and its position same as running code you are able to run that in Notebook so if I open the folder you got demo. iynb that is a notebook click on that there you got the complete code so it's just matter of clicking run all it'll automatically run every single box and here we are providing two different Image Google page and multi-tab windows it's using the same code which I explained before and then finally it's giving you this output ability to detect elements in the screenshot and I'm really excited about this so when you see the paper GPD 4V as a general agent is largely underestimated due to the lack of robust screen passing technique so the current system lacks two different things one is reliably identifying icons within the user interface second understanding the semantics of various elements so to solve these two issues we have om Passa which is capable of identifying icons and understanding the semantics of these elements this is achieved by curating interactable icon detection data sets using popular web pages and an icon description data set then these data sets are used to fine-tune specialized models which is able to identify elements from the screen and caption those elements with semantics so technically they use two models one is the fine-tuned version to detect elements second one is the caption model to understand the semantics so here in our code first we used this model to identify elements this is the fine-tuned model then we use Florence to the caption model to do semantics now you are able to easily extend this further to create a full-fledged application which is like this where you are able to provide a task and it's able to completed for you considering you like omnip Passa and it uses Florence model of already covered Florence model the captioning model in more detail which gives you in-depth knowledge on how these images gets processed and I highly recommend for you to watch that and I will put that link in here so I will see you there