Building an OpenAI o1 Clone with Nemotron, RunPod, and Open WebUI

Key Takeaways

the 01 model is recognized as one of the most powerful reasoning AI models today often outperforming experts on complex PhD level questions yet many clients remain cautious about using open AIS models due to data privacy concerns now imagine the possibility of building a model comparable to 01 that runs locally on private infrastructure a promising solution is nvidia's brand new model llama 3.1 neaton with just 70 billion parameters it's compact enough to run locally or on GPU providers like runpod yet powerful enough to outperform models like GPT 4 and Claude 3.5 Sonet in benchmarks in the next few minutes I'll show you how to build this gp01 clone and run it locally or on runp pod complete with an easyto use interface like open web UI we start on the AL website and we see that Al provides us with various versions of Open Source models we search for nimron and we see that the 70 billion parameter model is made available to us and that's exactly what we want to use first we need to install Olma depending on which operating system we have in my case I'm trying it locally first and downloading the Mac version after the version is downloaded and installed with a double click I can start Alma with the command Alma run and then the name of the language model I say hi and the answer comes however you can already see that the generation of the answer is relatively slow this is because the 70 billion parameter model is running here on my Mac Studio which wasn't specifically designed for language models this means for daily work it would certainly be too slow and not a real alternative to other options therefore I would like to present a way how we can install the language model not locally assum we don't have strong computers with a powerful GPU but rather how we can rent an instance with a strong GPU and access it locally with programs like open web UI the first step to enhancing your nitron experience is to visit the runpod website and sign up or log in and the first thing we need to do at runpod would be to check if we have credit as these instances with large gpus are not free in my case I don't need to top up because I still have 17.92% credit and I can start directly with installing a new pod here we see a selection of different GPU instances I choose the A600 which is quite expensive but also very powerful with 48 GB vram that should be more than sufficient to start the language model it's important that we increase the disk storage otherwise the model wouldn't fit when downloading additionally we want to expose Olma to the outside side so we need to expose Port 11434 then we have everything we need and we can deploy the instance it takes a few minutes until the instance is available and after that we can access it in different ways spinning up GPU instances with run pod is straightforward but it's essential to keep an eye on costs for example running an instance for an entire day would cost me approximately $18 when we click on connect we have the option to connect via SSH in my case I want to do this via a web terminal so we simply click on the start web terminal button and we see a shell in the browser that we can interact with I copy the command to install AMA on Linux and within a few minutes AMA is installed we see that Olma is now available on port 11 434 witha serve we can start Olma and now we need a second window to Testa for this we go back to the one pod page where we manage our pods we go back to connect and connect to web terminal we can download our chosen language model by entering Olma pull followed by the name of the model in this case nron I have accelerated this here significantly because downloading the 42 gigaby took a few minutes now we can start nitron with Olma run nitron and we say hi again and we already see that it produces the answers much faster I'd say it's as fast as GPT 4 which should be sufficient for our needs when it also accomplish the quality and the reasoning similar to gp1 we would have a perfect privacy solution let's pose one of the trickiest questions for language models how many ours are in strawberry and it answers one we see here why we need to refine our prompts and force it to think very carefully because when we ask twice it eventually comes to the right answer but what we want of course is for it to do this correctly the first time we'll get to that in a moment but first I want to show a way how we can access our Olma instance much more comfortably with a graphical interface for this however it's necessary that we can access our instance via an API call I'll try this out with curl first a curl request against Local Host where I simply specify the system role as you are a helpful assistant and just say hello and if all goes well this request should go through and be answered it takes a moment since we don't have streaming of the answer and it takes until the complete answer is generated there it is and we can see that under the key content we have a similar answer to what we just saw on the console good now our goal is of course that we can reach our AMA instance from the outside and for this we need to make a small adjustment this involves making Al respond on to external connections and for this we set the environment variable Alma host and set it to 0.0.0.0 now we can also try locally to access it via the URL of our runp pod machine we're using the URL of our runp pod instance the 11434 in the first part of the URL specifies the port we want to access and here it looks good too great the next step is then to try to make contact with open web UI to this Alma instance installing open web UI is straightforward links to the documentation and a step-by-step video tutorial are provided in the notes under settings we can adjust the alma URL accordingly and use the URL to our runp pod instance we verify the connection to see if it works and it looks good and now we should be able to use it with open web UI we select the model say Hi how are you and we see the answer and also with the same speed as we just had directly on the runp Pod instance the great thing is that you can also add your documents to open web UI ensuring everything stays 100% private perfect now it's about adjusting our prompt accordingly in the last step for this we go to the settings and enter an appropriate system prompt you are a highly logical assistant whenever answering a question you must provide a clear step-by-step explanation of your thought process and now we ask again how many RS are in strawberry and we see that it now starts to actually break down the task into different steps and I suspect that it will find a correct answer to this task that is very difficult for language models and the answer is correct and we see that we now have a very powerful language model that is exclusively available to us