Run LLMs Locally With Docker Model Runner

Key Takeaways

Hello all, my name is Krishna and welcome to my YouTube channel. So guys, today in this particular video we are going to discuss about how we can run LLMs locally with the help of Docker model runner. Now there may be scenario guys that you may be working in various use cases that may be related to different kind of LLMs right and as a developer I definitely want flexibility to run LLM locally so that I will be able to uh do the development for my entire project even to probably test it and also see the comparison between various open source models. Now this running LLM with the help of docker uh locally is now possible with the help of docker model runner. So in this particular video I will be going ahead and showing you step-by-step process how you can probably go ahead with this. Now first of all what you really need to do is that you need to go ahead and download the docker desktop. So if you have been following me following my lot of tutorials related to dockers related to LLMs I hope everybody may be having the docker desktop. If you do not have the docker desktop I would suggest go ahead and download it. And uh this LLM's uh like if you really want to uh run any kind of LLM locally it is now both possible in Mac and Windows. So first of all what I will do I will just go ahead and download the docker desktop. So here uh you can see uh as soon as you go ahead and search for docker desktop here is the second link just go ahead and click on this and over here you'll be able to see that um all the possible options how to probably go ahead and do the installation right so first of all I will just go ahead and click on setup here there is a install option then you have Mac Windows let's say if you want to go ahead with windows you can go ahead and download this particular file based on the version of the windows that you have if you have Mac you can also go ahead and do it for the Mac itself. Let's say you want to go ahead and download the Docker desktop for Mac with the Apple silicon, you can do that. Or if you have Intel chip, you you can also do that. Okay. So once you do the installation, just go ahead and do the installation and probably if you're seeing my Docker videos, I've already taught this particular tutorial. Now once you have done the installation, this is how your Docker desktop will look like right now. one very important feature to probably run this docker uh model runner. If you go to the settings and if you probably go ahead and see like which version your docker is like your docker desktop is it need to have a minimum of 4.4 version. Okay. So if you go back to the article which I have shown you over here it is basically written that now it is available in beta with docker desktop 4.40 40 for Mac OS on April silicon and uh since this article was posted on April 4, 2025. Now in Windows also it will work if you have the Docker desktop of 4.40. Okay. Now uh I already have this particular version. So if you probably go ahead and see my version, it is somewhere around 4.41. Anyhow, now the next thing that you really need to do, you just need to go ahead and click on features in development. And you also need to enable this enabled docker model runner. And along with this since you also want to use the LLM models uh which is running in some kind of docker containers with the help of docker model runner you would also want to probably use this in your coding application. So that is the reason you need to go ahead and click on this enable host side TCP support. So once you do that you just need to go ahead and click on apply and restart. Once you do that that basically means that your uh docker model runner is enabled uh from this particular docker desktop. You can also do this with the help of command prompt. So let me just go ahead and show you with the help of command prompt also and how you can probably enable this model runner once you have installed the minimum version of 4.40. Okay. So first of all just go to the command prompt. Okay. In order to check let's say that you have installed the docker desktop you have the recent version okay or you have to probably go ahead and do is that you have to go ahead and just copy and paste this command. See over here we have written docker desktop enable model runner and since we have already enabled this TCP 1 2 4 3 4. So if you go over here your 12434 is basically checked and your port number is enabled. Right? So once you do this you just need to press enter. Since I have already enabled this so I don't have to do it again. So it is not giving me any message that basically means my model runner is enabled. Okay. Now the next thing is that uh in this we definitely want to check whether our uh you know the model runner is probably working absolutely fine or not. Okay. So how do I go ahead and check it? For this we go ahead and write a command docker model status. Okay. So here once you go ahead and probably uh just go ahead and write this particular command that is docker model status here you'll be able to see that docker model is running and it is probably giving you this specific status. So once it gives you this status that basically means your docker model runner is working absolutely fine. Okay. Along with this, we'll also try to understand what all important commands we can specifically use uh when we want to probably run any kind of uh LLM locally with the help of docker model runner. So for this we will go ahead and write docker model help. And once I go ahead and write this you'll be able to see as soon as we write this command docker model help these are the list of commands that we can specifically use. The first command is inspect which shows display detailed information on one model. The second command is something called as list. List the available models that can be run with the docker model runner. Then you have logs. Then you have pull. Right? With the help of pull, you can download a model. With the help of push, you can also upload a model in the docker hub. Downloading a model also happens from a docker hub. Then you have rm. It can remove the models downloaded from docker hub. Then you have run. Okay. You can also run a model with the help of docker model runner. Okay. All these specific things we will try to use. You'll also see tag a model is there and version. Show the docker model run a version also. So these are all the possible options that you can specifically use. Okay. Now the next step is that see there are different kind of LLM models which you can run locally with the help of docker model runner. But you need to understand which all models you can run. So if you go back to the docker desktop and uh if you just go back to the home okay here there is an option of docker hub right and when you probably go ahead and search in dockerhub right here you'll be able to see various models so let's go ahead and search for it I will go ahead and write a slash let's say I will go ahead and write llama 3 and if I go ahead and search over here right here you'll be able to see that I have various models llama models or a series of llama models which are available in the docker hub. Not only this, right, let's say if I go ahead and click this. Okay, now once I'm clicking this here, you'll be able to see all the information of the llama models. Okay, just to see in a much more better way, what I will do, I will just go ahead and search for DockerHub in the browser. Okay, I will just go ahead and click this. And now let let us search for some of the models that are available. So for this, I will just go ahead and search AI slash. Okay. And when I search AI slash, let's say that okay, uh here AI slash is not coming. Okay. Let's let's search for llama 3. Okay. Llama 3. Now I think you should be able to get the model, right? So let's say that llama 3, llama 3.3, 3.1 are the open source LLM models that are available over here. I'll click on this. Now what I will do I will uh just go ahead and see some of the models like how these are the information of the variants of the models that are there. In order to pull that model I have to use this pull command right like even though if you have seen that how to pull a docker image you can also pull it just by using this uh there will be a pull command that will be available with respect to various images. Similarly in any kind of LLM models that are available in docker this kind of command will also be available. Now see over here I have gone to this AI llama 3.2 and llama 3.2 is again an open-source model. If I go ahead and click on this AI okay there are so many different models that are available. Let's say Quen 3 small language model DeepS R1 distil lama llama 3.3 3.2 2 mal 54 QWQ 2.5 gamma 3 deep coder preview gamma quad 3 quad llama 3.1 right so all the specific models are there right and most of these models and uh all the models that you are actually seeing over here they are like open-source models now in this specific video I will just try to show you with this particular model that is small LLM2 okay now what is this model all about you know so it is a tiny LLM built for speed edge devices and local deployment or development sorry not deployment and then uh you can basically it is a compact language model with 360 million parameters designed run to efficiently on device so we'll try to use this particular model because uh let's say that I want to go ahead and create a chat assistant text extraction rewriting and summarization I can use this model uh the provider is hugging face architecture is llama 2 uh tool calling option also it has uh and various options it has so that basically means like when I we are trying to create our own um chatbot assistant, we can definitely use this. There are also different different uh model variants uh which are available over here. Okay, like 360 million, 135 million, all these variants are there. Now let us go ahead and see that how we should probably pull this model. So for pulling this model, we have this command docker model pull a/ small lnm2. So this model is somewhere around 250 MB, I guess. uh let's see uh 256.35 MB. Okay. And there are other variants of 135 million and all that is just 85.77 MB. So I'll go again back to my command prompt and I will go ahead and paste this docker model pull AI small uh sm2. Right now it is saying that it has basically used the cach a model that basically means I've already downloaded this model. Right? So I'm not again downloading it because it may take some amount of time. You can go ahead and once you probably go ahead and write docker model pull ai/ small lm2 it will take some amount of time based on the internet speed. Once you have the specific model the next thing is that how do we run this model right? So for running the model again there's a command docker model run ai/ small lm2. Okay I'll copy this and I'll paste it over here. Now as soon as you see this uh when we are uh when we wrote uh docker model run ai/ small llm2 now this is my chat window okay so I can basically go ahead and chat let's say I will go ahead and write hi it says hello how can I assist you today can you u write me a poem or can you tell me a fact on cricket to get okay so this is my question so here you can see that I'm able to get an answer so this is just acting like a chatbot assistant wherein I ask any question it is able to reply me okay uh over here you can the best thing is that you are just able to do this kind of interaction and you are able to work it uh as a chatbot assistant and understand this entire model is basically getting run locally with the help of docker model runner right so you can go ahead and ask any question I'll say that hey uh what All things can you do? Okay. So here you can I'm a helpful assistant named small lm trained by hugging face. I can do a lot of things. I can do a variety task. Summarize find information. Summarize news article. Create short stories. Let's say I'll go ahead and write create a short story for a king and a princess. Okay. I I'm just writing something. So here you can see once upon a time in land far away there was a king and princess named Arthur Merlin all this information is basically there. Now since these are our open source models that we can specifically download in our local and play with it you know it is quite amazing. Let's say you want to probably go ahead and exit it. You just need to write /by and here you can see that the chat session has ended. Perfect. And it's quite amazing over here that you are able to do this uh in a very much easy way. Let's say uh I will try to show you one more way how we can probably call this specific or run this model also. Okay, just by directly giving the input during the runtime. Okay, so for this what I will do now I'll go ahead and write docker model. Let's say I'm going to run this AI small small llm2 lm2 small lm2 and here I will just go ahead and give me a give a question to the docker itself to this particular model give me a fact about let's say elephant and here is my question directly right isn't it amazing because here I'm not just going to act like a chat session it should not be a chat session Here we are just going to give a input during the runtime and here is what we are getting an output. Now these all are amazing. Now let me talk about some more interesting facts about this models that we are running locally with the help of dockerm right docker model runner sorry. One amazing thing is that this is completely compatible to the open AI library. Okay. So if you have been using this openAI library uh you can directly go ahead and probably use this over there the URL that you specifically give you know will be this URL that we have already seen over here enable host site TCP support this will be the port of the URL along with local host. So let me show you one example and this will probably give you a very good example. So here is my docker demo. Please focus on this example. When I say it is completely OpenAI compatible, that basically means we are using the library of OpenAI. Okay. So here you can see from OpenAI import OpenAI. Now I'm going to go ahead and create the client. This client I'm creating it with OpenAI. The base URL is nothing but localhost col 434. And now you remember this from where this port came, right? So if you see over here, the port is nothing but 1 2 4 3 4. You can also go ahead and give your port. But by default this is the port that we have got. The default port is this. Okay. Now here you can see that I'm giving /ash engine / v1. Okay. By this URL you will be able to access the model uh the lm model that is running locally. Okay. And here the second parameter is open API key. Here we need to give the docker by default. If you probably go ahead and see the documentation you'll be able to see this. Then we go ahead after creating a client we write client.comp completion.create. Here we give the model name. Now this model name you can again verify from the docker itself. See if I cancel this and if I probably see the models here is a model name AI/ small lm2 and that is the model that you'll be able to see over here. Right? Then here you have messages. The role is system. I'm saying answer the question in a couple of sentence and this is my question like share a happy story with me. Okay. And here we are probably printing the output of this specific model. Okay. So if you want to probably go ahead and see this, how to probably go ahead and execute it. I'll just go ahead and clear the screen and I'll write python docker demo. py. Okay. So once I execute this, you'll be able to see that here I'm able to get the answer. I had a wonderful time at the beach with my family. We built a sand castle together. Something something some information is basically coming up. Okay, which is good because here we are using this openi library itself. The best part is that you don't need to probably learn anything other than what you have already learned, right? And many people who have been working in building generative AI application, they can definitely they know about this OpenAI library. Let me show you one more way of doing this. So here is a streaming. So there is also an option of streaming in uh this OpenAI library. Okay, here we need to just set this parameter to stream is equal to true. All the code will be set same. My base URL will be same. My API will be same. My model name will be same and here you can see that my messages for system and user will be same. The best way is that I can also convert directly this into a chatbot application or chatbot assistant. And here we have just mentioned that stream is equal to true and we are trying to probably go ahead and display the content. So once I go ahead and also run this okay python streaming.py you should be able to see that my entire output will get visible. I'd be happy to share a short story with here it is last summer all this information here. Now this you can see that is not directly displaying all the content all at once. It is streaming it one line by line by line it is probably going and displaying it. Okay. Similarly u I've also developed something related to function calling. Okay. So if you are very familiar with this open air library it also supports tool calling function calling all these things and here uh I will try to provide this particular code in the description. uh you should also be able to go ahead and use this. So I hope you like this particular video and understand whatever things we have discussed with respect to this. You know you can now run LLMs locally with dockers with the help of model runner docker model runner both in Mac and in windows. Go ahead and try it out. All the steps are almost same. All the code description of this particular blog all the information will be shared in the description of this particular video. This was it for my side. I hope you like this particular video. again a new day where I have probably taught you some amazing things which you can definitely go ahead and try it out. So yes, this was it for my side. I'll see you in the next video.