Implementing DeepSeek-Coder-V2 on Free Google Colab using Ollama

welcome fellow Learners in this video we are going to implement DEC coder V2 a recently released open source mixture of expert code language model we are going to implement this model in free Google collab using ol this model achieves comparable performance with respect to gp4 Turbo when compared in code specific task if you go through its paper they have compared this model's performance with various close Source models such as gb4 turbo claw 3 Opus and Gemini 1.5 pro model this model model achieves Superior performance compared to these close Source models in coding and math benchmarks this model comes up with 128k context length and also supports 338 programming languages this model comes in two variants the base variant and uh instruction ftuned variants and also with respect to the number of parameters this comes up in two variants 16 billion parameter and 236 billion parameters so these are pretty huge models and we cannot run these model directly in our free Google collab or either in our local uh PCS or laptops which are having either 8 GB Ram or 16 GB Ram or having 6gb or 8 GB of gpus so we need to quantize these models for the quantization part we are going to use AMA so let's jump into the implementation part of this DFC coder V2 model in our free Google collab using AMA so uh let's first connect to a GPU server I have already connect to a T4 GPU you can connect to GPU in this uh collab notebook using this runtime then you can connect to a change runtime type and then you can select this T4 GPU and then click on this save I have already connected to this one uh you can connect it from here and the first thing that we need to do here is we need to install this olama uh to install this AMA you can go to this uh GitHub page of this AMA here they have provided a command to uh basically install this AMA in a Linux server you can copy this command and uh you can paste it over here just append a simple exclamation mark at the start of it and run this cell this will basically install this olama in this particular kab notebook uh once we have done this uh installation part of this AMA in this particular notebook we are going to use Python to use this AMA and deeps coder V2 model to test it out in this particular UL notebook for that part we can go to this python page of this AMA where we can see how to use this Ama in Python so for that part we first need to install AMA using pip also so you can copy this one and just paste it over here and run this cell now this AMA will be installed in our uh this particular collab notebook next uh we can copy copy this code directly from over here this particular code to run this AMA in this particular PAB notebook copy this code and paste it over here uh just we need to change this Lama 3 to this deep C coder V2 to check the correct name for the DC coder V2 uh in this particular AMA how they have used it in this ol you can go to this uh web page of this AMA where they have provided all the models available in this AMA so you can see like GMA 2 Lama 3 qu 2 these are models available and then you can see this DC coder V2 is also available you can click on it and then you can see all the different quantied versions of this model are available uh in this page of this AMA you can see like both of these 236 billion and 16 billion variants of different different Quant versions of these models are available like 4bit 2bit 8 bit and so on so we are going to use this 16 billion light version with 4bit quantied and instruction fine tuned one so we are going to use this 16 billion uh light instruction quantied to 4bit model so let's click on this and uh from here you can see uh this is the particular name of this particular model uh DFC coder V2 available in this particular AMA web page so you can copy this one and just paste it over here just remove this uh particular AMA run and let's uh uh run this cell so so it shows like there is connection refused so we first also need to connect to uh AMA in this particular notebook now this AMA application is being installed but we haven't run that ama application in this mod for that part we uh we will be using this sub process uh because we don't want this AMA to restrict our this main threat so for that part we are going to create another process and to create another process we will be using subprocess and from subprocess uh we can directly pop in so process. popen and then we can pass this AMA uh serve this command will basically run this AMA in this particular notebook we also need to pass this cell equals to true so now run this cell and this particular olama will be running in another uh process of this particular notebook now we will be having this AMA running in this particular notebook Let's uh run this uh cell and we will see uh See output of this okay so so it shows that DFC coder V2 is not found over here so for that part we first need to pull this uh particular DC Cod V2 in this particular notebook so one thing I missed it I just need to copy this one and then we can run it directly over here and this will basically pull that particular basically DFC coder V2 model in this particular uh collab notebook now you can see this is basically downloading this uh model and if we go to that page it is of around 8.9 GB size so uh this will take a while and uh let's wait for a while so okay so now you can see it has downloaded uh this model and successfully uh F that model from this particular AMA and uh it is now running an interactive cell so we need not to use that so just Interra that uh interupt this execution and let's run this particular AMA using our python code so now let's run this cell and let's see uh okay so it's again giving us giving us this error so uh it might be that sub process has been uh uh closed or basically uh killed because um it was being used for earlier process so let's uh again use uh run that subprocess subprocess do Po and then uh pass that ama serve and then uh pass another argument s equals to true and then run this cell this will again run over this AMA in this particular notebook and then now run this particular cell now you will be seeing uh this uh particular uh AMA is able to use this DC coder V2 model which is basically quantied to four bits and it is instruction fine tune model so the query is simple why is the sky is blue and let's wait for a while it might take a few seconds to answer this query now you can see it has given us this response but this model is basically a uh code language model so instead of this let's ask some different questions like write a bubble sort algorithm algorithm okay uh so let's see how does this gives us the result and does it give the correct algorithm or not so let's wait for a while okay so it has given us the result you can see first it has given us the pseudo code and we have we didn't mention any particular programming language but uh uh even then it has given us code in Python language so let's uh copy that code and uh paste it in our code cell and see if it has given us the correct code or not so let's uh pass uh simply pass an array so let's pass an array uh which is unsorted like 5 2 one or then something 10 or something let's pass this array to this particular uh function and see if it is able to sort that uh array or not so you can see it is able to sort that array and you can see like it is correctly using this bubble sort algorithm so let's ask some uh a little little difficult algorithm like write uh edit distance algorithm okay so uh let's see uh is it able to write that edit distance algorithm correctly or not so like we also need to consider like we are using this quantized version so might be like the answers may not be as accurate as it is with the uh full Precision U version of the 16 billion parameter of DC Cod V2 so now it has given us this code let's copy this code okay this one and and also this this one okay they have al already provided some uh example and let's see uh is it able to gives us the correct result or not okay so it is giving us zero which cannot be correct because you can say kitten and sitting are very much different than each other so so what could be the problem uh this is not giving us the correct result so like uh okay it is using recursion okay it is uh taking these one but it is not adding any cost to this so so this this might be a problem so so you can see like it is not giving the perfect result when we are using the quantise version okay so so like deep seek people have also provided a a UI where we can use this uh code V2 directly and they might be using uh the bigger version of this particular DFC coder V2 model they might be using that 236 billion parameter one so let's ask same question to that model and let's see if it is giving correct result or not and uh let's copy this one and paste it over here so let's see uh it is giving us the code now it is writing the code over here and it is it looks similar till now and uh let's see if it is able to uh write the exact correct code or not okay now you can see it has added the cost and the uh basically cost or distance you can say over here like one plus this one which is missing in that particular code so let's copy this code and uh paste it uh in our notebook and let's see for the same problem uh what is the distance that it is giving so now you can see it is giving us uh the edit distance equals to three so you can clearly see that um sometimes uh the quantied version might not be as accurate as the full Precision versions so but but for the learning purpose or for the basically the testing out purpose we can always use that and if we are uh need to use this quantize version it is uh uh suggestable to basically fine-tune this model on the basis of the domain specific task such that they can perform better and have less hallucination so great thanks for making till the end of this video goodbye until the next time