LLMs with 8GB / 16GB

Key Takeaways

after my last tutorial on olama and web UI which I did on my 64 GB MacBook Pro some of you are wondering if this is even going to run on an M1 MacBook Air well we're going to do that and I'm going to try it on the M2 base model MacBook Air and we're going to go back in time a little bit for this one I got to keep this one plug I got to keep this one plugged in this is a 2017 Core i5 MacBook Air let's plug it back in that one is struggling a bit but let's start with the M1 I downloaded AMA for Mac OS AMA run llama 3 I believe that this has to first get that 4.7 GB file downloaded which is the Llama 3 model and if you want to learn about all this and what models to pull and get and where to get them I'll link a video down below where I show you how to do all that now that 4.7 GB model size is important especially on these machines with lower amount of ram llama 3 comes in different versions if you go back to the AMA page where llama 3 is hosted and you go to models you can find llama 3 there and it makes two versions available the 8 billion parameter one and the 70 billion parameter 1 yeah guess which one is going to download by default the 8 billion one if we select 70b this one is 40 GB now this is quantized and the quantization it uses is q40 which is about four times smaller than the original 70 billion parameter model what's all this quantization blah blah blah well these llms these models are huge they have so much data in them and they need to be always available if you're running an unquantized model the 7D billion parameter one you're going to need probably close to 140 GB of RAM on your machine which you can get on certain modern Macs like the Mac Studio maybe the next uh MacBook Pro but not this this a 16 so we need to make sure that whatever model we run is going to fit inside the 16 GB here or if we quantize the model which means we just chop it up and make it smaller that should fit within 16 GB or 8 GB or however big your RAM is on your machine this one is quantized by four times so the original is probably 13 GB size for the 8 billion parameter model so we could theoretically run the original 8 billion parameter model on a 16 GB MacBook Air but we wouldn't be able to on the 8 GB MacBook Air but the quantise version of it which is the one that's automatically downloaded by amaama it's four times smaller so yeah that'll fit on both of them hey it finished and now it's running so we can use it through the terminal like this is 8 GB enough in 2024 the internal question memory in test of applications like AI machine learning and multimedia become prevalent and high demand for higher fishing capacity Ram grows blah blah blah let me skip over all this this just writing a lot of stuff but it's going pretty fast and that's the thing about these quantized models the smaller the model the faster it's going to be the 70 billion parameter model is going to be much slower than the 8 billion parameter model but if you further quantize those they're going to be even faster each one of them for general computing and mainstream uses 8 GB of RAM might still be sufficient in 2024 but for memory intensive applications 16 to 32 could provide better performance and multitasking capabilities these things are all true good job llama 3 now llama 2 comes in a 13 billion parameter mode too that's this one right here now if we say AMA run llama 2 how does it know which one is going to get well you need to specify that and you do that right here by saying O Lama run llama 2 13B while it's downloading let's see how we do on this M2 MacBook Air with the original model here 8 GB of RAM enough in 2024 just going to think about it but while it's doing that oh no oh I think we have a problem folks so my mouse suddenly just stopped moving and now it's moving again but I I just clicked on activity monitor and it's not opening up okay now it's opening up good o that's that's a problem the usability is just not there memory pressure is in the orange which means we're we're needing a lot of memory now the way this works is when you do a query a prompt that model gets sucked into the memory and it stays there working away at your prompt and giving you an answer and that's when memory pressure goes up and memory used goes up and when it's done it goes and it just releases the model and now memory is available and now I'm free to use my computer but it didn't quite go away because Mac OS knows that it still might be needed cache files is 4.5 GB compressed is also being used that's half a GB and swap we are at 900 MB swap let's issue a query and see how all those numbers change I'm going to just ask the same question and here we go memory pressure goes up memory used is up 7.48 almost the same amount as the actual model for memory use 7.46 GB swap used is about the same so this is just at the edge of what this machine can do and it's struggling it's struggling when the answer is being generated also you can check out GPU history to see what's happening with the GPU the RAM on Apple silicon machines is shared between the CPU and the GPU but the GPU is much faster doing machine learning inference that's what this is inference is when you are actually using the train model and generating text or images or whatever so it's good to use a GPU for that and you can see in our GPU history that during that time that it was generating the GPU was being used so the GPU was pulling stuff out of memory processing it putting it back and then putting it onto the screen when it was done there's that little blue Mark right there and that correlates to when we were having high memory pressure now we know that the 13 billion parameter model is going to be too big to fit in here on this 8 GB machine but let's do it anyway AMA run llama 2 colon 13B it's 7.4 GB this is the quantized version it might it might run but where is all that model going to be kept is it going to be kept in memory is it going to push everything else out and just use the model it's a tough question answer we're going to see right now it's got to download back on the M1 we got the 13 billion parameter model running now let's take a look at activity monitor now is 8 GB enough in 2024 probably sick of that question memory is up to 14 GB and it's going much much slower than the Llama 38 billion parameter model noticeably slower so why would anybody use the slower models well the bigger models are supposed to give you better ansers because they're trained on more data so the quality is going to be higher I didn't notice any usability issues while it was running though let me try that again and nothing as far as uh usability now I'm not doing a real world test where I'm running all sorts of software at the same time as doing this because if you're doing something like this with a large model you're probably not going to end up using much of the other pieces of the machine if you're going to be doing something like that get 32 or 64 or 128 GB RAM on your machine or run a smaller model so we're at 14 gigs 100 megab of swap that's barely anything we're not exceeding this machine's capabilities and this machine it's an M1 MacBook Air with 16 gigs of RAM is perfectly capable of running this 13 billion parameter model ready for the M2 let's see is 8 GB of RAM enough in 2024 get that model in Ram look at that memory pressure going up going up and it came down it's going up again it's going up again oh and it came down oh we got the word in printed the machine is still usable oh no it's not oh oh yeah that's that's not good but I can't really drag these windows around tooo much it's hitting a wall and then it's like bouncing off of that wall that memory pressure chart is going crazy up down up down up down and we're not getting really much of a response here so this 13 billion parameter model is pretty much unusable unless you want to wait a few hours for the answer you can do that not a pleasant experience the swap used is only a gigabyte so not too bad but this machine is not happy it knows it can't fit the whole model in there so it's emptying the memory and trying again and again and again I'm just amazed that we're getting anything out of this okay here we go now this has a different operating system all together Blast from the Past Monteray and this machine is just sitting here and already it's at 96 94 96° C and the fan is spinning at 1500 RPM that's right you heard me correctly this is a MacBook Air with a fan in it cuz they used to have those we're going to start up Chrome we we'll start up Chrome ol is supposed to be able to run on anything greater than Big Sir so this theoretically should work just can't imagine how much it'll struggle now AMA itself is not a huge program and it's not super intensive it just basically orchestrates your models this is not what's going to be the tough part for this machine Let's Do Run llama 3 it's downloading it and this Mac has 8 GB of RAM in it even though it's from 2017 this is slow Ram by the way 1,600 MHz DDR3 when it comes to machine learning and AI applications like this Ram is very important the more you have and the faster Ram the better Ram bandwidth matters a lot also you might have heard people talking about how they use their M3 Max because the M3 Max has a lot of ram bandwidth the highest end model and the M2 Ultra has up to 800 GB of RAM bandwidth which is a lot I made a video describing all this and testing it out I'll link to it either up here or down below somewhere it says success and right now it says 2.4 GB used let's create a message since this is an older machine I'm going to say is 4 GB of RAM enough in 2017 why not cuz I think in 2024 you can safely say that 4 GB of RAM is not enough it's pretty slow but it's much faster than the 13 billion parameter model on the M2 and let's take a look at this memory it's not using that much memory 2.37 GB it's got 5.6 GB in Cache but I'm really surprised that it's not sucking up all that the whole model into memory what's going on if you know the answer to that put it in the comments I'm sure everybody would like to know now my theory is that memory management was very different in Monteray than it is now in Sonoma again just a theory and perhaps montere is reserving some of that memory for operating system use that's one idea another idea is we have unified memory over there which is slightly differently implemented because the sharing memory between the GPU and the CPU and here we've got the old school DDR3 so to sum up AMA with a quantized 7 billion or 8 billion parameter models will run fine on an 8 GB machine even going all the way back to 2017 MacBook Air but not without this or that fan you want to run something bigger more RAM if you want to check out my Ram bandwidth video watch it right over here and if you want to see my olama tutorial watch it over here I'll see you next time oh