Vector Embeddings and Tokens

Key Takeaways

When we talk about AI or LLM, the foundation part there is embedding. Now before we talk about embedding, let's go with this. So let's say you talk to the chat GPD or Germany. Basically what you do is you write some text and that becomes your prompt and then you get some response. Now of course this can be text, images, right? So you can pass an image to to the model and it will give you some information or you can give text and it will give you images right so it goes in that way but the thing is when you talk to the AI model let's say if you type a prompt now what happens there of course it has to process it and then it will give you some information there are a lot of algorithms which goes behind the scene which make it work but even before we go there even before your AI try to understand what you're talking about what's a process so what happens is let's say if you give your sentence right now this sentence gets converted into tokens and then this tokens get converted into embeddings. Now based on this embedding it will apply those algorithm whatever formula are there we are not concerned about the formulas or algorithm here we are more concerned about what you give to the AI model right so those are the embeddings which you pass to the AI model because AI as it's a software it will only work with numbers right now you might be thinking okay we are doing this from a long time right so if even if you write a normal software or computer in general we if you want to pass a sentence first you convert that sentence into characters from characters to binary format that's how thing goes right even for the images you convert the image into a binary format but then if you want to work with the AI model you can't basically work with binary format it's not just searching for a word example let's say if I if I have a document and in that document if I'm searching for let's say cat now of course if I write an essay on cat and maybe in the entire essay I'm not using or the let's say article in the entire article I'm not using cat anywhere everywhere I'm saying kitty. Now in this case if you say Ctrl F in that document and say I'm searching for cat your finder will say I'm not able to find any cat here but then your AI model is smart right so it will know you're talking about kitty right so kitty's cat so how that relation builds and you can get that with the help of embeddings okay so let's go back here so there's something called vector embedding so we are on open AAI documentation and most of the models provide you embedding not every model example Antropic they don't have their own embedding models. They are using some third party models if I'm not wrong. But OpenAI has their embedding models and they have multiple. So these two are famous. One is text embedding three small and large. Again that depends upon how much money you have. Of course it gives you more accuracy when you have a larger model. But do you need a larger model? That's a question. Anyway, that's a cost thing. Let's go down. So we basically use embedding for searching. As I mentioned before, if you're not searching for kitty, you're searching for cat. And if your document has kitty, it will still able to find it. You can also create groups based on the similar words, right? And text recommendations when you type something on your phone and you get some suggestions, right? So these are the things which you get there. But how exactly this thing works. Now when you want to do embedding what you try to create is a vector. Okay? Now again there are a lot of words here. Let's try to keep it very simple. So let's say you got a word cat. Okay. Now you want to find a number for it and randomly I'm giving it a number. So let's say cat is 55. Then let's go for dog. Let's say dog is 60. And then you go for the next word. Let's say laptop. And laptop is let's say five. And then you go for computer. Computer is let's say eight. Now what we're trying to do here is when you give a number to each word here. So we basically we're trying to embed into a number and what you get is vector. Again why is called vector? We'll talk about that. But you got vectors. So these numbers are here vectors. Now looking at these numbers, you can figure it out that cat and dog are similar because the numbers are close. But dog and laptop very far right. So 1 is 60 one is two but then or or five doesn't matter but when you compare laptop and computer they are near nearby. So this is how you group common words right and embedding can help help you there. Now question arise how will you represent that number is it only single number but the problem is if you just have a single number imagine the vocabulary we have not just in English we lm supports multiple languages if you just have this numbers there will be some confusion right. So if in future you get grass and grass is seven then you'll say okay grass is very nearby to computer then laptop uh so we we we don't want that. So having this number is not enough and that's where we can go in a graph structure or two dimensional structure where you just don't have one line where you have all these numbers maybe you can have two dimensions where you have x-axis and y axis. Now every number here can have two values because we got two dimensions. So let's say the dog is 60A 2 then we got or whatever the number was and cat is let's say 55a 6 and list goes on you can see on the screen. So if you have this numbers then you get two dimensions and you have to make sure that again cat and dog are nearby not very far just because you add extra dimension it can go far. So those things you can do you can have two dimensions or you can have three dimensions right so x y z. So the number of values increases as your dimension increases right and these are called vectors because when you represent something on the graph you give directions right. So from the point where that value goes and that's the direction and you do that with the help of vectors. So each value here is a vector. So whatever you got got for dog that's a vector. Okay. Now question arise how many dimensions are enough? maybe three uh no maybe five 10 100 let's see if you talk about the embedding model from openAI which is the text embedding three is small and if you see their embedding size so you can they have it the length of the embedding vector that means the number of values you get and the number of values depend upon the of course the number of dimensions you have so for small it is 1536 dimensions you can see the number of values again we'll see all those things in practical and for large it is 3072 huge right now I have actually also checked for Germany or entropic which uses some third party they support 1024 okay so I think open AI has a bigger dimension I'm not saying better but bigger dimension emitting models okay so what do you think how it works so let's say if you give a sentence of course it will break that sentence into words and then from words it will convert that into vectors uh no we have one more step in between which is when you get the sentence it will first break down that sentence into tokens not words and how that works so we have a concept of tokenizer or tokenization right so what you do is you get a text let's say I'm adding a text uh we say my name is Naven ready from telisco now this is a text right so what it will do is it will convert that text into tokens now the number of words 1 2 3 4 5 6 7 words but if you go down look at the tokens we got 11 tokens and This is how it breaks it. So my becomes one that's a common uh word. Name is a common word is a common word. Now n is not a common word. So it is breaking down into two parts. Nav in maybe because in is a word uh then r e ed okay from okay just going for yeah so one word then again we got three breaks there. Now it's not like every algorithm will go in this format. Example, we got JPD 4 which has 11 tokens but if you go for 3.5 okay still 11 tokens three still 11 tokens okay nothing changes for this sentence but let's say let's take the example you can copy paste any Wikipedia article here or when you click on show example they have their own example here number of words I'm not sure but the tokens is 53 4 but the moment I go for 3.5 57 when I go for 3 64 okay so different model has different algorithm for creating those tokens and let's not get totally into how they create tokens or something like that. It's just that they want to optimize it in that in their own way. One of the thing which I got to know is when you talk about a language, we got a lot of words there, right? So we have a huge vocabulary. Um example, if you want to search something, you say search or I'm searching, right? So searching becomes a word. But nowadays people are creating their own words. So for an entire decade, we were not saying searching, we were saying googling, right? So if you want to search something online used to googling but now we say chat jeeping. I don't know if that word exists but we got chat jippeting maybe we are uh geminiing it. Okay that's fancy but anyway you got the point right? So we create new words uh in this sense what it does is example if I add one more term here which is googling which is not a word googling and let's see the token. So you can see it's it create it break into two parts which is go link but if I type searching now that becomes one word because it's vocabularies right so it is known word but for googling it breaks down into two parts and that's how it works so you can also see they are making it commas here then indivisible becomes two part then the emoji right so the one emoji takes four tokens so next time don't don't use emojis and also tell your jpd don't send me emojis because the more uh tokens you use the more money you pay for that because you are getting charged for the tokens. Okay, so you know the steps now. So first it will take the sentence it will break down into tokens and then this tokens gets converted into vectors. Okay, now what next? Okay, there's one more thing. One token is approximately 3/4 of the word which is if you have 100 tokens that means you are representing 75 words. This is on average not exact. Uh and it changes from model to model as I mentioned before. Perfect. But if you want to understand this more, I have something for you. Okay. Now if you talk about vectors as I mentioned before, let's say I'm using a two-dimensional vector here and I'm plotting all these values here. Now if you see for the value dog, it's somewhere it is getting plot here, right? The value is minus0.97 and the y-axis is 0.23. So that's why it is getting represented. Lion is next to it. Puppy is almost nearby but cat and kitty are here. Then planet like earth and planet what itself nearby. Python came here and Java is here. That's different. Maybe it is not able to interpret or not understanding what I'm trying to say. Maybe Python is treating as an a snake. I'm trying to say Python is a language. Now what happens is when you give a text where you are saying my favorite programming language is Python. Now it knows that's a language. So your GPS or your transformers understands what you're trying to say. They know the context. But if you simply say Python, maybe it will not understand. Maybe if you say uh the biggest snake in the world is Python and I'm just giving some weird sentence or the you got the point right so that how it interprets. Now if could I have mentioned in the sentence that's a Python language maybe it could have added here. So what it does is it's connect all the uh common tokens or words together. Uh now one more example I can give you here is Quinn to a king is a woman to a man. Okay. So it's a female and male version. Right? So if you want to find the female version of king, what you can do is you can say king minus man plus woman. That's a formula you can use and you will get queen from the graph if it knows what queen is right. So it uses those type of formulas there. Okay. But if you want to create your own how will you do it? So if you go to web vector embedding again you can see we have some codes here. Again we don't want to write any python code or java code here or javascript code. I can use curl or I can use uh insomnia which is a API client and I can use this API. So I can just copy this and you can use any API client maybe Postman. The only thing is you have to send a post request and paste it here. Click on send and let's see what it says. It says that you didn't provide any API key. That's right. You have to also provide the API key which is somewhere here. So we have to specify content type. We'll do that first. So I will go to headers and I will add what I'm typing content type it should be application JSON then I'm going to add the authorization and that will start from the better and then you have to pass the open open AI key. I have a key with me if you don't have please create one and I'll just paste it here. That's my key. And once you have this, click on send. It should not give you problem of uh key now because you got the key. It says we could not pass the JSON body of your request. Okay. So we are not passing the JSON body. So let's go to the body and let's pass a JSON data. Now what should be JSON data here? It's very simple. You provide the input for which for what data you want those tokens or the embeddings. I will say for dog and click on send and uh okay so it says you you need to also provide the model okay let's provide the model and which model we are going to work with so I will go for text in fact you don't have to remember this since I'm doing this multiple times I know now I know I will use large and click on send let's see and you got it so you can see we got so many tokens and if I simply scroll down since I'm using large look at the dimensions you got so many values all these are dimensions but what if you want such a big value you want only two so I can specify the dimensions here and you can say two so it will convert this into two dimension run this and you got it so you can see we got this these values and you can try to remember this value the same as this okay so and you can find more values and add and see the graph and let me know what extra you got in the comment section okay perfect so that's about embedding that's about tokenization And that's about how do you get those embeddings. I hope it makes sense. Let me know if you have any more questions in the comments. I'll try to make another video.