Encoder-Only Transformers (like BERT) for RAG, Clearly Explained!!!

Key Takeaways

in code only Transformers can cluster things and classify things that's cool stack Quest hello I'm Josh starmer and welcome to stack Quest today we're going to talk about incoder only Transformers and they're going to be clearly explained this stack Quest is brought to you by the letters a b and c a always b b c curious always B curious note this stack Quest assumes that you are already familiar with the main ideas of how neural networks work if not check out the quest also note this stack Quest focuses on the main ideas of how encoder only Transformers work and what we can do with them that said if you want to see the nitty-gritty details check out these fine quests okay way back in 2017 the first Transformer ever made had one part called an encoder and a second part called a decoder and the encoder and the decoder were connected to each other so they could work together this first Transformer was based on something called a seek to seek or an encoder decoder model encoder decoder models were designed to translate text in one language like English into another language like Spanish for for example Squatch might say pizza is great and the encoder would crunch a lot of numbers to encode the input and the decoder would crunch a lot of numbers including the output from the encoder and ultimately decode those numbers into a translation La piz s magnifico bam however it wasn't long after this first encoder decoder transform was published that people realized that both parts could work just fine on their own it turned out that we could generate text including translations of text with just a decoder and these models which form the basis for chat GPT were named decoder only Transformers likewise models based entirely on the encoder started to be very useful on their own and these models which form the basis for Bert and many other models were named encoder only Transformers however over the years the models based on decoder only Transformers kind of stole the show pretty much everyone has used or at least heard of Chad GPT or gemini or any of the other countless decoder only Transformers out there and I've heard some super smart people who keep up with AI and everything ask me if people still use encoder only Transformers so in this stat Quest I want to dive deep into encoder only Transformers so we can understand their significant but understated power and that means starting with a deep dive into the first thing a Transformer does which is create word embeddings word embedding converts words bits of words and symbols collectively called tokens into numbers we need to convert tokens into to numbers because Transformers are a type of neural network and neural networks only operate on numbers one super easy way to convert words into numbers is to just assign each word to a random number for example if Squatch just ate a delicious pizza they might say pizza is great and we could assign a random number to each word now if Norm came along and said p is awesome then we could reuse the random numbers that we already assigned to pizza and is and assign a new random number to awesome in theory this is fine but it means that even though great and awesome mean similar things and are used in similar ways they have very different numbers associated with them and that means the neural network will probably need a lot more more complexity in training because learning how to correctly process the word great won't help the neural network correctly use the word awesome so it would be nice if similar words that are used in similar ways could be given similar numbers so that learning how to use one word will help learn how to use the other at the same time and because the same words can be used in different contexts or made plural or used in some other way it might be nice to assign each word more than one number so that the neural network can more easily adjust to different contexts for example the word great can be used in a positive way like pizza is great and it can also be used in a sarcastic negative way like my cell phone's broken great and it would be nice if we had one number that could keep track of the positive way that grade is used and a different number to keep track of the negative ways hey Josh deciding what words are similar and are used in similar context sounds like a lot of work and using more than one number per word to account for different contexts sounds like even more work don't worry Squatch the good news is that we can get a relatively simple neural network to do all of the work for us for for example let's imagine we have two phrases Pizza is great and pizza is awesome the first thing we do is create inputs to a relatively simple neural network for each unique word then we create an output for each word then we connect all of the inputs to at least one activation function and in this example we'll connect the inputs to two activation functions the number of activation functions determines how many numbers we will use to represent each word in this case since we have two activation functions we'll end up with two numbers or word embeddings representing each word then we add weights to the connections from the inputs to the activation functions these weights are initialized with random numbers so right now they're not very useful but the plan is to track them and thus change them using this data lastly we connect the activation functions to the outputs with some boring details that we don't need to worry about right now because we have one word embedding for each word going to the activation function on the top and one word embedding for each word going to the activation function on the bottom we can plot each word on a graph that has the top word embeddings on the X x axis and the bottom word embeddings on the Y AIS for example the word Pizza goes here because its top W embedding is 0.11 and its bottom word embedding is 0.10 likewise the word is goes here great goes here and awesome goes here now with this graph we see that the words great and awesome are currently no more similar to each other than they are to any of the other words however because both words appear in the same context in the training data we hope that training the network will make their word embeddings more similar the idea is that we want each word in the training data to predict the next word for example we want the first word in each sentence Pizza to predict the word that comes after it is and we want the word is to predict the words that come after it great and awesome so in order to see which word the network predicts should come after Pizza we put a one in the input for pizza and we put zeros in all of the other inputs then we do the math with the randomly initialized parameters and we end up predicting in great because it has the largest output value 0.45 thus with the randomly initialized parameters the network does not correctly predict is the word that comes after Pizza however after we train the model we end up with these new word embeddings and pizza correctly predicts is and is correctly predicts great and awesome now when we graph the words with the new word embeddings great and awesome cluster together this result is great and awesome because great and awesome are similar words used in similar contexts and they ended up with similar word embeddings bam so far we've seen the simplest way to create word embeddings we used a simple net work that we train to predict the word that comes after the input however just predicting the next word doesn't give us a lot of context to determine the optimal word embeddings in contrast if we had a more complicated training data set then we would have more inputs and outputs for our neural network and we could connect everything like we did before but now because we have more inputs and outputs and longer sentences in the training data we can add more context to the training process for example we can use the pizza came out to predict the next word of in other words instead of just using one word to predict the next we can use the preceding four words to predict the next increasing the context can help create better words embedding values but I want to point out that the way we are doing things right now ignores word order and because we are not currently keeping track of word order any jumble is just as good as anything else in other words the pizza came out of would give us the same inputs and output as the jumbled up phrase Pizza out came the of so it would be nice if there were some way to create word embedding values that also took word order into account and that leads us to the part of a transformer that comes after the initial word embedding layer called positional encoding positional encoding helps keep track of word order for example if Norm said Squatch eats pizza then Squatch might say yum in in contrast if Norm said Pizza eats Squatch then Squatch might say yikes so these two phrases Squatch eats pizza and pizza eats Squatch use the exact same words but have very different meanings so keeping track of word order is super important there are a bunch of ways to implement positional encoding but these details are out of the scope of this Quest anyway now we know that positional encoding helps keep track of word order however it would be great if there was also a way to keep track of the relationships among words for example if the input sentence was this the pizza came out of the oven and it tasted good then this word it could refer to pizza or or potentially it could refer to the word oven Josh I've heard of good tasting pizza but never a good tasting oven I know Squatch that's why it's important that the Transformer correctly Associates the word it with pizza and that leads us to the part of a transformer that comes after positional encoding called attention and specifically in an encoder only Transformer it's called self attention attention helps keep track of the relationships among words going back to our example about pizza coming out of an oven attention can help correctly associate the word it with the word Pizza note there are different types of attention but incoder only Transformers only use self attention so we'll focus on that in general terms self attention works by seeing how similar each word is to all of the other words in the sentence including itself for example self- attention calculates the similarity between the first word the and all of the other words in the sentence including itself and self attention calculates these similarities for every word in the sentence once the similarities are calculated they are used to determine how the Transformer encodes each word for example if you looked at a lot of sentences about pizza and the word it was more commonly associated with pizza than oven then the similarity score for pizza will cause it to have a larger impact on how the word it is encoded by the Transformer bam and now that we understand the ideas behind self attention we understand the ideas behind the three fundamental building blocks that make up an encoder only Transformer word embedding converts the input into numbers positional encoding helps keep track of word order and self attention helps establish relationships among words combining all three layers creates a new kind of embedding for each token that takes position and relationships among words into account and this new type of embedding is sometimes called context aware embedding or contextualized embedding because context aware embeddings include information about the position of each word as well as the relationships among the words context towar embeddings can help cluster similar sentences or even similar documents bam to summarize encoder only Transformers like Bert that only use self attention can create context aware embeddings and just like plain old word embeddings can help cluster similar words that are used in similar ways context aware embeddings can help cluster similar sentences or similar documents note the ability to Cluster similar sentences and documents is the foundation for something called retrieval augmented generation or r rag rag works by breaking a document into blocks of text and then using an encoder only Transformer to create context aware embeddings for each one then when someone gives an AI a prompt like what is pizza rag generates embeddings for what is pizza and finds the chunks of text that are the most similar double bam now that we understand the main ideas behind encoder only Transformers and context aware embeddings let's talk about another cool thing we can do with them one thing that we can do is use the context aware embeddings as inputs to a normal neural network that classifies the sentiment of the input for example we might want to see if people are posting positive or negative sentiments about pizza on LinkedIn and the context aware embedded are great inputs for a neural network that can do that type of classification alternatively we could use the context aware embeddings as variables in a logistic regression model that does classification anyway even though decoder only Transformers like chat GPT get all the hype the context aware embeddings that encoder only Transformers create can be used in a wide variety of settings to do very cool cool things triple bam now it's time for some Shameless self-promotion if you want to review statistics and machine learning offline check out the stack Quest PDF study guides in my books the stat Quest Illustrated guide to machine learning and the stat Quest Illustrated guide to neural networks and AI at stat quest.org there's something for everyone hooray we've made it to the end of another exciting stack Quest if you like this stack Quest and want to see more please subscribe and if you want to support stack Quest consider contributing to my patreon campaign becoming a channel member buying one or two of my original songs or a t-shirt or a hoodie or just donate the links are in the description below all right until next time Quest on