Topic Modeling with Llama 2

Key Takeaways

a little bit more link with large language models has never been easier with the rise of llama 2 open source llms are starting to become more and more competitive they are nearing the performance of jet GPT in this tutorial we will be using llama2 for topic modeling where we go from thousands of documents to just a few interpretable topics we will be using per topic a highly modular topic modeling framework on top of which we can use llama to to derive topics to natural language so without further Ado let's just get into it what we have here is a Google collab notebook and The Notebook shows all of the code that we will be going through in this tutorial you will find the link of it in the description let's start at the beginning what is birth topic Albert topic is a topic modeling framework that allows us to go from documents to just a few interpretable topics and they are generally represented through keywords the approach that we're typically using in word topic is that we cluster the documents into semantically similar clusters of documents when we have those clusters we assume that each of those cluster contains a single topic of which we extract a number of keywords to try to represent it so if you have a topic about cars you can have the keywords car cars Ford Etc and that by itself is very interesting and extremely helpful to understand what the topic is about but you still have to interpret it yourself right I mean if you have the words car cars Ford you have to think about what the topic means and that thinking about we want to prevent entirely of course but we want to make it easier to interpret those and that's where llama2 comes in so when we have those weighted words generated through CTF IDF we basically give them to llama2 along with some representative documents and ask glamor to okay can you create a very interpretable label for this topic so instead of the word car cars Fort it might be this topic is about electric vehicles for example so there are many ways that we can represent the topic but llama2 makes the interpretation of such a topic much much easier we have this notebook and to run all of this we will need to make sure that you have a GPU typically you can choose a T4 inside of Google colab pro but if you go to CPU route and this will all become very difficult we will start by installing a few packages of course per topic for the topic modeling we will install data sets to make sure that we can load in the data that we're interested in and then a few packages that make sure that we can run llama2 more easily on the GPU that we have because there's large language models well it's in the name they're quite large and we want to make sure they can fit into the virtual Ram of our GPU but as always let's just start at the beginning of the data that we have so I've already loaded in most of the things that we're going to do I've also already trained the model we're skipping over a lot of boring training stuff um but the data that we have are archive papers and more specifically the abstract of those papers and those abstracts are perfect introduction into topic modeling in general because they contain nicely written text they are very short which generally helps with embedding models and they contain all of the information that you're interested about we extract the abstracts and the titles from the data set that we have give you an ID what we have here is the abstract of the well-known paper attention is all you need you can see it's well roughly the size of a paragraph and most embedding models I think are able to capture this very nicely so that should be okay we have quite a few abstracts let me quickly see how many we have over a hundred thousand abstracts and with a hundred thousand documents of course we're not going to go through them all individually topic modeling is an excellent technique where we can look into those abstracts and see what kind of topics we can derive before we can go to Lama 2 and load it in we actually need to accept its license and make sure that we're logged in into hogging phase so it knows that we've accepted it so if you go to hugging face itself and go to your settings and access tokens you can find the token here you can copy it go back to the notebook simply log in my token is valid which is great it means we can use lamba 2 and then we can continue using the model now comes the interesting part loading in llama2 and basically playing around with it for a little bit before we go into the topic modeling so there are a number of models that we can choose from we have here the 13 billion parameter variant I believe there's a 7 billion and 60 billion and at some point maybe a 40 billion a parameter model there's some choices to be made here and I am specifically choosing now to go with the 13 billion model and the reason for that it is that it's a nice balance between on the one hand speed inference of the model and also accuracy so as you might expect a larger model will be more accurate but it also require way more compute power necessary to run the model at a nice base so personally 13 billion is a nice balance between inference and speed accuracy and speed but it all depends on your use case now we're using this model which should be sufficient for the data that we have now that's step one the second step of this loading in llama is optimization and that's really necessary because oh it's a large language model and it really is very large and very difficult to run without doing some extra steps one of those steps is quantization or essentially say okay those 13 billion parameters they are saved generally in a 32-bit preparation but if we compress the 32-bit representation into something smaller and it will also mean that the resulting model will be smaller we'll lose some accuracy but ending on the quantization method that you use it shouldn't be that much what we're using here is a 4-bit quantization where we go from 32-bit to orbit which is quite a big leap in difference but the performance generally should be okay for this type of model we're using bits and bytes for that which is a special configuration special library that we can use to do this kind of efficient loading in of models of the number of parameters here so we loaded in the four bits obviously we normalize um the type of bits is what the paper generally uses and also shows very nice performance there's a small trick that they use even not small it's very interesting where they use a second quantization after the first and by doing it again in a way it further reduces the necessary bits that you will need in order to run your model which again becomes more efficient but they've also shown it doesn't reduce the importance accuracy so to say of those models that much and we say okay we want to have of course A B float 16 during inference which generally use for these kind of models then we load in the actual model we have a tokenizer and the model in itself and the tokenizer is used to go from text to individual tokens so let's say the words telephone might consist of the token Stella and phone or the word configuration of config and duration and these kind of tokens are necessary for the model to have some generalized performance because when it notices a word it hasn't seen before it can still uses those individual tokens to kind of derive what its meaning is supposed to be we have the tokenizer that we load in and then we have the model in itself or we if it's a configuration that we created before so the four bit representation in order to properly load the model and I've already done these steps because they might take a while you need to load in a model things like that but it loads fine in a T4 instance this is of course where it becomes more interesting because we're slowly approaching the prompt engineering but before we do that we use a Transformers pipeline that's a hugging phase pipeline that makes it much easier to run all of these prompts and it's also the pipeline that we're going to be using in bird topic later on setting the number of tokens it can generate we give it a penalty for how much it's uh repeats certain phrases we don't want to repeat stuff and we set the temperature low and the temperature basically shows you how let's say creative the model is by lowering the temperature it's not that creative which we actually need for this specific instance because you know we're creating topics and the more creative it comes in representing the topics the less accurate it will actually be so we want to lower the temperature as much as possible now for the fun part from the engineering so let's see how all of this worked I already created this prompt but let's run it again could you explain to me how 4-bit quantization works as is I as if I am Phi which is the e-live 5 representations or the Eli 5 question that you often see in tutorials explain it like I'm 5. and as you can see it actually takes a while to generate the response and that's what happens with these kind of models they are quite large you need very big gpus to make sure they run in seconds or even in milliseconds which is also why you will see in open AI is that you don't immediately see the output but tokens generated one after the other so it has given us the output a generated text and it tries to explain four bit quantization with a big box of crayons and I'm not going through all of this right now but it's very interesting to read how it tries to explain it using just colors I'm basically saying okay we can represent something by using two colors instead of itself so you can represent orange with red and yellow so you don't need orange thereby reducing the amount of crayons that you need super fun to read llama 2 has a specific template that you need to follow for the specific prompt engineering looks like follows what you have here is a system prompt that's what you start with it's how you guide the model into making sure to understand what it's well it's uses or we should model or what kind of assistant it should be it's describing its personality in a way then you have your user prompt where we essentially ask it question and then you have an answer and it has certain tags that it was trained on to make sure it differentiates between okay this is a system prompt this is user prompt this is an answer etc etc we need to make sure that we follow this template for Pronto engineering as you can see we didn't do it here that's because we just asked it the question and didn't do any advanced engineering here now that we have this template let's go a little bit deeper into how we're going to use it for topic modeling we have here the system prompt so like I mentioned that's basically describing who llama2 is and in this case lamma 2 is a helpful respectful and honest assistant for labeling topics kept it simple on purpose just for this tutorial but we can also extend it to make sure it gives short answers to make sure it doesn't say anything weird or you know how creative it can be how repetitive it can be can everything you want into system prompt it will take up some of the token spaces token limits but for now just as one sentence should be should suffice into creating the topics that we're interested in then we're going to the user prompt and normally we would just ask the question like we did before right but this is a more advanced example and when you're approaching Advanced examples um what you want to give it is an output of okay this is how what I want to see as a result this is how you should work so essentially what we want to do is we want to create an example prompt a prompt that basically shows comma 2 if you do this like exactly like it is shown here then you're doing a good job this is called few shot learning where you give it to the example of what it should output in this case I have said okay I have a topic that contains a number of documents and these are about meets and their impacts on the environment then I say okay this topic is described by the following keyword so we have meat beef missions processed stuff like that then based on the information on the topic above create a short label and the label I think should be environmental impacts of eating meat this is the example that we're gonna give llama to so it will become much more accurate when it wants to create or when it creates label step interested in that we're working for this is a very awesome technique for now taking a large language model to the next step and then we have our main prompt which is exactly what we did above but this time without specific documents and without specific keywords because what we can do here is we can use this tag to put any documents that we want in there for specific topics so instead of doing the example that we did before we create a sort of a template that we use each time we ask it for a label for a specific prompt oh label for a specific topic of course so we have the documents that it should get it should a number of keywords that we give it after topic modeling CTF IDF and then again based on the information about the topic above please create a short table of this topic and then we will output as a label what we're interested in then we combine that all into prompt template so to say so where where we have a system prompt who is Amato we have an example prompt this is what the output should look like and then we have the main prompt which is essentially our question please create a short label of the topic what we're doing here is we sharing our embeddings her topic does that uh under the hood but we want to do it a little bit more explicit we want to create the embeddings for our abstracts because we can then more easily use them later on instead of having to recalculate them every single time and I'm using here a model that's well at least that's new with the release of this video and it's really high on top of the let's say the embedding leaderboard using sentence Transformers which is an amazing framework for loading in a bunch of embedding models this model really works very well and it was also just fun using a new state-of-the-art model 8-ish yard after we've created embedding for each of those models we will Define a few sub models and those are sub models will be used within bird topic and well they will be used automatically of course but here we Define a few things that we're a little bit more interested in so umap has a tendency to consistently generate new answers if you run it over and over and over uh that's not necessarily a problem but if you want to reproduce your results we need to set a random state we also Define the clustering model that we're going to be using hdb scan we're increasing minimum cluster size by increasing it we will get fewer topics or clusters that are generated that's something we really want to do because hdb scan has a tendency to generate many micro clusters which can be very interesting for the purposes tutorial let's just keep just just a few hundred of them after that we pre-reduced the embeddings for visualization purposes what does this mean s that we go from embeddings which are I think 700 something in size just two dimension those two dimensions are important because I can try as much as I want but I cannot visualize 700 dimensions and visualize sudo and with two it gives us a proxy or what the embedding structure looks like and that makes it helpful for us to visualize then resulting topics their labels the documents that we can find in them etc etc that's what we did here are we going to something a little bit more interesting the representation models but we're using llama2 to represent topics that we have we're also using efidf Create keywords also representation here we're doing a few more representations on top of that specifically keyboards which is a keyword inspired representation a maximum marginal relevance which produces the or increases the diversity of the resulting keywords because if we have a topic that has the words car and cars you know that's that's redundant right either remove car or cars that's what maximum relevance does us what we get we get a bunch of representations and I'm doing this here to show you the difference between llama2 keyboard MMR CTF IDF so the keyword based representation and we can go through them in depth but there's a lot of documentation in the in the topic official documentation page he show you what representations would look like now we go into training I've already done this it takes after embedding the models 15 to 20 minutes and we pass bird topic a number of things so the embedding model that we used to go from abstracts to embeddings we use the umap model to do dimensionality reduction of the embeddings the hdb scan model to Cluster the reduced embeddings and then our set of representation models to generate all those different outputs the great thing about that is that it doesn't just generate one output it generates multiple tremendously helpful when comparing topics when seeing what a label should be etc etc because a topic is more than just one perspective of keywords can be a label it can be a summary it can be a poem for all I care it really doesn't matter it's up to you what you think a topic should represent of what the thing is that should be should represent the topic after training we now have our topic model and this is where we can a little bit more of interactive coding when we run topicmodel.getinfo we will see a data frame and in this data frame you will have a number of representations of specific topics the keyboard the llama maximum marginal relevance representative talks there's a lot of things happening here so instead of you know a lot of text a lot of things let's just go into a few specific topics and I've chosen one here what we do is when we run get topic on the topic model for a very specific topic so zero here it will shows us the representations that we can extract let's see what happens if we take review take a look at the main topic but this is generated with CTF IDF and topic zero then has the keywords policy reinforcement RL agent learning Etc and that's clearly about reinforcement right but it also has words like to and in which doesn't really tell you much about about the topic let's see how MMR has done it doesn't change the representation that much because it reduces the or increases the diversity and to and in he'll die first in a way they're just not that accurate stats oh let's see what keyboard does and keyboard does a much more interesting representation already enforcement learning robots Dynamic algorithms Model Behavior these words are much more interesting compared to the words we had before because they were upwards in there this already describes much better but there's still keywords right there's still keywords that we have to interpret can be difficult because now I have to manually assign hundreds of topics that we created really does not make sense oh let's see what amatu has created for us and it has created a very interesting label instead of those keyword the label here is deep reinforcement learning challenges now we already saw that it wasn't about reinforcement learning but now it has created a label for us to interpret look at few which also makes communication much much easier because if you're going to communicate keywords to stakeholders they are going to ask but each and every single keyword means what those values will exactly mean and those are important questions but for domain expert maybe not all that relevant but to have a llama2 representation instead that you can show next communication much much easier we can continue exploring topics as we go so topic 2 audio visuals speech separation and recognition those are the kind of labels that you would expect when you do topic modeling right keyword based is important but having this additional perspective if you as I mentioned A New Perspective I've receive preserving Federated learning something I'm working on quite frequently nowadays but decentralized distributed differential it's indeed about privacy preserving Federated learning and using all of this we can explore these topics and we could go in depth in what all of this means next we run this piece of code and what this is doing it essentially assigns the lamba 2 labels have created to the topics and this is important because as a default the labels will be created based on the keywords and now what we want of course is the llama2 labels in action so the last step in this pipeline is to visualize the documents what we can do is we can run visualized documents with the reduced embeddings that we have we can use the lamba 2 labels to visualize them who highlight them and to even see which documents can be found in these topics here we have one about multi-agent reinforcement learning which obviously is about deep reinforcement learning challenges you can continue doing this and exploring the interesting topics do adversarial robustness and deep learning audio visual speech separation and recognition if we go over this we see in the text to speech we see a lot of speech like topics or at least documents that appear here this allows us to go in that with topic modeling using lamba 2 using these kind of techniques allows us ectopic modeling much more easier much more intuitive with these kind of labels still requires relatively large deep views but in the future this will become easier and easier as we go forward this was bertopic together with llama2 and illumination that I think works relatively well if you enjoyed it if you like this video please leave a like or give some feedback make sure I do the best that I can but this is a first for me so any feedback definitely appreciate it if you dislike the video just dislike it also communicate for me but I'm doing well and what I'm doing correctly hopefully the videos will get much better in the future thank you for watching and to the next tutorial