Steering LLM Behavior Without Fine-Tuning

Key Takeaways

Imagine you're working with a large language model and you would like to tweak its behavior or its personality. A well-known solution is to use prompt engineering. You specify in the system prompt what you want to achieve. Another option is to fine-tune the model, but for that you need enough data demonstrating the behavior or the personality you are looking for. And of course, you need to have enough compute to perform this finetuning. So today we're going to talk about a third option, steering the model. And it turns out that steering a large language model is loosely analogous to what neuroscientists call neuro stimulation of the brain. So neuro stimulation, it's the idea of artificially stimulating certain areas of the brain or specific neurons using electrodes or magnetic fields. When you stimulate biological neurons this way, neuroscientists have observed that it might trigger or inhibit certain motor actions and even elicit certain emotions, feelings or memories. In neuroscience, neuros stimulation is used for research to better understand the role of the various brain regions, but also for clinical purposes, for instance, in treating Parkinson's disease. And what's interesting is that it's a technique that obviously does not modify the brain. It just intervenes on the fly. And it turns out that you can do pretty much the same with artificial neural networks in general and LLMs in particular. By targeting carefully selected neurons in your LLM, you can control or elicit certain behavior without having to rewire anything, without changing the weights of the model. This procedure is fairly easy to use and to illustrate it, I applied it to a Lama 3.18B model to change its personality and make it obsessed with the FL tower to the point that it sometimes even believe it is the FL tower. Look at that. And again, this change is entirely controlled at inference time when generating the tokens. What is loaded in memory is still the original LMA model. there is no fine-tuning involved. So you want to learn how to do the same. Well, today I'm going to explain the basics of this method and show you how you can easily use this technique to steer pretty much any open source LLM using aface transformers library. So first of all, let's recall the internal workings of a typical LLM. Most of them today are auto reggressive models that generate one token at a time. For that they are based on the transformer architecture and organized as a stack of layers. At each layer each token goes through an attention block and a feed forward block. So as you know the attention block is where each token can receive information from the other tokens preceding it in the sequence and the feed forward network block is a traditional multi-layer perceptron. After those two blocks, the result is then passed to the next layer. The stack of layers essentially represents successive stages of processing until the logits for the next token are computed by the final linear head. If I zoom in at the boundary between two layers, what gets passed here is actually a vector sometimes called the hidden state. This vector actually lives in a highdimensional space typically a few thousand dimension that we'll call the activation space. We can think of this huge vector as representing the model's internal state, its hidden thought at this point in processing the token. With LLMs, we don't generally visualize those numbers as coming from neurons. But you could very well imagine the output of each layer as a series of neurons that produce the coordinates of the vector that gets passed to the next layer. And these are the kinds of neuron that we can target with our steering, our artificial neuro stimulation in order to modify the thoughts of the LLM at inference time. But now the question is how do we do this? How can we stimulate the neurons in a way that elicits a certain behavior or a certain personality? So to answer that question, we need to understand our LLMs represent abstract concepts. You may remember that the very first layer of an LLM is the embedding layer. This layer will map every possible token of the vocabulary into a vector of the activation space. This token vector correspondence is by design for the embedding layer. But something remarkable happens as the model processes information through deeper layers. It continues to represent concepts as vectors in the activation space. This is called the linear representation phenomenon. An empirical observation that seems to hold for most LLMs. They tend to represent interpretable concepts as vectors in the activation space going from one layer to another. What's useful here with linear representation is that you can always add vectors. If you have a vector that represents the concept of say a car and another that represent the concept of the color red, if you sum them, you get the concept of a red car. And you can even vary the amount you add so that you can navigate between different degrees of the concept going from a car that happens to be red to something like maybe an intensely red sports car. Maybe you remember from a few years ago the results from the famous word tove vake paper. This paper showed that embeddings of words were following certain kinds of arithmetic relationships. And you could for instance obtain the vector embedding of the word king from that of the word queen by adding the vector for man and subtracting the one for woman. Word tove demonstrated this for word embeddings specifically but with LMS this ID holds throughout the models layer and it is a consequence of the linear representation they develop during training. An implication of this linear representation phenomenon is that for given concept what matters most is the vector's direction not its length. If you have a vector for the concept of a car, doubling its length won't give you a concept for a bus or two cars or a traffic jam. In general, increasing the length of a concept vector does not change which concept it represents, only how strongly it's expressed. Something important to note is that this linear representation phenomenon might be realized differently at every layer of the stack. So after embedding the token car is represented by a certain vector but after each layer in each intermediate activation space there is possibly a different vector for the concept of a car. I told you earlier that between each layer the LLM transmits a vector in the highdimensional activation space and that we could see each coordinate of that vector as a neuron outputting a signal to the next layer. It might be tempting to think that every such neuron represent a certain concept but this hypothesis turns out to be wrong in general. LLM actually encode concepts as distributed patterns across neurons. This is called superposition and through this they can manipulate far more concepts than there are dimensions. So I won't go into too much detail. If you're interested, you should check out Entropic's series of paper about superposition and monoseity. Another important observation regarding the encoding of concepts in activation space is that different layers might play different roles. What researchers observed is that in early layer those vectors tend to be activated when the concept has just been explicitly seen in the input tokens. For instance, the model has read the word car. In late layers close to the output, the vector corresponding to a concept tends to activate when the model is about to output the token. And as we'll see later, the most interesting cases for us are intermediary layers. This is where LLMs tend to represent abstract concepts in order to reason on them. So to recap, concepts are represented by vectors in the activation space between each successive layers. And the good thing with vectors is that we can add them. So it means that if we take the activation coming from a layer, we can add to it a given vector in order to reinforce that concept in the thought of the LLM. And this is what is called steering. So now let's see how we can do this in practice. So for now, let's assume we found a good vector that represents the concept we want to stimulate. And I'll come back later to the different ways to actually identify those vectors. So as I explained earlier, when you want to steer the behavior of an LLM with a concept vector, you don't change the LLM. The model is the same. The weights are the same, but you will intervene on the activations at inference during the generation of new tokens. So more specifically, if you have a vector X representing the activations at the output of layer N and you want to steer it in the direction of the vector V, you will simply add V2X. Of course, as I mentioned before, when you add two vectors, you can scale each one with a coefficient controlling how much of each you add. So usually what we do is work with normalized concept vectors V but we multiply them by a coefficient before steering and this coefficient will govern the size of your intervention. Okay, but how do we do this on the computer? Well, actually it's just a few lines of code using face transformers. So here I have a small snippet that loads lama 3.18b from the face hub and calls the model on a simple prompt like give me some ideas for starting a business and you see the response. Now let's say we want to steer the model to change its behavior and its personality and maybe some of you have seen a few months ago that entropic had created a model that pretended to be the golden gate bridge. So, I wanted to reproduce this, but as you've probably heard, I'm French. I live in Paris. So, I had to try this with the EFL Tower instead. So, here I'm loading a vector V that represents the concept of the FL tower at layer 15 of the model. Lama 3.18B has 32 layers. So, we are in the middle. I'll explain later how I found this vector, but for now, let's assume we have it and we want to steer our model with it. To perform this while generating our tokens, we need the equivalent of the electrode that delivers electrical stimulations to the brain. So in our case, the solution is called a hook. A hook is simply a function that you attach to the model and that gets triggered during the forward pass right when inference is happening. So let's choose a coefficient and my hook will simply take the output of a layer and add the vector scaled by the coefficient. Very simple. And I will register this hook at layer 15 so that it will be systematically called after the model has processed layer 15. Now let's run my model again with this hook and see what happens with the coefficient set to four. As you can see, the model starts deviating from its natural behavior. When I was asking for ideas for starting a business, the base model was suggesting things around e-commerce and services. And now you see the answer is different. It's talking about food, bakeries. It is not explicitly about the EFL Tower, but you feel a change of perspective. Now I can remove my hook and replace it with a stronger one. With a coefficient of eight, llama starts to suggest ideas about wine and travels. It is clearly influenced by the concept we are simulating. And now if I ask who are you? The model will start pretending to be a large metal structure called the FL tower. And here's a fun detail. The original response with no steering started with I'm a large language model. Now it says I'm a large metal structure. You can literally see the steering kick in right after the word large. So of course doing this you will quickly realize that you don't want to push the coefficient too high. That's expected if you add too much of the vector you completely derail the model's reasoning and it will output gibberish. That makes sense. You could imagine it would be the same with electrical stimulation of the brain. So you want to choose a good value for the steering coefficient. So luckily there are some systematic techniques to help you identify the sweet spot and there are also some ways to improve the stability of the model by tuning certain generation parameters like temperature or frequency penalty controls. I cover these techniques in a blog post and I'll leave the link in the description. So at this point I'm sure you're convinced that steering could actually be an interesting technique to study and use. But I left open a big question. How to identify a steering vector for your concept of choice. How did I do for the EFL Tower? Well, there are actually several techniques. One is called contrastive activation. The idea is fairly simple. You have to gather pairs of prompts, positive and negative example of the behavior you want to elicit. Then you compute the average activation across positive examples and subtract the average activation across negative examples. If you have enough pairs, you will end up with vectors that represent the concept you are looking for. This method has been found to be pretty effective in some cases, even better than prompt engineering and supervised finetuning. Another completely different technique uses sparse autoenccoders. These are autoenccoder models trained to reconstruct the LLM activations through a sparse intermediate layer. The key insight is that each dimension of this layer tends to correspond to an interpretable concept. The method is unsupervised. You don't tell it which concepts to find. I'm skipping the details, but what's nice about this method is that it gives you a large library of vectors to choose from. And a lot of people have been sharing their sparoto encoders on the face hub. So you should have a look for your model of choice. The drawback is that these vectors don't generally come with predefined concept labels. So if you are looking for one specific concept, it might be particularly tedious to use. Fortunately, the great website Neuronipedia created by Decode Research is the perfect place for that. You can browse through visualization that will help you identify the proper features that suit your purpose. In my case, I searched for the FL Tower features in the Lama 3.18B model and I found for instance this one that I used for the demo. One important aspect of steering vectors whether they come from contrastive pronto encoders or other techniques is that they are always located at a given layer of the model. So in general you might have the choice between steering early layers, late layers or middle layers. And as we discussed earlier, in general, if you want the model to be influenced by a concept without necessarily reproducing the exact same word, it is better to steer the concept vectors that are located in the middle intermediary layer where the abstract reasoning is supposed to happen. But you might have to experiment yourself. Okay, that's pretty much it for today. I hope I convince you that steering LLM can be an interesting method to elicit certain behaviors. It does not require any fine-tuning. It just works at inference time. And it has many benefits like being able to control the intensity of the intervention and maintain it over the whole text generation which is sometimes hard to achieve with prompt engineering. Of course, it also has drawbacks. As I mentioned earlier, it might not be easy to find a sweet spot where the model is properly steered but still maintains its fluency. Also, steering works best for concepts the model has already learned to represent. It won't teach the model new knowledge. If you want to know more about the technical details, I encourage you to go read the blog post where I explain how I constructed the FL to Tower Lama model and what kind of methods you might use if you want to do a similar thing. It contains a lot of useful tips to investigate the proper way to do steering in particular with spare autoenccoders. Don't forget to visit the Neuronedia and the Ugging Face Hub for finding steering vectors and maybe sharing with the community your own recipes. Let us know in the comments what you were able to achieve with this technique and have fun steering.