Information Theory for Language Models: Jack Morris

Key Takeaways

Hello, this is L in space just switch today with uh our special guest Jack Morris I guess from Colombia. That's your affiliation right now? Cornell. It's actually confusing because I go I'm in a the New York City outpost of of Cornell. So you have the city, right? But it's Cornell Tech, which is like a small uh Cornell campus in New York. I just you're you're a student of Sasha Rush who teaches at Cornell, so I I should have made that connection. Okay. Yeah, I'm sorry. Um well, that's that's a horrible mistake to make right off the bat, but uh you're one of look, you're one of the there are not that many PhD students that make an impact with their research. The last time someone like this happens was Shinyu from Princeton and uh he joined the OpenAI operator team quite shortly after he graduated. So, like you're one of those like high-profile PhD students at least that's like coming out of the program and like I figured like it was a good time to just like talk about your work and also the fact that you're looking for like which lab you're you're going to join. That's like a whole interesting meta discussion especially with like the insane market for AI talent these days. What's it like to be AI grad student these days? Yeah, and thanks for having me. I guess maybe we can go back to when things first started or like like put yourself in my shoes in 2017 2018 I really learned a lot about machine learning and at my I went to a state university. It's a good school but they didn't have like a deep learning research department or anything. They had people doing it but it was just not as big at that time. But I was getting really interested in those topics especially as applied to language. And then in 2019, I kind of was starting to do research and I think thinking about my career. I mean, at that point, I was 20, 21. I was thinking about like where do I want to be career-wise or like who's doing the coolest stuff right now? Like looking at like what kind of stuff is coming out of that time. I mean, I think Alph Go, I thought Alph Go was really good. At that time, I was playing a lot with like BERT and BERT based models. So like, you know, Google, Deep Mind, they're doing great work. GPT2, GPT1 from OpenAI were like interesting, but I think most people were into bird at that time. I still have a soft spot for like that parameter class of like 100 million to 1 billion scale models. But this is all to say I think at that time I felt like the people doing a lot of the most impactful work were like professors and PhD students. Like just a ton of like interesting ideas being explored and cool opportunities in academia. So I ended up applying to grad school. Well, I first I did this Google AI residency program which was mostly during the pandemic like 2020 and then 2021 and then I was also applying to grad school. Started grad school in 2021 that's still what was going on at that time like around when I guess GPT3 175 billion had been released but not instruct GPT. So like we had pre-training and sort of the science of pre-training was emerging but that's where the models were and I still think like I'm glad that I went to grad school and like I had a great experience but the last 5 years have changed a lot like the whole meta has shifted you know like uh the kind of power dynamics are completely different the ideas are coming from different places most stuff was open now most stuff is not open um the types of questions people are asking are different and so yeah I mean for better or for worse I did go to do the full grad school thing and and here I am it's been really interesting perspective watching the science kind of emerge with the products like the biggest thing that happened by far was like chat GPT coming out and which was right in the middle like what 2022 before Christmas like November I remember that year like all like my grandma was asking me about it and that's when it hit me like, oh, this is actually becoming like a real area that people will know about and understand. Like I was trying to explain it to my parents and that's when I think things really started to change in terms of the types of questions you wanted to ask can't always be answered with academic resources. So a lot of the like fundamental kind of like boundary pushing and AI science moved into companies. That was the year when like uh you know just around Europe's as well everyone in in NL NLP and deep learning were were were like very confused at like I think some people were like kind of expecting this already uh in a sense that they had they were obviously more clued into large language models but I think that the sheer amount of consumer level interest that had that that was around at the time in 2022 that completely changed the world like now now we're just like in a different sphere. uh did you have to pivot your research or were you already you just went from bird to like other stuff? Uh you you've done a lot of embeddings work. I mean you're always heads down working on a problem. So I don't think most people in academia are the type to say oh look at this new product that came out. I'm going to abandon everything I'm doing. That can be the right move, you know. Oh, it definitely can. Honestly, if if if I were to give advice to a younger grad student, I think the way to do it would be literally just like sit and wait until the next kind of paradigm shift and then just immediately start working as fast as you can to like reimplement it. Like I I don't think that's like maybe the best way to do science, but it's probably the best way to play the sort of academic game in the in the days of AI. Like you've seen that so many times most recently probably with the reasoning models like 01 came out of open AI September 2024 last year and then there's just been this explosion of like you build like abstraction ladders on top of that like first it was reimplementation like how do we even do this and now it's like a lot about the data what's the right data what are the right evals what are the right training schemes like there are so many different axes you can test and publish research in And like I think the easiest way to do that probably is just work in a field like that that like has it only existed for less than one year and so no one has any like big advantage. I guess that is mostly correct. I think anyone who jumped on reasoning in RL for Labs is doing super well. I just saw this morning that one of the recent Stanford grad students who worked on RL, they just started their company and they're worth 500 million. It's like absolutely bonkers right now. Like just like no product, just three dudes, you know, sitting in some basement somewhere. I mean, undoubtedly cracked, but like also not worth 500. Yeah. But maybe it's not paying for the product, right? It's like the the ideas behind it or the Yeah. Yeah. There was this big shift from in scale of working with 100 million parameter models. Really what happened is like I think the companies invested a ton more into training and infra and like we all kind of had to catch up like you know me I go to Cornell work with a professor there he has to buy GPUs like should he buy last year's GPUs or this year's GPUs how many should he get that we were kind of like trying to figure that out and there was there was like a big lag I think where basically the the seven and 8 billion parameter scale like there's a huge difference between the birds size models which are 125 million parameters to to 200 and then like the 8 billion parameters. I mean obviously it's two orders of magnitude but just like this idea of emergence like if you're talking to a model that's 100 million parameters no matter how well it's trained it knows nothing like if you ask it like what's the capital of a state or like if you ask it who's who was president of the United States in 1990 or whatever it'll just always say George Washington because it just associates the words like president United States with George Washington and then when you get to the 8 billion parameter scale suddenly it knows every single president. It knows every single capital of every single country. And I really do think that changes the type of research you can do. And so like it took us a while I think in academia to catch up like getting good 7 billion parameter models and then running them and getting GPUs to run them. Now I think things have stabilized a lot like we have access to compute and we can kind of like fine-tune and inference that scale of models and that's like kind of fine but there was like kind of two years where everyone in academia was working on like smaller models and none of it really mattered. I can sort of branch that discussion in two ways and we should we should sort of go to your research at some point but I'm enjoying this because I think like we don't get to talk about this on the podcast too often. One is there there's there is an often there's an often bit of advice from the industry people to to grad students which is give up don't work on models just do benchmarks right like a really good benchmarks will will get our attention and then we'll hire you and then you can switch to models later you have for better or worse avoided that which is cool and we can talk about that as well but the other thing I think is that around about 7 8b maybe 4B is when you start switching from like a single GPU setup to like a distributed setup and And I'm wondering like do grad students get HPC training? How much do they teach you of like just how to work with like large clusters of stuff? Oh, to be clear, they don't teach you anything like anything. Like if you see a paper coming out from even, you know, Stanford, they're probably the best school in AI if you had to choose. And it's not like they're learning how to do like multi-node distributed FSTP training like with whatever deep speed. You have to learn that from the internet and from other people and like there's no classes that really do that. I mean it's that's hard to facilitate like as one person. I would say most grad students are doing stuff on single GPU. Some people are doing multi-GPU training. There's probably basically no grad students doing multi-node training. I mean there's probably a few especially if they have like company affiliations but that's really unusual I think. Okay. For grad students who are looking to get up to speed on that, I would recommend the GPU mode Discord where basically the PyTorch team is hanging out in there just waiting to help you. And then the other one would be the fast AI team. If you have some kind of thing, Jeremy Howard will basically help you out and uh they they have some uh distributed training. Honestly, try to reach out to the deep speed team at Microsoft. Like actually they're reasonably accessible. Nobody talks to them. Like it's so so funny. I I like met them at Europs and like they had nobody at their like they was presenting these speech three I was the only one asking questions like um yeah yeah that's good that's good advice listen to this guy yeah I mean just basically like the people are there if you want to ask this is very very valuable experience once you're like a GPU god like you're basically you know in a like a different tier as a researcher because you don't rely on someone else helping you out like you can just sort of be your own research engineer you know yeah I'll comment on that quickly because if someone has been listening to this and also following me online for a while, I think I've made a couple comments like saying something like you shouldn't learn about CUDA or things to that nature. And I'll I'll give some more color to that. So, it's definitely a great idea to learn CUDA if you can. I think my point was that if you're trying to enter this space like learn about the models, learn about how they're trained, what the data looks like, what the compute looks like, one axis of that is how to do more efficient training and inference and one part of doing more efficient training and inference is studying the hardware which is GPUs. So, like I think that's a very small subset of all possible knowledge that you could acquire and it's probably not the best place for a lot of people to start. That said, if you do it, you you've got to be one of the most hirable people in the world. Like, if you like really deeply understand the architecture of the new GPUs coming out and and how to control it, you're in a very small handful of people and like everyone will want to hire you. Actually, the sweet spot is not even CUDA right now. I would say actually it is Mojo. I don't know if you've been paying attention to modular mojo. Oh, I listened to your podcast, man. You had that guy on uh the other day. The whole story is Chris Lanner, industry legend, LLVM, Swift, all these things. And now he's turned his attention to the Python CUDA relationship, right? And he wants to basically create a viable CUDA replacement. It's basically Python married with Rust. for the last two and a half years. He was basically kind of stealth, not ready for production. When he came on our podcast, he was basically announcing to the world like we're open for business like you can use us now for for most models and like we actually are faster than like the native like uh sometimes the PTX implementation. I don't know how that works precisely, but he's a compiler language got. I think there's a nar there's one of those windows now like like you said like you know bet early on something that's there that's there's a shift. It's one of those windows now where you try to implement things. You basically like, you know, modular is 100 people. If you run into issues, you'll get Chris's personal help on things. Like I'm not promising it, but like probably, you know, like cuz he wants to work on improving the toolkit. And um all you have to do is just like it's not really about becoming a CUDA god because obviously like once you ramp up on on the the general concepts and principles, you can probably translate ecosystems pretty effectively. A lot of people switch from like jacks to to CUDA, but like the the thing is just like being able to experiment very quickly on a limited budget. Like efficiency is not just because you are trying to be an efficiency guru and that's your career and that's kind of boring. But it's really also just about being able to experiment very quickly uh and finding these these ideas. I also think uh VLM and SG lang seem like really good and important and here to stay. Like they'll probably just get larger and more complex to accommodate future systems. But if I were like a starting out grad student, I and working in that area, I'd probably like want to learn more about how they work. Awesome. Let's go to your research. I like to mention that I I first came across you because of CDE the contextual document embiddings paper. You can uh tell me the story about that but I just want to show you proof that you know I get one slot per day to highlight the number one AI story and you were the slot of the day for October 5th. Oh no way. I mean obviously you were producing work before that but like I thought CDE was a really cool exploration of like oh yeah you know embedding models are kind of like stuck in a rut. like here's actually how to make them very efficient by just doing it in two stages. That seems like a you know relatively simple insight that was done very well. But you have a general maybe information theory thing that maybe we should start with and then we can sort of grad our way. Yeah, sure. That sounds good. So we can we can circle back on that. That's that's really cool that you uh wrote about it. What was that almost coming up on two years ago? Yeah, this is the post I wrote. I I called it a new type of information theory. We don't need to go into this. There there's this paper about a concept called the information. Maybe I'll give like the most simple explanation which is if you say you have two text files. One text file contains a paragraph of information about New York City and then the other text file contains the same text but encrypted with like Shaw whatever encryption algorithm. So it looks like random letters, but if you decrypt it, it has the same text as the first text file. From the perspective of like Shannon's information theory, these two files contain the same information content. like relative to everything they have the same number of bits but it's it's very clear to the observer that the first text file which is plain English text is like much easier to read and easier to process even though they have the same information and so there's this theoretical framework proposed in this paper which is a theory of usable information under computational constraints from 2020 it really doesn't have that much rest there not aren't as many citations as you would think but I think it's a really really neat idea. It's like we should measure information with computational power as a constraint. So like they have this idea they call V information of how much information is extractable from a given like file or or code. So in that case we could say the left text file actually has more extractable information than the the right text file. I think that's like really good. That captures a lot of our ideas of how these deep learning systems work. Like why does pre-training work? Like if you have two sets of weights and you you want to train on some downstream data set, one set of weights is pre-trained, one set of weights is randomly initialized. Why is the pre-trained model better at all even though it's never seen your data? Maybe one way of looking at that is that it has like it makes the information like more extractable somehow. Like there's this concept of like computational processing that you can almost like store up. I like this as a just like a lens to view problems with like how much information is stored where. Like if you if you get a a set of model weights or like an activation vector and you open it up like print some tensor numpy array, it looks like random numbers, right? Like there's nothing human intelligible about that. But really it's this complex combination of like the training data and the training algorithm which get compressed into model weights and then the actual computation that the model is doing which involves like manipulating these numbers in ways that we don't understand. So it's like this really highly compressed nonlinear combination of all these information sources mixed with like computation and I just think we don't have like the right words of of discussing this. I think I like the information theory analogy because back in the day, you know, we had phones and like telegraphs and and people were just sort of like building the phone system with these crazy horistics to like send information across the country or send telegraphs across the Atlantic. People were just like trying stuff and then uh we kind of found stuff that worked and we we ran with it, but it wasn't really optimal. And it wasn't until someone came along and proposed this concept of like a bit like a one or a zero that tells you something. And once we have a bit, we can do all these things. We can like count the amount of information a signal. We can do really good error correction. We can measure properties of distributions of things and and we can build like a really good system for for phones and then eventually which led to computers. I'm bringing this all up because I don't think we have I don't think we know what a bit is yet in terms of like deep learning models. I'm going to graduate for my PhD this year, but I didn't figure it out. So, if you're listening to this, maybe you can like, I don't know, spend more time on it or you're smarter than me or you have a, you know, group of collaborators, you can all get together and figure out what the right lens to look at this stuff is. But even by just asking these questions, I think I was able to conduct this research agenda that I'm kind of still working on actually. Yeah. Uh what do you call this field? I don't know. I don't know. I called the post a new type of information theory. I I don't think it exists yet, I guess. So maybe it'll it'll get a name once uh someone actually comes up with the right set of definitions. I think V information is a is a really good start. There's a couple related threads. Uh so first of all, you don't know this, but I actually have been trying to accumulate uh data about Shannon like this like a Shannon information theory view of language models. I have a lot of notes. This is actually on my GitHub for people who are watching along. But um you know like at the limit if a language model has 175 billion parameters using 16 bit you can it will take up 350 GB. You can compare that to Wikipedia. Wikipedia is about 150 GB. you know, let's say GPC3 can store two Wikipedias, but like is is that a relevant measure of information storage, right? It is not because you can compress Wikipedia a lot. There's a lot of repeated patterns. tokenization is is like the first form of compression. But I think there's a there's a related talk from Ilia Sutsgiver about how deep learning is machine learning kind of is is is compression like it you have a data set you compress it into a model that is smaller than a data set but generalizes and like uh has like you know some some amount of acceptable loss. I think that one of your commenters on on the on the post made this direct comparison with complexity which is what Ilia how Ilia sees it. So I think like people have this information theory idea or approach to language models. It is just not precise because exactly what you say like it's we don't know what a bit really means. We don't know what like the most legible legibility is is a word that comes to mind in terms of like like it matters to us that it's human readable like even if it's SH one SH 256 I don't care but like that is less readable and therefore there's more I guess I don't know entropy is not the word because it's it's directly convertible but it's just less useful. Yeah. Yeah. Yeah. Useful is a good word. I think maybe useful information or usable information is is the right lens. and Komagarov is a really interesting connection like Komagarov complexity. I think that's a really good concept for for computer scientists. So I'm not sure exactly about this specific talk or like what he was trying to say but I I think that we have a very good understanding of language model pre-training and there's a deep connection between language models and and compression. Actually may maybe let's let's start with the embeddings. We can come back to that. Okay. So, is this uh are we going to the first paper? Actually, let's go to your Wikipedia uh numbers if if you still have access to that. So, this 50 GB for text of Wikipedia, that sounds like a pretty high to me. Is that that's uncompressed like text files? I don't know. I grabbed it from Andrew Wang, so I don't know. Okay. Okay. No, no, I'm probably off. I just sort of have the sense that like when you store text, it's generally like very very small, especially when you zip. Maybe he's including all the languages, all the edits. I don't know. Yeah. Yeah, that could make sense. That could make sense because I I guess what you you say from, you know, if you want to do apples to apples comparisons, GP3 can store two Wikipedias. Is that right? 2.3 Wikipedia something. So, I thought it would be a lot more. And this is actually an experiment that you could do. You could like just train a model on Wikipedia and keep training it until you can perfectly extract all of Wikipedia. And that would be like a good way of knowing like how many Wikipedias can GBT store. I like I like that idea. But I think this type of like back of the envelope math is it's really useful for thinking about problems and like grounding yourself in the real world even if you can never quite answer the questions you want to answer at least like in 4 years. If we think about embeddings, you know, vectors that people use for search, we can do the exact same kind of math. So if you use the OpenAI embeddings, which last time I checked, I think have 1,536 dimensions. So that if you say there's 16 bits per dimension and like half precision floating point, it's something like 20 kilob of information in a vector. And if you want to store 20 kilob of text, that's a lot of text, like many many paragraphs that you can perfectly compress into 20 kilobytes. And so I think this is kind of like the idea we had. I'll give you the practical explanation which was I'm well first of all I'm a second year grad student. I'm like going to these conferences seeing all these other things people working on and thinking you know like what the heck like how am I going to like have my own little area to do work in that no one else is working in already. And so I spent a lot of time coming up with bad ideas and my adviser would say no like that's not a good idea to work on. Many times this happened and like even my first year and a half of grad school was like a lot of exploration and a lot of like coming up with bad ideas. And then honestly I'd be interested to see how he remembers it. But I think I wrote a sequence of proposals about different projects and then I came up with this idea. I was like, "Oh, we should just try to do as well as we can to reverse engineer the text that's in embeddings." And I and then we were talking about it. He was like, "Oh yeah, you should just do that." And then that was the end of the proposals. And then I was just working on that problem for a long time. Which at the time I was really motivated by that because I was like cool like my first as a grad student my first sort of like official like sign off on like coming up with a good research idea. And at the same time there was this big rise of this startup business model called like a vector database. And there are all these companies popping up raising money raising money uh getting like crazy funding and then actual applications being built that do something where instead of exchanging customer data they exchange vectors. So we had this like very grounded question of like what data are they actually sending when they when they send the vectors. Like first of all you have this information theoretic argument that when you send one vector there should be a lot of text recoverable just in terms of like a lot of these things represent very short documents but they actually have many many bits. So like the problem seems trackable. And then second of all we had this justification of how the product is actually being used. like if if someone hacks into a vector database, what do they actually find? If that makes sense. So, we were working on that for a while. I think I have the the the talk that you did from that Sasha highlighted uh is this one. Oh, yeah. Maybe that has the graphic that that would kind of Oh, go one before. I think one before. This is actually Yeah, this one's good. Yeah, I like having visual aid. I like how I like giving people breadcrumbs to follow up if they if they're interested in digging more. But yeah, I I remember this is a pretty uh hot area research uh at the time and there's been some really interesting follow-ups like we we ended up building a system that can do this quite well like taking an embedding and I think our highlight number is like at a certain length like a long sentence length we can get 90% of the text back. Exactly. And uh a lot of people were able to do stuff with that like they can for example I know these people that work on a problem of like debiasing embeddings and like in one data set they do something they have a procedure for like removing all latent features that correlate with gender. So they can produce like useful embeddings that from some perspective have no like information or usable information about gender. And they'd been doing that for a while. And then they actually just used our tool and they so like they would put in a sentence like this woman is a doctor at Wild Cornell Hospital in New York or it say this woman is a doctor. She works at Wild Cornell and then they would run their procedure and then they run our embedding to text model and now it would say like this person is a doctor. They work at Wild Cornell which is pretty cool. So they have like sort of textbased evidence that their method is actually removing gender features. But let me talk for a second about the research phase here cuz I thought it would be I mean I know if if you've ever heard me talk about this I probably told you about it but just for a wider audience I like thinking back on this because it was probably my in some sense like my greatest victory of grad school was like working on this embedding inversion problem for a while for quite a while and and proposing a lot of approaches and like testing stuff. I think sometimes you do stuff and it's clear it was a bad idea. Sometimes you think you should have figured it out earlier. And then sometimes you do stuff and you kind of realize it's really complicated and and probably not worth it. So I was testing different decoding algorithms for embeddings that are closer or text that's closer to the text that's in embeddings. And I was testing these kind of like inference time adaptation models for samplers. I think we tried a lot of architecture and like kind of training tweaks. We should have tried RL. I think that would work. But well, finally we found something that ended up working. And I guess I'm just saying this all because I thought it was like so rewarding. Like we were just banging our heads against this the wall. I would have bi-weekly meetings with my adviser who kind of suggest things. Sometimes we would agree we were mutually stuck. Sometimes I would get feedback one way or another and and try something new or try a couple things. and and we had this idea that it was possible from the information theory arguments and this other thing where we would we would kind of like take our best guess at what the text was and rembed it and see that it was kind of far from the true embedding. So we had this proof that like a better method could like leverage this kind of information. And then when we finally solved it, it was it was awesome. Like we had this number that was like 30 for months. I think at one point I got it to 35 and actually I think I was like, "Oh, I'm done." Like I got it to 35 and and my adviser told me, "Oh, no." Like that's you can't really just propose a new problem and show you push a metric from 30 to 35. That's like confusing and probably not that meaningful to people. And I think I was, you know, that was kind of like a local minimum for me where I was like bummed. But then we ended up getting the number to like 97 or something which neither of us knew were possible. We were all just we were just kind of staring at this graph like oh my god like who knew you could get this much information from an embedding and that was like so great like um just sort of this it was so rewarding and so it was invigorating honestly like that research process of like we picked a good problem and then we spent so long trying stuff that didn't work which I'm probably forgetting how frustrating that was. I'm sure it was terrible, but then like actually solving or at least like coming up with a much better way of solving the problem. I don't know if I'd say we solved it, but we definitely learned a lot from where we started was like was great and it completely solidified for me the fact that I should have gone to grad school to have this like life experience and like makes me want to do research forever. you're clearly clearly um sort of in love with the the the journey uh which I think is is important because this is what keeps you going through the the tough parts. Is this a good time to talk about the universal geometry side then? Yeah. Yeah. Yeah. Let's let's do that next. I think that's a good idea. So So we have this more recent followup and the So the first part I was talking about ended up in this paper called text embeddings reveal almost as much as text which was published in 2023. And then we recently had a paper come out on archive which will hopefully be published at some point and it's called harnessing the universal geometry of embeddings which was also that was probably like the only other time I felt like we've made like maybe there have been two more times but that that was probably the the second of three times where I felt like we made like a real discovery about like the unknown and it was like very rewarding just for its own intrinsic kind of elusiveness. And I'll start from explaining it in terms of the prior paper. So we we built a system that can you know do embeddings to text and and it works very well and we're we're very pleased with ourselves. And then we went to a conference, we talked to people about it. We talked to like the vector databases. I think some of them changed their privacy policies which was like somewhat gratifying. Um and then we kept getting this perpetual question which is like well you're just assuming we use the open AI model or you're just assuming we use the most popular text embedding model. if they fine-tune their own model or if they use a model that you're not training an adversary for, then you can't solve the problem, which is like true. Like none of the vector detect stuff works unless you have this assumption of like knowing the encoder and also being able to make a lot of queries to it. But we had this kind of underlying theory that all of the models learn very similar things. Like we have some preliminary evidence for that. like certain models that are fine-tuned from the same base, you can kind of swap their representations without doing much. Or if you look at the nearest neighbors, a lot of the models will give you the exact same nearest neighbors even though they have completely different training bases. And then there's this paper that came out last year called the Platonic Representation Hypothesis from some folks at MIT, which is really really compelling and I think just like great intersection of philosophy, representation learning, deep learning research. Like I I love this paper and that it's it's such a beautiful idea which is something like all models are trained on data from the world and there's only one world and so as the models get better by scaling data and scaling model size they're sort of converging to learn the exact same thing. And in this paper they have evidence based on correlations for doing this with vision and language models. It's very neat and so we saw this. So basically think about you know you're us you see this platonic representation hypothesis paper a lot of people have this shared idea like you know Claude and GBT4 probably do a lot of very similar internal computation because both of them are trained on trillions of tokens of human written text even if they have different architectures like maybe you know the actual basis or like the the numbers if you look at them look different but in some way they're like kind of computing the same thing and I think it's even more true with these like embedding models which have like really only one objective that works and they're probably all trained on like MS Marco which is a really popular data set and pre-trained maybe on Wikipedia. But we wanted to basically combine this platonic representation hypothesis idea with the vectex thing and produce a system that can like align models so that we can do embedding inversion. But, you know, it's it's valuable for more than just embedding inversion. Like, you can use this to kind of glue together models. Like, that's what actually got me like super excited. And by the way, like I think there's a few related threads. Uh, I think we did an episode with Nicholas Carini where he had an extraction attack uh on on one of the GPT models and uh they got it fixed. The other thing I want to I just really want to spell out for people just in case they're not thinking it through. Being able to invert embeddings also means that you can you can back out uh like secret prompts or context that might leak customer information that's potentially harmful and like obviously attack vector issue. I think one of the things I I had a question about was whether or not position embedding does affect it and like extension of position embeddings affected because obviously that like context are going to get longer and longer. Your ability to invert will obviously decrease with longer context. Well, now you know uh maybe not that important. No, no, no, no. You're totally right. So, we're operating in this space in in our work where the sequences are relatively short and the embeddings are relatively large. Like I think we're kind of at a great advantage from that perspective. And you're definitely right. Like if you embed an entire book to a 500dimensional vector, there's just no way you could get the entire book back. Like there must be this these kind of collisions. Like it, you know, in information theory, like if you have lossy compression, two different inputs mapped to the same code, which means that you can never determine which input formed the code. And I think that's probably what will start to happen. Like if you have two books and you swap just one word and you embed them, I don't know, someone can try this. You'll probably get like a perfect collision and in that case inversion is impossible. And even like when you don't take it to the limit, it probably just gets very very hard. Like things get super compressed. So I don't know how well this work scales. Like it's a great question like exactly how much information you can sort of cram into one of these vectors and I I don't have a sense of where the boundary is. It'd be interesting to talk to some one of the like linear algebra people from like the math department on like how literally can we take inversion like you know how like what what measures of a matrix do they have where we can like kind of run that and like try to get some meaningful information out of that. This is like where information theory starts to collide with uh linear algebra and all all the other stuff. Totally. Yeah. There there's always this um this detail where we're we're running these on computers and so we don't actually have like real decimal numbers or real numbers. We have like floating point representations of numbers which are like very it kind of like throws a wrench into the mix. Do you have any consideration of like superposition when like sort of nonlinearity like you could like stuff information in the lower bits? But I don't know if that matters. I I really don't. It's just like a nice thing to think about. Yeah. Yeah, it is a great question and and I get a sense that like a lot of the less important bits are more useful for computation and maybe the higher order bits are more important for like storing data or something like that. But I'm not sure. These are the kinds of questions I'm actually hoping to explore over the next few years. Like um I'll skip ahead for a second. So we we have this result that's like maybe the the third sort of like discovery I was alluding to which is like a way to measure the exact capacity of a language model and we get this number if you train a language model on a ton of random data and you measure its rate of memorization. Yeah. Can you open the right curve? This is sort of the discovery I'm talking about like no matter how you scale the training size you hit this like perfect perfectish plateau in order memorization which we call the model capacity and the the question I've been stuck on in the back of my mind for a while is like how is that actually implemented so like this is a transformer that is trained for many many data points and many many training steps and so like it's almost like if you have okay the 10 to the 6 point on the x-axis this the capacity uh we don't have to actually say the numbers but it's basically perfectly dividing its computation between all of the data points like every one of the 10 of the six data points gets like a tiny sliver of the model parameters because they're completely independent random strings so I don't really know if superp position is occurring here like it seemed possible to me that the model would learn like these completely independent columns of computation one per data point, but it's also possible it's learning some kind of like combined thing where it's maybe it learns like a load and a store and it's like sort of like loading and storing bits using these generic operations and then in the end it reconstructs the random strings. So even though like the data is completely independent the kind of like compute is is very similar in terms of like predicting random strings. But yeah, I guess this is all to say like about superp position and everything. I have no idea how the mechanisms are actually implemented inside the models and that's like one thing I'm hoping to learn about in the next couple years. It's a reasonable question whether it's meaningful to learn. I think there's a lot of things that is like nice to know but maybe not that useful. Lat space lat space alignment is very very useful. data set efficiency in theory. Cool. But like practically people are just going to go for the biggest data set they can like like the scaling laws are kind of worked out in so far as like the relationship of comput and data amount of memorization. I I don't know. I think maybe this is a good point to maybe also bring in the idea that Andre has been pushing for the last like I think year and a bit of um the cognitive core like the what is the dumbest possible model that knows nothing but is you is smart enough for tool use to do everything else right so you can run it on device and fast inference it's open source whatever uh so jumba 3N is like a really good candidate right now because it's like a 4B model that is like claimed to be better than llama 4 and GBC 4.1 according to you know certain arenas that shall not be named. This is where things get complicated. Like I it feels like language models kind of implement things and know things almost in the same way and it's like really difficult to disentangle like whether they're memorizing facts from whether they're like learning useful ways to generalize about new stuff. But I I agree this would be really nice. I don't think we have a lot of evidence that we can build a system like this that like is really really good at reasoning but really dumb about the world. Like I don't know if we have the tools. Yeah, maybe maybe not. I think the existence proof is humans, right? People always lean on humans as like the existence proof. It's not a great existence proof because I think if you talk to people about the number of neurons that we have and you make a neuron roughly equivalent to a parameter, we have something like 100 trillion in our brains. So like and like we consume like 20 watts of energy. Like it's nothing. Like we're so much better than than uh language models. It's not even funny. And then the the last feature of us is that we're self-proing which is uh not something that language models do as well. Oh like we forget stuff. No like we are not deeply densely connected like we like connections will drop therefore we're more efficient you know. See I see. Unlike a language model where everything is always connected all the time. Yeah. Yeah. Or like you preset the skip layers or whatever and that's it, you know, like it's not it's not really actually anything involved with learning. It's just like something you do based on ablations and like guesstimates. Even if we did want that, I'm not sure if we have like the right frameworks or methods for actually building like what you're talking about yet. I think the world is much closer to where you're at than where Andre is at. Andre is like kind of wishing for an optimistic world. Our conversation with Nan Brown was like, "Yeah, reasoning is emergent. If you gave the 01 harness on top of GPT2, you would get nothing because GPT2 didn't know enough. You need a GPT3 and GPT4 in order to then get 01 like as as GPT4 is the base model, which is like yeah, that's I mean that's that's reasonable. The way I put it is like in order to use tools, you need to like in order to search Google, you need to know at least search terms. In order to like then search Google and then learn what you need. And if you don't know what what to search, then like you might just be too dumb. Uh, I like the kind of uh ethos like maybe you could do some kind of pre-training or whenever the model doesn't know something, it can just Google for it and that way you try to encourage it to learn words without or like to to guess words correctly without actually storing the information into its weights. Yeah, it seems like a nice like goal at least. Yeah, you need some kind of online learning probably or memory uh and some combination of that. Yeah, it's exciting, you know, like I think like if that is the the direction of of where this all lands up, that's great. But like people aren't are not doing that. Instead, we're building, you know, $500 billion data centers in the middle of Texas and like, you know, all hell the the the god cluster uh that just will, you know, eventually wrap around the sun and consume solar energy because that's that's what we need. Do we finish out the universal geometry thing? Let me finish the kind of uh methodological description. So, so we had this goal. So, so, so yeah, back to the embedding universality. We started with going from embeddings to text. We know about this platonic representation hypothesis. And maybe I'll skip over the details but basically we had total inspiration from computer vision in this model from 2017 called cycle GAN which is among other things uh it's a way to map between two different distributions without any underlying notion of like which thing should be mapped where. It's just based on some kind of idea of closeness. So like the cool thing about this if you look at the top left so I guess the the top left is Monae so impressionist paintings and this picture on the right is a photograph. So like it's learning this kind of like semantic notion of what content goes where just by mapping a distribution of Monae pictures to a distribution of photographs without actually telling it which Monae picture should map to which photograph. It's kind of a subtle point I'm making. It it takes a little bit of time to wrap your head around or maybe like go to the middle one if you don't mind the zebras and the horses. So like it's clearly learning like what an animal is and what legs are and sort of like more abstract stuff like what uh the camera position should be and and what grass is and stuff like that. And it's learning like what a horse that looks like a zebra is which is actually like a complicated semantic concept. like we don't have a a data set that has a horse and then that horse as a zebra. We just have separate horses and separate zebras, but somehow this this GAN system is able to like elicit this sort of mapping property. It's like kind of a magical connection that it learns and I'm still like in awe that it's possible at all. But we mo more or less like repurposed this system and like we we built our own but like this idea we took it and we applied it to model embeddings where instead of zebras and horses we have like BERT embeddings and GPT embeddings or like two completely different models with different architectures. So I think these are GTR which is a T5 based retrieval model and GTE which is based on bird. So they have different training data, different architectures, different downstream objectives, different embeddings, but yet when we do this cycle GAN in the embedding space, they just perfectly sort of snap to the same place, which is amazing and has some pretty deep implications of like the Platonic stuff. Like maybe the models actually are learning a lot of the same functions or something and in some semantic way they're like very close. And yeah, this is a diagram of how our system looks. It's weird to me how profound it seems uh like you seem you seem like deeply impressed by it. And then the other thing is like uh when we talked to the uh the to Emanuel from Enthropic who did the circuit tracing and mech interpret me mechanistic interpretability work they were like excited that like the same thing in different languages maps to the same circuits and I'm like what you would expect? Yeah. Like I I don't know like why like I I I I don't know. I think I feel like this this feels more profound to you than it does to me. I'm like, "Yeah, obviously." No, that's that's so fair. Maybe it's just like self- congratulatory and we're happy that we're like the people that got it to work. Yeah, exactly. Yeah, it does it does seem obvious in retrospect. And I think that's like constant feedback I've gotten from research from, you know, people will tell you that this seems obvious to them. But you have to realize that like you came from a perspective of no one ever having done this before and they're coming from a a perspective of you telling them it's true. And like if someone had told you that this was true, it would be like maybe obvious to you too, if that makes sense. The way I would put it is that we have the intuition but not the proof. you have the you did the work and you have at least some evidence that it's true whereas we just have intuitions right so uh part of research is just confirming intuitions the applied part com