Microsoft LongNet: One BILLION Tokens LLM + OpenAI SuperAlignment [SINGULARITY APPROACHES]

Key Takeaways

morning everybody David Shapiro here with a surprise update so yesterday July 5th this paper dropped long net scaling Transformers to 1 billion tokens now to put this in context this is a three gigapixel image which you can make sense of at a glance and I'm not going to dig too deep into the uh the cognitive neuroscience and neurological mechanisms about why you can make sense of so much information so quickly but if you want to learn more about that I recommend the forgetting machine but what I want to point out is that you can take a glance at this image and then you can zoom in and you understand the implications of every little bit of this image this is clearly an arid mountain range there's a road going across it there's uh some Haze there's a city in the background you can keep track of all of that information at once just by glancing at this image and then when you zoom in you can say oh look there's a nice house on the hillside and you can keep track of that information in the context of this three gigapixel image this is fundamentally what sparse attention is and that is how this paper solves the problem of reading a billion tokens so let's unpack this a little bit first I love this chart this is this is a really hilarious Flex right at the beginning of this paper right under the abstract they're like okay you know 512 uh 12K 64k tokens uh 262 a million tokens and then here's us a billion tokens so good job oh also I want to point out this is from Microsoft this is not just from some Podunk you know Backwater University this is Microsoft you know who's in partnership with openai uh and so I saw a post somewhere I think it was a tweet or something someone's like Microsoft seems like they're really just falling down on AI research and I have no idea what rock they're living under but pay attention to Microsoft my money is on Microsoft and Nvidia uh for for the AI race and then of course there's Google um but I don't understand get Google's business model because they invented this stuff and then sat on it for seven years so I have no idea what Google's doing anyways sorry I digress okay billion tokens seems kind of out there kind of hyperbolic right the the chief uh Innovation here is one they they have a training algorithm which I don't care about that as much I mean distributed training okay lots of people have been working on that but the chief Innovation here is what they call dilation so let me uh bring that up uh so what they do is let's see hang on where did it go where did it go where's the dilation diagram all right so what it allows it to do a dilated attention sorry so what dilated attention allows it to do is to zoom out and take in the entire sequence all at once which controls the amount of memory uh and computation that it takes to take in that large sequence just like you and your brain zooming in and out and keeping the entire context of this image in mind at the same time and so the way that it does that is actually relatively similar to the way that human brains do it hang on hang on where did it go I'm missing the diagram okay so what they do is they create sparse representations and those who have been following me for a while you might remember when I came up with the idea of sparse priming representations this is something to pay attention to because what I realized is that language models only need a few Clues just a few breadcrumbs to remind it to cue in as to what is going on in the message to what's going on in the memories and this is actually why it's really good at you just give it a tiny chunk of text and it can infer the rest why because it has read so much that it is able to infer what came before that text and what came after and so by zooming out and creating these really sparse representations of larger sequences it can keep track of the entire thing and what it does is it will take the the up to a billion token sequence break it up slice it up and then makes a layered sparse representations of the entire thing and it will therefore be able to keep track of it now okay that sounds really nerdy uh but here's here's what it does for the performance so with this sparse representation with this dilation and doing it in massively in parallel it solves a few problems so one you see that the runtime stayed under a thousand uh milliseconds under one second it's more about half a second all the way up to a billion tokens so because of that it's basically zooming in and out of the text the representation of the text that it creates in the same way a very similar way that your brain keeps track of a three gigapixel image as you zoom in and out you're like okay cool okay I see a bunch of cars parked on the side of the road um and you can just remember that fact oh let's do a quick test what else do you remember about this image maybe you remember that the the Hollywood sign is in the background over here somewhere there it is oh no that's not it but it's somewhere in here so it's like okay based on the cars and the Arid desert and that I'm based I'm guessing that this is Los Angeles right uh anyways point being is that oh there it is Hollywood uh so this is these are the Hollywood Hills and you can remember oh yeah there was a nice mansion over here there's cars parked over here that's probably downtown LA the Hollywood sign is over here so by keeping by basically creating a mental map this treats gigantic pieces of information not unlike a diffusion model when and because I I got I was clued in on that when I looked at the way that it was it was um mapping everything and I was like hold on it's creating a map of the text by just breaking it down algorithmically and saying okay let's just make a scatter plot of all the text here uh or scatter plot's not the right word but the it's basically making a bitmap of the text of the representations of what is going on in the sequence and I'm like okay this is a fundamentally different approach to representing text and this is also really similar to some of the experiments that I've done if you remember Remo rolling episodic memory organizer which creates layers of abstraction this does it algorithmically in real time so this just blows everything that I've done with memory research completely out of the water it also has the ability to basically uh kind of summarize as it goes and that's not necessarily the right word because summarization means that you take one piece of text and create a smaller piece of text but this creates a neural summarization a neural network summarization by creating these layers of abstraction and this allows it to zoom in and out as it needs to so that it can cast its attention around internally in order to keep track of such a long sequence now okay great what does this mean as someone who has been using GPT since gpt2 where it was basically just a sentence Transformer it couldn't do a whole lot more you know like in in this model up here the original GPT was 512 tokens and gpt2 I think was what a thousand I don't remember maybe it was five uh 512 as well and then the initial version of gpt3 was 2 000 tokens we got upgraded to 4 000 tokens then we got GPT 3.5 and gpt4 so we're at eight thousand and sixteen thousand tokens as these attention mechanisms get bigger and as the context window gets bigger one thing that I've noticed is that there are one these are step changes in terms of algorithmic efficiencies but in terms of what they are capable of doing as I tell a lot of my uh my consultation clients do not ever try and you know get around the context window limitation because one a new model is coming out within six months that's going to completely blow open that window and two it's just a limitation of the model so when you can read a billion tokens which by the way humans read about one to two billion words in their entire lifetime when you have a model that can read a billion tokens in a second that is almost that is half a lifetime worth of reading and knowledge that this model can take in in a second so when you have a model that can ingest that much information suddenly retraining models doesn't matter you just give it the log of all news all events all papers whatever tasks that you're doing you just give it all of it at once and it can keep track of all of that text in its head in its virtual head um all at once and it can pay attention to the bits that it needs to with those sparse representations it is it is impossible for me to oversell the long-term ramifications of these kinds of algorithmic changes and so a couple months ago when I said AGI within 18 months this is the kind of trend that I was paying attention to there is no limit to the algorithmic breakthroughs we are seeing right now now that doesn't mean that there won't eventually be diminishing returns but at the same time we are exploring this blue ocean space and we've we've all right for those of you that have played Skyrim and other RPGs we unlocked a new map and the grayed out area is this big and we've explored this much of this new map that is how much potential there is to explore out here and the other thing is this research is accelerating there's a few reasons for that on one of the live streams like someone asked me like how do we know that this isn't an AI winter and I pulled up a chart that showed an exponential growth of investment where the money goes the research goes and because the money is flowing into the research it's happening what you I mean we saw the same thing with solar and literally every other disruptive technology is once the investment comes you know that the breakthroughs are going to follow it's just that simple and this is one of those kinds of breakthroughs so what does this mean uh put it this way rather than trying to you know play Tetris with memory and you know trying to fit 10 pounds of stuff into a five pound bag now once this becomes commercially ready which it's coming it's it's possible on paper they did it so even if we don't get a billion tokens this time next year it's coming the what this allows you to do is let's say for instance um you are working on a medical research thing and it's like okay well you know we've we've got a literature review of literally 2000 papers per month to read put all the papers in this model and and say tell me exactly which papers are most relevant so the the the ability for in-context learning uh is incredible and it can hold more in its brain in its mind than any 10 humans can and this is again this is not the limit imagine a year from now we're six months from now when uh long net two comes out and it's a trillion tokens or 10 trillion tokens and what they say in this paper is that maybe we're gonna see a a point very soon where it could have its context window could include basically the entire internet this is a step towards super intelligence make no mistake that the ability to held and use that much information in real time to produce plans to forecast to anticipate to come up with insights this is a critical step towards digital super intelligence I am not being hyperbolic here and neither is this paper when they say we could conceivably build a model that can read the entire internet in one go so with all that being said I wanted to Pivot and talk briefly about open ai's announcement also yesterday that they are introducing super alignment so the tldr is that openai is creating a a dedicated team to aligning super intelligence uh which you know again I am super glad that we are living in the timeline where someone is doing this it's about time uh you know I've got my book out there benevolent by Design where I talked about aligning super intelligence and my solution is that you really can't but one thing that I want to point out is that whether or not you can align one model in the lab is that's part of the that's a necessary part of the solution I don't want to disparage the engineers and scientists at openai and Microsoft and other places working on this but while it is a necessary component of the of the solution it is not a complete solution and this is where uh researchers like Gary Marcus and Dr Rahman Chowdhury have testified to Congress saying look you know they expect that open source models will reach parity with closed Source models and then overtake them and so when open source models who anyone can deploy are aligned any which way that you want you lose total control so that while I definitely appreciate in value because we need to know how to align super intelligent models the good guys right the the aligned models need to be uh as powerful as all the unaligned models because in the AI arms race it's going to be AI versus AI in the example of cyber security where we already have adaptive intelligence in uh in firewalls and other security appliances basically you're going to have you know an AI agent running in your firewall versus an AI based DDOS attack just as one example you're going to have ai based infiltration programs versus AI Hunter programs on the inside so it's going to be spy versus spy and so we need to make sure that the that the models that we build that that do remain aligned that we do remain control over are as smart as possible and also trustworthy absolutely needs to to happen basically you fight fire with fire and I know that that sounds like mutually assured destruction and it kind of is which is another reason that the nuclear arms race metaphor is very apt for the AI arms race So This research absolutely needs to happen but what I want to drive home is that it is a necessary but not sufficient set of solutions that there also needs to be the adoption the implementation and deployment of Alliance systems and we also need to make sure that those Alliance systems can communicate and collaborate together so with all that being said uh big steps in the right direction but it is coming faster than anyone realizes and I stand by my assertion AGI by the end of 2024 actually by by the mid basically uh let's see September or October 2024 any definition that you have of AGI will be satisfied and then from there it's a very very very short period of time to Super intelligence now fortunately for us right now the only computer is capable of running them running these models and researching them are like the Nvidia supercomputers that they're building so that but that barrier that threshold is going to start going down because remember remember Nvidia at their at their keynote speech and and for several months they've been saying hey you know our machines are literally a million times more powerful in the last decade and we're going to do it again in the next decade well when your desktop computer is as powerful as today's super computers in 10 years you're going to be able to run all of these and then when you combine that with the ongoing algorithmic efficiencies everyone is going to be running their own AGI within five to ten years mark my words so time is of the essence we do need a sense of urgency and I am really glad that open AI is doing this again you know I'm not I would like to see more governmental participation more universities uh I would like to see something like a Gaia agency a global AI agency or an Aegis agency an alignment enforcement for Global uh Global intelligence systems um because the thing is is corporations and governments are not ready for this and uh that to me is the biggest risk because from it from a purely scientific standpoint I 100 believe that we can align super intelligence I wrote a book about it I demonstrated how you can take unaligned models and align them to Universal principles very very easily I've done it plenty of times the data sets are out there for free just search for heuristic imperatives and core objective functions on my GitHub but again aligning a single model is not the entire solution you also need the deployment you need the the security models we need to update things like the OSI model and defense in depth we need to look at the entire technology stack but we also need to look at the entire economic and governmental stack to make sure that companies are aware of this and that companies are start deploying uh these uh systems whether it's security checkpoints whether it's internal policies that sort of thing because when you've got a really powerful Cannon you have to aim that Cannon really really well otherwise it's going to kill everybody uh and again you all know me I am a very very very optimistic person when it comes to alignment and the future that we can build but at the same time when you're playing with fire like you need to make sure that you wear the proper safety gear uh because the the more energy something has the more dangerous it is and the level of energy or intelligence or however you want to look at it whatever metaphor you want to pick is going up very quickly so thanks for watching I hope you got a lot out of this it's the long net paper and then of course uh introducing super alignment but yeah thanks for watching cheers