Google AI Documents Leak about "Google and OpenAI"

Key Takeaways

so there's some interesting news out today there's a leaked Google document that talks about the state of AI and where Google is in terms of AI and its progression called we have no moat and neither does open Ai and it's got some pretty big claims um it was leaked by a Google employee it's from a researcher inside of Google so it starts out saying we've done a lot of looking over our shoulders at open AI who will cross the next Milestone what will the next move be so let's talk about the fact that Google was seen as the number one AI company for a long time then openai kind of comes out of nowhere and maybe I would even go as far as say dominates the AI playing field of Chad GPT all their other software and now it's front and center in the talk about Ai and where we're heading then it continues but the uncomfortable truth is we aren't positioned to win this arms race and neither is open AI while we've been squabbling a third faction has has been quietly eating our lunch I'm talking of course about open source plainly put they are lapping us so this this document goes on to say that a lot of the big problems that Google was working on they couldn't quite figure out was solved kind of quickly when some of these models were just released to the public people are running Foundation models on a pixel 6. people can fine tune a personalized AI on your laptop in one evening their entire website's full of art models with no restrictions whatsoever text is not far behind and the current multimodal science QA was trained in one hour this person goes on to say while our model still hold a slight Edge in terms of quality the Gap is closing astonishingly quickly open source models are faster more customizable more private and pound-for-pound more capable we have no secret sauce there's nothing there's no moat that protects them from everybody else sort of coming in and having the same software the same capabilities Etc well interesting point was this giant models are slowing us down in the long run the best models are the ones which can be iterated upon quickly this is basically how quickly an open source model is doing 90 percent of what you know ninety percent of what Chad gbt is doing the kuna I'm gonna assume it's pronounced vicuna an open sourced chatbot and pressing gpt4 with 90 Chad GPT quality and then then it goes on to show the side by side comparison and where the kuna is next to Chad gbt and Bard Etc so and then the researcher asks what happened so basically meta or Facebook as they used to be called uh had this AI model called llama and it was leaked to the public so this model it had no instruction or conversation tuning and had no rlfh which means um reinforcement learning with human feedback nonetheless the community immediately understood the significance of what they have been given and tremendous outpouring of innovation followed and within just days between major developments and they're going to have a timeline at the end of this article that I'll show you but basically barely a month later there were huge variants with instruction tuning quantization quality improvements human vowels multimodality rlfh and many of which build on top of each other basically what this means is as soon as this thing got out there it was sort of the global Community started working on it and the Innovations just came much much faster than they would if they would have stayed behind closed walls at a big Tech firm now he's saying why we could have seen it coming and and in many ways this shouldn't be a surprise to anyone so basically he's comparing this to stable diffusion where low cost public involvement was enabled by a vastly cheaper mechanism mechanism for fine-tuning called low rank adaptation or Laura and in both cases access to a sufficiently high quality model kicked off a flurry of ideas and iterations from individuals and institutions around the world in both cases this quickly outpaced the larger players these contributions were pivotal in the image generation space setting stable diffusion on a different path from Dolly so what this is saying is that the software the AI models that get put out to the public where everyone contributes those tend to dominate versus you know these closed models in terms of cultural impact Innovation product Integrations marketplaces user interfaces Etc and I say Dali while it was impressive at first comparing it to something like mid-journey I gotta say like stable diffusion majority there they seem much farther advanced than Dolly at this point so whether the same thing will happen for llms remains to be seen But the broad structural elements are the same many of these projects are saving time by training on small highly curated data sets this suggests that there's some flexibility in data scaling laws and of course he says that directly competing with open source is a losing proposition then he goes on a bit to explain that basically Google can't compete with this they can't just lock people up for using it people can use it for personal use they will understand it better like the illegal cover afforded by personal use and the impracticality of Prosecuting individuals means that individuals are getting access to these Technologies while they are hot being your own customer means you you understand the use case so basically large like monolithic institutions can be as good as basically the whole World contributing and working quickly and sharing what what they've learned and he says paradoxically the one clear winner in all this has met him because the leaked model was theirs they have effectively garnered an entire planets worth of Free Labor since most open source Innovation is happening on top of their architecture there's nothing stopping them from directly incorporating it into their products so what he's saying is that Facebook actually this was kind of a big win for them even though it was a massive scope the only time that they couldn't do something genius is when they screw it up and it just happens I mean he's not saying that I'm saying that but yeah Google and openai have both gravitated defensively towards release patterns that allow them to retain tight control over how their models are used but this control is a fiction and he says we cannot hope to both Drive Innovation and control it and he's saying that in the end open AI doesn't matter they were such a big thing that happened but in the end they're just not gonna matter no one's gonna have a moat and on Google not opening eye nobody open source Alternatives can and will eventually Eclipse them and unless they change their stance in this respect at least we can make the first move meaning that their opening eye overcame them and is ahead of them in some ways but they're saying if they see where the puck is going they can start off start moving towards that spot first before opening eye but becoming more open and open source and so this is the timeline so February 24th 2023 llama is launched uh March 3rd 2023 the inevitable happens which is llamas leak to the public I remember the day that happened that was huge and people were kind of going nuts over this thing and so then a little over a week later so March 12 language models on a toaster um a week a little over a week later artem gets the model working on a Raspberry Pi now the model works too slowly but it sort of sets the stage for an onslaught of medication efforts great use of the word onslaught I love it the next day Stanford releases alpaca they're able to do fine tuning within hours on a single I think that's an Nvidia card so they're able to do training on a single RTX 4090 so suddenly anyone could fine-tune the model to do anything kicking off a race to the bottom on low budget fine-tuning projects papers proudly describe their total spend of a few hundred dollars this is why one of the reasons why AI is so exciting and so mind-blowing to me because the the things that it could do potentially is unlimited and the cost to do it seems to be so much cheaper than we can even realize and it's getting cheaper still it's like the most powerful thing at and ease it's it's crazy and then March 19 2023 we have that vicuna I don't know if I'm saying that right but it's uh catching up with Bard and Chad GPT and then there's some sort of an open source gpt3 and these models are trained from scratch meaning the community is no longer dependent on llama they took this leak of llama and they tinkered with it until they basically replicated now they have their own thing that is not sort of dependent on Facebook or meta in any way it's out they they basically copy and pasted it out into the wild so I'm going to post this article on natural20.com that's natural 2020.com go check it out my name is Wes Roth thank you for watching