No Priors Ep. 32 | With NEAR’s Illia Polosukhin

Key Takeaways

[Music] a blockchain operating system might just be the key to a democratized web3 in fact more than 25 million users are already getting a taste of this thanks to near this week ilad and I are joined by Ilya palosuken the co-founder of near and co-author of the landmark Transformers paper to discuss the interaction of blockchain and AI Technologies what we should expect from AI agents how to handle the content authenticity problem and why the alignment problem in AI is really a human problem Ilya welcome to no priors thanks for doing this thanks for inviting you are one of the authors of the original Transformers paper we've also had gnome and Jacob on how did you get involved with that seminal work in AI I worked on a team on natural language understanding it's focused on question answering and the state of the art at this time was lstm's recurrent neural networks which you cannot launch in production at all because they're too slow and take a fair bit of time to process as documents scale so Jacob at the time was using attention for query similarity and he had this idea like using attention for encoder decoder type um I kind of jumped into it and with the shoes were playing around with can we actually get it to train and understand the order of words and do translation just based on you know attention so yeah it was pretty cool to explore that and obviously grew into something very interesting and awesome you originally co-founded near in I think 2018 meaning for it to be an AI focused company what was that initial Mission and how did it become a blockchain company yeah so we started this idea that we wanted to teach machines to code you know we have Transformers coming out there was a lot of kind of really interesting push in 17 16 17 around Ai and so our expectation was we kind of would ride the exponential growth of AI which has happened in this year we thought it will happen in 1718 and so with that we got a really interesting data set around language to code but more interestingly we had a whole community of developers mostly students who were doing crowdsourcing for us so we would give them code they would write descriptions we would give them descriptions they would write code for them write tests like all kinds of tasks and we actually faced the challenge of paying them because a lot of them were in China in Eastern Europe and kind of other countries where there's monetary control problems people don't have bank accounts and so we started looking into blockchain just like to solve our own problem the eye kind of uh expansion explosion didn't happen at the time and so we saw an opportunity of we can actually build a blockchain that we would use to solve this first and focus on that uh while kind of waiting out the AI thing to really happen and as you go into the blockchain rabbit hole you realize there's a lot more that meets the eye yeah yeah ended up being a pretty big mission exactly so you call near a blockchain operating system for any of our listeners who haven't used it like what does that mean so the idea is that we want to kind of go abstract right we want kind of an environment where you can discover and use web3 experiences you know benefit from them and not need to think about the low level you know implementations and quote-unquote Hardware that runs under it right so similarly how operating systems on your phone you know kind of abstracts out all the complexity of you know networking and payments and everything you just use it and you have apps that developers can build and so that's really what we're trying to achieve and kind of build this framework and platform for everybody to build their applications in web3 and really deliver it to the user to Consumer where do you see a lot of that overlap coming in terms of web3 and AI you've thought very deeply about both I remember when I first met you you were just switching from sort of nears original Mission into the blockchain based mission and you know you were known as a team that could literally build anything right like you had yourself and Alex and Pie guy and all these like amazing people and you went down the direction of building blockchain in part I think originally around this data labeling kind of mission and the ability to do payments and things like that and now I know you've been thinking a lot again about how these two worlds interact or intersect where do you think are going to be the biggest places of overlap between Ai and blockchain or web3 there's few levels of interesting intersections I think the the most obvious one that everybody talks about is various marketplaces for resources right be that compute model or data right so data crowdsourcing so those are pretty obvious right web 3 is really good at Market creating marketplaces creating traceability and uh providing like an equitable place for everyone to participate now the more interesting ones is where AI kind of Agents right which you know we've seen like initial versions of but obviously they're going to continue evolving if we you equip them with a blockchain account right they are now becoming an economic agent that is able to pay other people and pay other AIS to do work right and they can communicate right and I think one of the things that a lot of people who are oh like just language models are just the same advancements like as everything before missing the point that this is the first time that a machine is able to communicate with people in the same way right there's no more need in an intermediate human that interprets data and then tells it to other people now machine can communicate directly to people and so it can task them with work it can provide them context and so I really think one of the most interesting cases is organizations that are run completely by AI right where quote unquote CEO role is taken by AI agent who is tasked by you know by Community or board of directors or whatever is oversight governance is to you know hit specific apis and follow specific Mission they can even give specific feedback with training data when they don't think it's doing the right job but what it does is like creates this kind of a new layer of management that potentially removes a lot of middle management right now which is like transforming information and context for each individual person and giving them specific area of work and then gather like kind of harnessing their creativity and putting it back together right I think that's is very interesting use case that kind of really melds blockchain AI together why like you have a traditional biotech cancer research commercial entity like why blockchain and why AI for that I use this example right we want to you know continue making progress on solving cancer right and it's a very complex problem right there's a lot of like specific sub cancers that you know need research and so all of this and like coordinating people doing experiments propagating information recruiting you know people recruiting the candidates right all of this requires like somebody to do this work and kind of organize the process and really set up a lot of Pipeline and you know funding and all those things and right now there's so much overhead around everything from you know how grant funding is allocated from the non-profits that collect money for research how you know like experiments are set up the information sharing like all of those pieces are really kind of broken and so you can actually have you know like recording coordinated effort that is designed just to do that and it can consume all this information and kind of specifically task you know who is the best person at doing the experimental which lab is the best at doing this specific sets of experiments you know fund them for this you know amount of money you know over oversee their delivery and then kind of iterate and you know if if it thinks this lab is not doing a good job fire them without having like extra you know personal affiliations that you know people do have I'm actually excited about some folks are already building some examples of this in like a simpler uh forms but I think we'll see you know first organizations like this probably even this year where potentially it was a simpler missions and kind of more straightforward like kpi metrics but where kind of this information propagation and onboarding of people happens already through a kind of uh language model AI agent a simpler version of this that I've heard people talk about and it may be the first step towards it is actually providing on-the-job feedback via an AI versus like a human manager with the idea that it depersonalizes the feedback right so if you have a agent or an AI providing feedback some surveys at least have suggested that the average employee may be more comfortable with that because it feels more objective it feels depersonalized it feels like it can be provided in a directive way and it seems like that's one aspect of sort of this Ai and CEO concept that you're describing do you think the first place that it'll show up is Dows or do you think it'll show up in a different part of the community yeah I think Dallas is and especially what happened to his dad's there was a lot of people who were really excited about Dao's kind of as a concept and so they put a lot of time running them but it's actually a very like not interesting job right it's like in young board new members you explain to them all the same thing you know you answer to their questions and so that's the part which like you can already automate right you can like have a Discord bot that is like have all the context about the Dallas you know interactions and kind of onboard new people and it gives them like new you know tasks to start with and kind of coordinate them so I think that will be the first place where this kind of starts showing up and as well because you have like payments kind of very like there and you don't have any social constraints that usually you have in like regular organizations like you know I I a lot of people will Revolt if you like tomorrow say hey by the way your new boss is the say I models yeah yeah how do you think about AI in the context or I should say blockchain and AI in the context of things like alignment yeah so I think this is a very interesting topic so I have this view that we need human alignment instead of AI alignment so right now kind of when we talk about you know hey we need to align AIS with like human values but the reality is that you know all the problems that exist they all exist because of humans doing things and and they've existed before I actually like to use the Byzantine fault tolerance problem right which is basis for blockchain but the its roots are in you know history where there was people propagating misinformation and you were trying to like figure out how to prevent misinformation in the Army right so this is like a really old problem of misinformation and kind of um like how to work around that and so I think what we need to start doing is figuring out how do we build a society that is actually able to deal with uh kind of effective misinformation at scale right so like we've kind of built like a lot of our society has started building up tolerance to misinformation around you know TV and mass media but we don't have like a system and framework around dealing with it at scale and that's what AI brings brings just scale to the same problem and so this is where reputation identity and kind of systems around our social code operating system that powers our community kind of communities is really important like how do all these pieces work together and how do they actually operate when there is malicious actors who potentially are able to you know in Mass create like very personalized misinformation or create you know fake political actor that is you know convincing every individual exactly in what uh they think you know that government should do to get elected and the this is where web3 comes in as like a set of Primitives right we have cryptography to authenticate content and create uh a path everything from you know you take a picture of his camera some of them already have a secure Enclave that can sign the image that's taken and so as that image gets processed we can actually propagate that information and have a proof that it came from you know specific time and place and then being processed by specific set of filters right so that can give you like an anchor then you still need to know kind of who is publishing what right like recording this podcast you know people listening to it it could have been completely generated at this point right but if for example we all sign the you know the final podcast and say hey yes we've recorded it and this is valve content now when somebody's listening to it they can check that indeed hey this content is signed by us now the question of us comes in right so this is where kind of identity and reputation is important and so this is where uh kind of Unchained identity becomes your kind of coalescence of all of the content and all the interactions that you do and then that links to kind of you know reputation in different communities uh and uh provide context for people who are watching for this content to be able to understand you know who is this person who's talking or where they're coming from and what are the information values they have so I think like it's it needs to be a kind of systematic approach and it'll start with pieces right I think one of the important pieces will be kind of a green lock similar to SSL transition on the content right like as you go to YouTube as you go to uh you know New York Times you actually will see that like hey this content been signed by this party and this party is in some trust root or trust Community like uh graph of communities that you are following right so that's probably like one important piece and again blockchain and cryptography is just like tools to enable that product variants and then from there you know we need similar things on the government level right when you file paperwork when you file you know your identity the fact that your SSN is a you know number that you give to everyone which is like supposed to be secret is like for example ridiculous right so things like that is like all of this needs to improve and kind of upgrade to this new level where like a massive amount of kind of at scale of things that have been happening now are possible what do you think is the most likely form of blockchain based identity because you know the blockchain really has been the earliest place where you've had programmatic actors interacting around economic and other utility functions right it really is money as code and effectively smart contracts are ways to programmatically interact with that right so you you had almost like the execution layer without the intelligence and now we're adding the intelligence you have the cryptography but you're missing a real sense of identity which is needed if you have an agent or bot representing you interacting with another agent which is probably where a lot of things will work in the future online what do you think is a most likely form of identity on the blockchain and why hasn't happened yet it has happened to some extent right we have you know like millions of people actually using blockchain right now and they're using it more for financial use cases and kind of that's their financial identity the wallet is identity kind of thing yeah well it has became an identity right and the reality is like your quote unquote private keys are your identity but that's just too hard of a concept for people to actually work with right and so on near we actually changed that we you know you have a properly named account so like mine is root.near which can have lots of different private Keys accessing it with different permissions right I can give a key and in a way permissions to an agent to for example interact on behalf of this or I can withdraw it right I can give it to specific application Etc so like a more extensive model is needed that's one we need to have more social interactions kind of being spawned from this and so this is again blockchain operating system is powering actually social interactions and kind of communication we actually have a project working on chat and other ways of using now this identity it in more places it's mostly because we didn't have a critical mass of this applications that are using this identity so to for it to really become kind of the core and if it's not the core it's not as useful because nobody you know like hey you don't have it so like we're not going to use it as a default thing everywhere so like we really need to kind of go over like again I think SSL is really good example of something that's like it it delivers value it's clearly valuable but it was such a like uphill battle to get it there right and so I think like until you have this critical mass of like kind of website switched and browser support it didn't become a default right so we kind of need like the same here to happen like we'll need to have a critical mass of applications using you know identity and then uh then we kind of seize it like in browsers or wallets or whatever like applications to hold it and then we'll see a transition function happen where like hey oh you don't have it like you should get it because it's actually easier and better to use it and it gives you like more Financial Freedom as well and more upside where do you think the most likely failures like system-wide are are to be like with um you know growing capabilities in AI like where where do these mid against in terms of uh reputation systems with blockchain or like content Providence are are likely to how is that going to manifest in ways that affect us yeah I think there will be probably next year will be very interesting in U.S because I think this this will be a place where everybody will just take whatever their toys they have in toolbox and do it even Just for kicks right even if it's not malicious although some players will be malicious and I think that what we'll see everything from like completely fake narrative candidates uh to like I would be very interested to see like a web page where you land and you know you log in and it literally generates specifically for this user based on their interest a agenda for this candidate right so like hyper focused you know marketing for candidates based on like who's this um voter is right so things like that like we'll have all those possible things where the media will kind of be flooded with like you know you can spin up New Media right now and just generate content about your candidate like that you want uh and then Market that so like you can have like all kinds of things now just exploding without any way of like framing it on the user side if like does this have history is this coming from the right sources has it been validated right and so I think that's going to be a really um uh important I think the other side actually is law enforcement and this is sadly already happening the people are using these tools now in very malicious ways right now and law enforcement don't have a like really good ways to deal with this and so I think everything from this like on camera like signing we need this now like they really have no way to like kind of identify uh if the image was generated or not and similarly Like For You Know audio recordings and things like that like there needs to be kind of additional kind of levels of uh verification and this goes into actually like video calls and voice calls because right now somebody can call you on the phone and play a recorded record like generated audio of somebody that recorded 30 seconds off right and this can be this very nefarious means right it's a huge Consumer Fraud problem already well it's huge consumer but it's also like beyond that is becoming like a real criminal problem like criminals are be able to use these tools now and it's like the barrier of Entry there is like very low and so uh this is where like you really need like you know the phone calls the kind of all of this like you need more information identification and like kind of cryptography embedded into the system otherwise it's completely going sideways really quickly yeah this is where people would be using apis like element or lfg or 11 labs to create a void snippet right where they'll upload to your point 30 seconds of voice train a model and then the output sounds like close enough to the person that you could fool uh financial advisor or a bank or somebody else to you know do transactions on your behalf or things like that yeah or and you like swipe their phone and and now you're able to like impersonate completely right so yeah so this is like a real problem and like having kind of authenticated passes required there to really establish and like we have actually like the phones are actually have so much already like we have face ID and fingerprints we have you know there's secure enclaves that sign things that are like haven't been hacked as far as I know like so there's like a lot of the pieces are there now we just need like a product stack that actually pushes it uh to the user and and like to the products yeah that makes sense I guess one other area where some people have talked about overlap between the blockchain world and the AI world is around training and there's almost like two or three different forms of that one form of that is there's a lot of GPU capacity that was purchased for mining on the crypto side and given how valuable GPU is now on the training side there's all sorts of sort of models to aggregate gpus specifically for training in different ways you know aggregating access capacity and then separate from that there's ideas around well can you train a model in a distributed way across a blockchain more generally do you think either of those things are Concepts that will work or how do you think about them relative to the Future yeah I mean it's interesting because it it sounds like such a no-brainer that hey let's grab those gpus that for example ethereum just moved from proof of work to proof of stake let's grab those and start using them the challenge is the gpus there are like not the ones that AI folks want to use right uh like kind of old AI is really zeroed in on like how do I get a100 so h100s and the gpus that like folks used for ethereum mining and like similar um is like older ones like uh that are not also focused on like floating Point arithmetic for example as much and so the challenge was more around like people who did did that like core weave is probably a good example right they were Mining Company like it's more that they had a know-how how to build data centers and they can like get access to massive like talk to Nvidia and like get massive access to that versus like repurposing the same gpus although I mean obviously like for smaller models for some specific uh maybe in for instance there's there's maybe a transition there's a question of decentralized training right uh in general right like hey we have like lots of gpus everywhere can we train it and the reality right now that the requirements on bandwidth right like people who are training these models right now they have like a you know 800 gigabit connect right between the gpus right so maybe you have 100 megabits on between this usually not and you need to like replay and like uh work around problems for decentralized so I think decentralized training right now is like still not as realistic although there's some research people are trying I think an inference is really interesting because we do need so much more compute for inference than we need for trading right like it's it's a very interesting like economy of scale you train once like llama trained once and then everybody runs it everywhere and so the inferences where I think there's a lot of interesting cases one is you want it to be private right right now if you're doing inference uh you need to send it to some service and that service may or may not record it and uh okay both input and output second one is you want large capacity at like that can scale with more usage right tomorrow I have you know 10x more users I want to be able to scale with that and so this is where I think using some of this Hardware that exists as well as kind of leveraging maybe new methods of privacy and coordination that can again crypto has like NPC like multi-party computation there's zero knowledge proofs Etc like they can be leveraged to uh achieve that and have kind of uh secure like secure decentralized inference so I think that's way more realistic than training and also many more needed and then I guess one of the really early applications that Nero was thinking about was Data labeling and to your point the ability to pay people who are doing data labeling for AI purposes right and since that time I think a number of companies have really grown out in terms of the data labeling World in a centralized way there's scale.ai there's serves there's a few others do you think the best solution in the long run is still a decentralized model where you're using tokens to pay effectively for labeling do you think things will stay in the centralized world like how do you view all that evolving over time yeah I think decentralized kind of the web stream Marketplace is a more effective way to do this and it kind of provides few interesting benefits one of them is that it opens up kind of the market right where you don't need to set up like a local office and kind of hire people and train them Etc like you can just open up Global Market anybody can join and you have a very specific rules right that if they follow they get paid right so I've used Mechanical Turk before for example and you can actually as a client you can just decline them paying them right so people in Mechanical Turk like the workers have very low kind of way to push back if if I say at the same time they don't have any like quality and knowledge assessment on the platform right so so I think having quality knowledge and this kind of escrow model all embedded into one Marketplace that opens up for everyone and get you know anybody everywhere can get paid at any time like offering that both the people who doing this work want because they kind of are more protected actually and it's like fair game and then the people who want to give tasks they can actually get access to like way larger uh Workforce they can like specify specific parameters they can you know price it at whatever level they want that's going to be the kind of future of it can you talk a little bit about what makes the quality control problem for annotation hard here right because one thing that I've seen with significant research Labs is like still continued uh in sourcing of annotators um for both pre-training sets and lhf because some of the external services and marketplaces can't get to the level of quality that they're looking for in particular domains so can you just describe the Dynamics there yeah so I think there's two parts one is like domain knowledge right um that generally like heart like it's hard to tap in in into like a very specific centralized service right because they need to kind of like for them to do payments do all those things they need to set up a subsidiary in whatever country they have the workers so you need to train them they need to hire them maybe it's contracts but like they need a lot of overhead that they do that for example developers let's imagine you know you're building a new really cool developer platform uh which uses you know language funnels and you want to fine-tune on code right well the existing platforms like them hiring a bunch of developers uh to actually do this right and you know if they're doing this full-time that's like super complicated then uh kind of building out the validation tooling for how to like cross-validate that the work has been done now on Webster Marketplace you know any student can join and like do do this right they don't need like you know join it like get a contract with a specific company they don't need to have the company in the local region to work with them um and like students you know for coding for example are really interested in doing this because they are usually don't have much money and this is a way for them to practice their uh work anyway and then as a task Giver you can actually specify the specific way you want the cross-validation to happen and uh one of the things we've done uh it's like honeypots right where you actually specify specific types of incorrect answers that people need to Mark as Incorrect and otherwise they actually lose uh the buy-in and so there's like actual like very clear like economic Game Theory where people have buy-ins they uh they lose them if they like do for quality of work and so they have um like way more incentive to do this versus like let's say if you're working on a contract there's like way more leeway usually uh if you're not doing your work right so it's like just way higher kind of uh self evaluation as well that happens and so I mean there's a lot of pieces that needs to come together for this to be like high quality but again it just opens up this Marketplace and makes it effective and it in a way removes a lot of the human part as well one thing that I think is really neat about how near approaches Innovation is you do both internal sort of near Road mapping and product development and then you also have a series of things that you either spin out or spin up or you're sort of involved with sort of these ancillary companies or projects or efforts what areas are you most excited about over the next coming year in terms of either nearer some of these other efforts that you're involved with so we do actually have a project uh in this web 3 AI data Marketplace that we are spinning out um to focus on not now like they build a product they have all the pieces now it's like ready to actually go to market and bring customers I think the the really interesting area is kind of partnering with existing kind of I'd already website enabled or interested in web3 teams who want to give access to more functionality to their uh users right we have for example sweatcoin which is really good example of like it was a web 2 project that had 120 million installs that had a ton of people using it every day kind of for a very specific use case right kind of tracking their steps and you know maybe getting a discount on their next shoes but now as they transform into F3 they're kind of opening up right and you can now participate in economic activity you can you know learn about new kind of innovations that happen in the ecosystem you can now you know but like as they integrate more into block Library System I can potentially interact with like on the social side do the tasks and gigs and so like you kind of really open up the what before was like a very limited kind of economy to really do this like you know composable Open Lab I think that's really exciting and like we will see probably more and more examples of that uh and finally I'm really interested in kind of as I mentioned like because we have now open the web and social wear the kind of what I call future of SAS so I think a lot of between web3 and AI a lot of SAS will actually start being uh replaced because right now what SAS is is like one database with a specific UI for a specific problem the database is the same between CRM the hiring tool marketing Tool uh even some of the project management tools right the database underlying is like not that different and it's been just like the front end and like interconnecting all of those databases is like ton of work it always breaks right um but now you can have like the database you own right so using kind of step three tags and then you can build all of this front ends on top either through kind of block sharability system shared components or even through describing with natural language some of the interfaces and business processes you want to have right so the way people will interact is like kind of their business operations and all the tooling they need will start to change and I'm so I'm really excited about this space and like we have one company that is kind of you know starting uh to build out some of the things in this space and over next year we'll see kind of that evolving do you think that moves to an agent-driven World in other words when you imagine the interfaces on top of this that are sort of driving these business processes for future SAS applications to view them as sort of traditional uis or do you view them as agents that are interacting programmatically or some hybrid it will be a hybrid so I like in my imagination right now at least I expect like you can describe a business process which is like hey you know when we have a new creative from like marketing department spin up a Twitter campaign and create me a dashboard that tracks the conversions on our product right and so what it does it like creates you know the pipeline of those things and then it also creates a page where I can see like normal user interface of like analytics so it might be more generated Dynamic UI exactly yeah and it's like adjusted for specific use case you need and probably there's like a bunch of templates that is like you know fine tunes for your specific problem like and this is possible right now yeah I guess it kind of moves you um down the path of what you were talking about in terms of like AI CEO or AI is project manager where you're kind of morphing into a world where you're delegating to an AI to drive a bunch of activities and then come back to you with the results like you would an employee or a co-worker which is very different from the world of UI today where you just go to the same spot to see analytics you go to the same spot for communication you're good which is your email you go to the same spot for you know interacting with the workflow and you're saying this should be more of a dynamic world where things get brought back to you based on a series of tasks that you provide out yeah and there's like probably a shared environment as well where you know we probably will co-work on a business process and you know we'll share one display but then we'll maybe Fork it because I'm more interested in conversion and you're more interested in retention for example and so so that's kind of the dynamism right now that also doesn't exist where like we all look at the same you know jira task management and I'm like I don't really care about half of this stuff right but it's not a filter problem it's like I want different information showed in a different way author of the paper that changed the world here we are in 2023 is it bigger Transformers all the way are there other architectural directions that are worth thinking about that you're paying attention to I think there's definitely something around like how do we get these models to have the capacity to like let themself think before outputting or like kind of uh process more and I I think it's like still within the Transformer structure and it can be like Advanced but I haven't seen anything that's like really matches my intuition around us but I think the like the Simplicity of this architecture and like indeed like the the amount of optimization that's going into this right now is just it'll be really hard to match uh and kind of you know there's enough exclusivity you can express any function so like it's not this is not a problem at this point of like hey we don't have an exclusivity right it's more around how do we how to like compose a data set that's you know cleaner better or add some you know self-critique and understanding of like is this content correct or I need more time to think what versus you know hey I'm forcing you to Output next token even if you don't have an answer yet so I think that that parts we really need and and I think uh they kind of fit in the architecture but um just require more engineering and more different types of tasks as well for training I think like you know the fact that we're just using a big language model is kind of interesting because this is not the task you would expect uh everything to be able to you know just predict next token so like you know starting to obviously at all of H being already helpful but like starting to like hey can you critics the center what would be the balance Etc that is a training or fine-tuning thing or do you that as an inference thing I mean it's going to be like a combination right so I think we just need an architecture that at training time your enable to so like I mean this the simplest thing is like instead of outputting a token in the next right you can actually give it like you know empty token for example for some period of time and then when it says like okay I'm ready give a child to the next token right and so this way you can train into like think more before outputting and then at inference time you can vary it right like hey I'll give you more time to think you know uh or like no you have no time to think but then you can train it to like actually be able to like dynamically to uh so again this is like a very simple thing but like you can keep expanding on this you know output it and then feed it back and like is this the right answer like Etc so there's a few different models but I think the toyakov's point like the the fact that this model is like doing a really effective search in kind of this knowledge space means that probably like pushing more into that concept is more useful than doing more searches at inference time because like it means you already lost all the semantics if you're doing searches the first time I think you made a really interesting point where it's possible that Transformer architecture increasingly is getting locked in and there's two components of that one is it just seems to run really well on the main silicon that we're using right now for AI which are gpus and then secondly there's so much optimization we're going into it and so much being built around it that it effectively creates optimization that just won't happen for any models anytime soon and so you effectively end up with this interesting feedback loop or lock-in effect for this set of models do you think that we're in a spot now where this is just kind of the future for the next five years or 10 years or something or what do you think is the likelihood that other approaches or architectures will emerge anytime soon I mean there might be an another architecture that like reasonably fits with the same silicon I think that there's an interesting question example of there's a company that built an alternative right silicon that is kind of allows to process things in pipelines and so like the chips are actually like kind of smaller compute chips but they kind of all uh like in a grid and the data flows from one side to another right so the example there is on one side it's like a really interesting architecture you can build really cool things with this but it doesn't fit Transformers very well right like you can do Transformers with it but it doesn't fit very well and like your your cost like you get like you know cost to to Output ratio is not that interesting and so in comparison to you know you're just optimizing on gpus or using some of the new hardware accelerators and so this is where exam like I mean I'm not to speculate here on specific company but you know I wouldn't expect they will have like a ton of people lining up because like there is ton of Alternatives flow Transformers with coming in and like somebody would need to like go in and develop a lot of new architectures that fit better uh this model and so uh it'll be really hard for them to like be a viable business and kind of have the economy of scale the Nvidia is having right now just kind of continue optimizing and building best state-of-the-art chips right so unless somebody's like really investing in this I think it will be more around like what what else we can do with current silicon right and kind of combinations of this and then I mean maybe there's something new will come out yeah but with things lock in technologically they actually tend to lock in pretty strongly until there's a really big sea change or sort of the optimization of those things hit a asymptote and it's interesting because I think a prior example of this kind of um chip plus software reinforcement Loop was really the windows and Intel monopolies of the 90s you know they used to call it Wintel for Windows and Intel because it was such a strong Mutual lock-in effect where you had chips they were optimized for Windows and windows optimized for the chipset and it just kind of kept going from there and so it's interesting this is I feel like a stronger version of that in some sense where you have the underlying compute architecture and the most important model reinforcing each other in a way that kind of locks both of them in yeah and what what changed that is pretty much come of mobile right and creation of irm devices I run chips that are kind of optimized for mobile and then came back into PCS right so yeah so and unless there's like a completely new form factor which hard to predict right but also it's like that's a lot of investment to go from not just software not just Hardware but like full stack right Innovation yeah I think it's unclear if this is a strong enough Market Force but the short-term you know demand Supply imbalance around gpus with all of the growth of applications especially as like you think any of these applications work like inference needs grow right your ability to build enough for NVIDIA really to build enough gpus to service the demand is like it's blocking a lot of companies right and I think the question is like there is more incentive to make heterogeneous Hardware work than there ever has been and it's like can that catch up with the full stack optimization that you described the Cuda like investment that Nvidia has made it's super unclear but I think like there's been no reason to chase that until you know this past 18 months and I think now there is yeah but at the same time we have like every single you know large companies doing their own Hardware accelerator as well as you know a bunch of folks who are kind of spun out of those and so like we're going to have a you know a market full of Hardware accelerators which are still optimized for Transformers or at least like similar structured architectures hitting the market like this year in the next year yeah Elliott this is great I hope you will uh after a lot and I work through all of the Transformers authors like Pokemon style gotta catch them all I hope you'll come back for a reunion episode but thank you for doing this yeah thanks for jumping on for sure thank you find us on Twitter at no Pryor's pod subscribe to our YouTube channel if you want to see our faces follow the show on Apple podcast Spotify or wherever you listen that way you get a new episode every week and sign up for emails or find transcripts for every episode at no dashpriers.com