No Priors Ep. 8 | With Neeva’s Sridhar Ramaswamy

Key Takeaways

foreign I've learned so much from you as an investing partner founder and friend welcome to the podcast thank you very excited to be here same I learned so much about companies and investing in Tech from you let's start with the background tell us about the motivation to start Neva when you are already part of creating the dominant search product yeah so Neva is a little bit off Back to Basics thinking uh and I left Google I knew I wanted to start a company I spent a lot of time with Vivek about what we wanted to work on and we ultimately came to the conclusion that like we're actually really excited about search there's the geek in us that like to help people find information that they needed and we were also ambitious enough to think that you know 20 years in we could rethink the search product and create a better one our aha moment is a little bit of an abstract aha moment which is we said if we didn't have to deal with ads if we didn't have to worry about like monetizing we truly could start from Back to Basics as both of you know in startups it's as much about taking advantage of opportunity as it is the original direction that you said so the first three years of nivo were really about building a better private search engine and honestly it also taught us a lot of pretty harsh lessons about consumer and you know whether they were ready for change um or not and really what we saw happen with AI and large language models last year was that aha moment when we realized wait we can have the great principles that we started Neva with and create a much much better experience and so that's that's a little bit of the journey to where we were but at our core Neva was like there must be a better search product it cannot be that there's one company one religion one product for the whole world so I think many people who use Google every day would say like it's actually pretty good and as somebody who was working on this um you could see I think sometimes users are blind when they have uh you know a default that's this strong what were the things you thought could be better and if I could add to that like how does that factor into the Neva mission yeah so I mean an important part at least early on was the private and the ads free and you know we have to say that we underestimated how much people especially in the US would care about it um as you know figuring out consumers is a very tricky thing people will often not do what they say they will do or will not even admit to things that like they will or will not do that's just the nature of the game for us for example we were surprised that um but it's so much better in Europe compared to the United States and you don't really think of them as being that different but in practice in terms of how many people care it is actually very different so a lot of the early Nava was really about how do we use the power of being privacy focused and ads free to create truly a better experience um so we've tried a number of things they have achieved varying degrees of success for example the integration of things like personal data personal preferences um but I would say the the fundamental challenge of Neva especially in the United States has been how do you get people to take that initial step of caring enough to want to change their search engine um once you actually get people to do that the job gets considerably easier and they begin to see all of the things that were not really that great about you know about that experience again as a startup founder as a consumer startup founder I think these are pretty harsh lessons in consumer psychology but one that you know one has to learn so more recently you guys had a big breakthrough in terms of experience and consumer openness to AI summaries which look very different from traditional search can you just talk about how this product came about and what you had to build to enable it yeah so you know in some sense AI summaries uh I am sure there are many Google engineers and execs that'll tell you hey we've been doing this for 15 years um it's kind of true uh Google launched something called featured Snippets like I think it was a long time ago 2010 11. um it was always known that you know an answer right in the main search experience trumps all um who actually knows this really well um elad will remember this Google knocked out live.com Bing's image search as the top image search product in the world by integrating image search right into the search experience turns out Bing you know live.com then was the one that had the best image search experience Google Now It Out by putting it into the search experience same thing happened with Yelp and with local it didn't matter how good Yelp was if you could show an answer right in the search experience that basically won similarly this featured Snippets which is really pick out the two or three lines from a website that is exactly the answer that the user is looking for was always a big win people love the product it goes back to essentially like Occam's razor like the simplest explanation um is anything that minimizes work people are going to love and so if you give an answer instead of letting people click on something of course they're going to like it this is the reason why you know the currency conversion widget on Google is widely popular it's not that you and I like can't click and go to another site but it's like ah why it's there and so answers in that sense are old um but the fundamentals of search have always been that you got back a set of opaque links and of course Google's entire business the trillion dollar business is built on you know this this again obvious fact that you and I cannot tell um really between a good link and a bad link we can say a little bit if it's the New York Times our brain like it basically tells us Ah that's a good site you know for most sites we don't really know we click we find out but the opacity and the linear scanning order is always an important part of how you know search has worked and so answers are you know this linear scanning is important to remember um this consistent desire whether they state it or not on the part of users to to get to the answer in the fastest possible way is an important thing to remember um but things like feature Snippets were never Deployable at scale the technology simply was not there um even if Google put the full mic off its Mighty Machine uh against the problem the coverage never really extended Beyond like five six seven percent and it would make website owners really unhappy you know like you're taking away my clicks um and so it was always like this edgy feature that Google would be like you know yes we can show this but not really too much um our aha moment with large language models where we were like wait a minute for the first time you know you have these models that can take like any content um and come up with a summary that gets to the heart of what this page is saying and oftentimes you have to do it in the context of the query if you have a Blog for example that has six sections and your query is really about one of those sections um then you better find out the right section to summarize uh and so a lot of it was just realizing that what was essentially previously unsolvable and summaries in particular are this frustratingly vague concept you and I can do a reasonable job if given a bunch of different kinds of content to summarize but actually making a machine learning model do that in general um is a tough thing so a lot of last year was really like understanding that but also trying to make it work at scale um which is a big effort on our part we decided that we didn't really want to be beholden to say like using open ai's API um for doing things like summarizing a 4 billion page index we built a lot of the technology in-house um but the final cumulative product is these cited summaries um which really is one fluid answer when asked a pretty complicated um question or or query um obviously many people are doing this uh doing this now but for us that was this aha moment of wait we can write answers a single authoritative answer for 50 60 70 of queries um and large language models as you folks well know are also general purpose Learners um the exact same Tech that can summarize a piece of text can also be used now to pull out structured information we realized that we were sitting basically at a gold mine Beyond Compare in terms of a better search experience you know most of what you see for cited summaries are in the context of information seeking queries but there's a whole lot of work coming that can tackle different kinds of commercial queries so this is the beginning of a lot of work that can be done to make the search experience better but the core really is if you can provide a believable answer to a question people are always going to prefer that over any number of links that you can give them people don't like clicking on links yeah it's really interesting because you know I overlapped with you at Google and one of the things I worked on for a while is mobile search and I remember to your point we tried to surface every single but at the time we were calling one boxes you know that would trigger with images or trigger with location information and it's pretty amazing that you're able to get to such high amounts of coverage just using the llm side how do you think about because I remember when we were building those individual pieces there's a lot of custom work there was custom indices for news and crawls and there was custom ranking algorithms you know everything you you had sort of specialization how do you think about the other 30 or 40 percent that you're covering or is the idea eventually to do everything via llms is that prohibited from a cost perspective I guess more generally how do you think about information retrieval related problems in this new world and how you map the different types of search queries and the different types of results against that it's a great question so for example in like the 55 60 that I'm talking about um I'm actually excluding the one box and that we already fire um so it doesn't include like the stock cards or the weather cards and stuff like that in fact we were working on a poor integration and part of what the pro team is saying is like wait wait did somebody ask for a weather just give it back you have the information already it's not that hard um for clarity Poe is the uh quora app yeah pause the quora app it's like I don't know what's the right way to put it it's it's like a chatbot aggregator it's a pretty cool app you can take uh some of the one box in and even there by the way um this code for triggering as you point out e Lots used to be like really annoying code sometimes it would be regular Expressions it's basically like a giant like you know Bala box when it comes to figuring out how to trigger right um LMS actually make some of that stuff easier if you want to extract structured information even from you user type queries and I'm sure like you know most tech people have dealt with this at some point in their life or the other all of us have nightmares about writing beautiful soup code in order to parse web pages uh it's basically regular expression parsing over ever-changing websites um it's horrible we have done a bunch of it in the first two two issues of Neva that stuff is also easily generalizable with the smallest model that there is at this point I don't feel um that there's like a natural limit to how much llms can be used with search I do feel however that there's a very strong uh limit to how many questions can be usefully answered and you realize with the shock that search engines are actually pretty terrible at a lot of tail queries that you and I will now no longer think twice about putting into a chatbot I mean what do I mean by that the other time uh you know Jason calacanus who like you know are you folks has a big podcast you know he just typed in how are the Knicks doing this year um into Neva and a bunch of other search engines and I was like ah this AI stuff does not work um but the real answer is no one in their right mind is going to think of typing in how are the Knicks doing this year into Google search because it just never gave great answers for stuff like this tail queries have always been served poorly I don't think that is going to change uh instantly um but queries that can be meaningfully answered I think a lot of them can be answered with llms for what it's worth the approach that we are taking which is very much like the beginning of how large language models can be applied to retrieval problems um is this technique called retrieval augmented generation again uh you know a lot of your listeners know this it's basically how do you combine a search engine as a tool that a large language model uses uh and even there there's going to to be generalization there is zero reason why we can't recognize that you actually typed in an arithmetic expression and fire off a python interpreter for doing this or some other API so again even in terms of what can search engines do we are very much at the beginning I think we're going to expect a lot more uh from these kinds of interfaces and the difference between like a chat bot and a search engine um that like combines a chatbot and retrieval is going to just look more and more bloody going forward um so hard questions will continue to be hard but a lot of questions that we expect answers for I think will be eminently answerable with llms as one of the tools that go in super exciting relatedly when I've seen people model out the costs of using llms versus more traditional IR approaches llms seem to be more expensive per query and you know I know that when Sachin ardello was talking about integrating these things into Bing he almost had this your margin is my opportunity style uh perspective relative to Google right I don't know if it's true or not in terms of how that would substantiate over time but it almost felt like the claim was that you know Bing was okay almost subsidizing llms integrated into search to try and draw or sort of hurt the margin on the Google side how do you think about the potential cost period of nature of llms for search is it is it really a thing do you deal with it with semiconductors or small models or other things or is it is it not really that that important of a consideration well first of all his um comment might have meant two things there are two ways to think about margin one is the cost of serving and uh the other is the margin that Google makes say on an Apple deal not clear which one he was which one he was talking about but this is a topic that you've written a lot on elad when it comes to like just llms and cast we saw something dramatic happen where openai reduced the cost of its API by a factor of 10. um that's a little insane this this early on but if you go back to the basics of your question um and think roughly like you know an average very large model call takes about five cents um that's actually that is astronomical because you're talking 50 um you know cpms for serving for serving a thousand queries um now the average RPM for U.S queries is about 40 to 50 dollars and clearly that will be a very high cost uh the rest of the world is a lot lower by the way like my memory is on the order of twenty dollars um if you if you average over the whole world so and um I'm sure you folks also know that Sydney for example will issue up to three queries uh for every question that you ask I mean it's an arbitrary limit but there are like sometimes you need to ask more than one question in order to answer it well put that way um yes this is an astronomical cost but personally I feel that there is more and more evidence that says that you don't need like the full power of the largest biggest model to get most things done certainly the way we think about cost um paid summarization for example um we have like we're very comfortable with using models that are in the 5 to 10 billion parameter range we are very good at fine-tuning them there's a human feedback loop that is about to kick off and be there so whatever can be done with very large models for large classes of problems our attitude is we'll do them all day long for the kinds of problems that we care about and we are fine running six kinds of models instead of running one model that is going to conquer them all and so I do feel like for a lot of like known problems um model size is not really going to be an issue and there's going to be an ongoing reduction both in the size and therefore the cost to serve them Satya of course might be referring to the margin that apple pays out and if either them I would offer you know Apple 100 of rapture in order to get at that traffic it's a way to establish a beachhead by the way there's precedence um Google gave more than 100 to AOL and close to 100 to Yahoo in its early years um that's how you make markets they obviously will be trying everything you're saying that we should expect these players or that it'd be rational to play even more aggressively from an economics perspective than um than we've seen so far oh absolutely absolutely you know part of the problem with um with Bing's growth has been that Google has fought it off very effectively on the business side um of course it hasn't helped that it is common perception true or deserved or not is a different story um that Bing search quality is not as good as Google for what it's worth there are very few people on the planet that can objectively judge search engine quality and so they need a way to break through and establish meaningful meaningful presence and so it is perfectly rational for them to start with you know a better product but then um go out of their way to establish a beachhead establish a market because that is going to pay off in a pretty big way for them down the line every part of this game feels like an expensive game to play and I want to ask you about just the building of search even aside from training LMS I remember there was a lot of skepticism when Eva first started including from yourself about how any startup could afford to build a new search engine from both an engineering Talent just ambition of technical project infrastructure cost perspective you've built an All-Star team but obviously can't spend a billion dollars as a startup can you talk a little bit about what's been most challenging to build yeah search is one of these things where you need a fair amount of scale before you have any kind of meaningful product um you know with like an ad system for example I can tell you how to build uh one with a three person team because it's like limited data or if you're building a new mail client it's a it's a small problem yes you'll have scale problems but only after you have a million users not on like day one search like setting up like a new mobile network let's say where you have to start from scratch is problematic from the that perspective simply because you have to do a lot of work to be seen as even vaguely competitive and so everything from how we went about doing our crawl to how we built our index um has been a struggle I I you know I want I I won't deny it and it's one of these problems where like you know grown men and women uh same ones will just run away after a while they'll be like they work on it for three months maybe I can't deal with this I just need to like go um and it's disconcerting to you know kind of watch that but having said that you know we do have an amazing team uh awesome for example um but just brilliant at engineering A system that ran completely on flash um in which we could do things like super rapid iteration replace the entire index um or the space of two days um or put in arbitrary amounts of information for experimentation in a much more flexible way problems that to Google like 15 years to solve we had solved out of the gate simply because he had run into many many of these problems but also opportunistic uh you know to the point of llms being these Universal input output machines we realize that a lot of problems that Google solved with massive scale and user data they could in fact solve with llms so we use a lot of them for things like query rewriting uh similarly extracting structured information turns out it's whether but people will ask about whether in like many wondrous ways we are in the process of actually replacing a hardcore system with one that's based on an llms to extract to extract structure so you have taken shortcuts wherever we can in order to uh in order to do this it is a daunting problem but I'll tell you the single biggest positive thing for the team is actually launching answers because up until then they sort of had this feeling of even if we were to be as good if not better than Google no one will care people can't tell between like you know list of links anyway Once you turn that into and yes here is like an actual answer that my mom can take a look at and say way better than a bunch of links all of a sudden there's excitement um and so there's the actual psychology all of you deal with teams of what excites the team um and really it's it's it's been over the past few quarters where people have realized oh wait this can be a transformational experience that just is like a big jolt of electricity through everybody just in terms of how excited they are how hard they work and things like that yeah it's very exciting progress I guess one one question related to that is when you look at distribution uh because you mentioned you know consumer habits are quite sticky on the distribution side and I remember even back when I was at Google many years ago like over a decade ago probably more than that now 15 years ago or something hundreds of millions of dollars a year are being spent on distribution and obviously that number's grown with the Apple deal and other things and so do you view it as like distribution through Superior product is it specific Integrations or Partnerships or how do you think about getting that consumer interest uh distribution is hard there's just no question about it habits are hard to change um you can dislodge some of this with uh with the superior product you can dislodge some of it with the dollars part of the reason why we released this app called gist which was a very different take on search is we very deliberately said if we wanted search to look like Instagram stories what should it look like it's an experiment you know we hope it'll do well and so sometimes you have to um look for change at sort of the locus of change uh the other thing that we are also actively looking at is uh you know in this moment where there's going to be enormous amounts of uncertainty about things like because search engine traffic basically going to disappear for websites um or you know or llms going to disrupt the aggregator uh publisher relationship in a fundamental way we are now realizing that we can offer a superior search experience to lots of Publishers whether it's a Reddit or a boston.com or anyone else we can give them conversational search on their Corpus so we are going to try a set of different things we've actually had a fair number of success working with privacy products like Dashlane and obviously other folks that we are talking to like protonmail about how we could work better together distribution continues to be like easily my top worry for how does Neva get skilled yeah and I guess related to distribution and business model you opted for a privacy Centric Subscription Service without ads quite early and I think at the time that was very Innovative thinking right I think now that other products chat GPT Etc all sort of coming out with the subscription-based approaches I was just sort of curious how you thought about it like when do you think a product should be supported by subscriptions when should it be supported by ads and how do you think about it in the context of this type of product I mean for us it was a way to stand out it was to give us a clear Runway um thoughtfully done uh as monetization is an incredible Juggernaut as everyone that's on this podcast knows in terms of the kinds of scale that it can you know that it can bring and how it can disconnect uh monetization from the product so it's almost like a separate team that is working on it you know when it's very successful it can actually kind of get annoying uh I'm sure like none of us likes watching broadcast TV anymore like Sports Broadcast drive me crazy when I think about like how many ads that I have to sit through uh ads sort of calm uh you know with elements of self-destruction built in it's part for the course when you're doing it it's always attractive to do things like show more ads um in some ways you know hybrid approaches of starting ads free and maybe using ads as an additional mechanism um might be more sustainable even though you know reasonable people will argue that most people that come to ads later tend to be even more discriminate um about how many ads they show and ads quality than the people that I've been working on it for the first time you know I worked on it it's also the team but um Google search ads actually uh tried very hard um to hang on to Quality bars uh to hang on to user metrics for a very very long time compare them to somebody like Amazon today I find Amazon search experience a joke um because it is so full of ads and is actually misleading ads or it's really hard to find what is going on I think they're viable options there are structural elements that then come into what should you order if you're in the business of providing answers like chat GPT was um ads just becomes a whole lot harder to do there really you're just uh you know you're betting on the quality of answers but for many other products that are about more casual consumption whether it's social media or even better search might go I think it's uh it's it's it's an open question where ultimately it'll settle I point to point out to people that something like a gist experience which is a summary followed by a series of cards uh you can stick ads in there you're not planning to do that um but there are many different ways to solve problems you know in the early days of Google one of the arguments are being made for ads was that the signal in terms of willingness to pay um was a way to actually boost a meaningful link to somebody in other words if there was somebody who's willing to promote a link that in and of itself was a signal on the potential quality of that link relative to the potential user do you think that's a true statement or do you think um it used to be or you know is is sort of our Commerce signals like good boosts for actual ranking they can be uh but I think the bigger truth is that smart people will come up with great explanations for everything that they do as long as it's convenient to them the best religion to have are the ones that are aligned with your business interests um and of course the ads team is going to say that there's some amount of Truth in it but that clearly is not an explanation for like two screenfuls of ads when you're searching on your phone I find this whole thing of ads enable Google to make free products or ads enable Facebook to be available for Ecuadorian people made by billionaires sitting in Palo Alto to be entirely self-serving my attitude is like yup we can make money with ads it works pretty well we're rich it's okay if we uh just sort of project out a little bit and say um these summaries cited or not uh chat bot experiences um answers are really compelling to Consumers how do you see the relationship between search and content producers changing in the long term right if these summaries take traffic from Publishers do we lose the incentive to publish content on the internet I think that's one of the big unknowns um I think what is uh going to happen um is that some of the larger content creators you know I would put people like Reddit and quota these are some of the Forward Thinking ones very much in that bucket they're going to say we want to be part of search but we don't really want to be part of your answers like you know taking our data and sticking into llms is not really Allowed by our crawl policy but smaller Publishers are not really going to be able to do this uh the bigger ones are going to have things like their own chatbots so that you can browse Reddit content or quora content so I liken the current moment um to you know basically we're going to be dropping um a bomb or a giant impulse uh to the center of how a lot of us get at information this is going to radiate out from here to a whole bunch of sites to the content ecosystem I think it's going to be a little bit it's a little hard in my mind to predict it does feel uh like there might be more centralization or more consolidation when it comes to content creation your average small blog which could subsidize itself or which could monetize itself with advertising is going to is going to find it hard to compete in this answer World especially if the expected experience for everybody is going to be I don't really want to read giant pages um I want to be like talking to you give me give me a bit of a summary of what you're going to say then I'll ask follow-up questions um all those experiences are possible but not for every blog that there is so I think that is potentially a very different platform that is going to evolve for how content is going to be created that looks a little bit different from how it is today you know when you were at Google um your team was doing machine learning and AI at a scale that I think roughly didn't exist anywhere else and you were very forward thinking in terms of then applying really interesting Cutting Edge Technologies at Neva and creating one of the really first and most interesting llm based uh search engines right which I think is super exciting work what I'll say you're predicting gets most disrupted within the AI World Beyond search or what are some areas that you think are coming over the next coming years I mean we talked about content how it's going to get you know how it's going to get disrupted I'm not even talking about synthetic content yes it will be but I think there will be techniques that's a cat and mouse game of detecting it but obvious places where content is generated actually ironically is going to be advertising I can see how personalized advertising actually plays a a pretty big role especially when it gets to be multi-modal I joke to people that like Michael Jordan is going to be telling you uh to buy his Air Jordans like you know look at you and I and speak your name and and so on and so forth so advertising with its closed loop for optimization and the Relentless focus on efficiency actually is a natural area I'm not saying there's not going to be um but obviously there are a lot of companies that are saying things like oh we can apply llm technology to every other information function whether it's mail or how we consume documents um but what I find you know interesting is that we have a set of incumbent technology companies that are actually very smart and very driven uh you know Microsoft to be this Innovative this late into the game you don't hear about stuff like that from IBM not at this scale of like consumers and the whole world so I think they're all going to react pretty quickly incorporate a lot of it so I don't know how like how much there is going to be pure SAS Innovation on products that we take for granted I'm not saying there's not going to be but it's a little bit harder one of the areas I'm personally very excited about is the generalization concept that I spoke about earlier um which is if you think of llms as like machine language um then the natural thing is how do you combine them with the various tools that we use um in terms of search engines calculators apis programs other websites um so I think like action Transformers is going to be an incredibly powerful area the technology is very nascent so unlike say you know open ai's ability to crank out new generations of llms I don't think that Tech is yet at a point where people can build lots of applications on top of it but to me that is potentially a big breakthrough not just for things like rpas but also potentially for hey can you create an AI SRE can you create an AI code reviewer can you create like fill in the blank um I think that's incredibly exciting but I think the technology is also quite a bit more nascent than what we have just come to expect will happen with language models yeah the agendization of the world is a very exciting future um so well well I'll wait out with baited breath um as we wrap up is there anything else you'd like to talk about that we didn't touch on uh you know it's it's right it is repeated um but as a technologist uh this is a really exciting moment uh where I do think that this is powerful new technology it's also getting democratized very rapidly my take is that uh you know WhatsApp was the seminal moment of like mobile Computing here a team of 30 people could create a product for the whole world um to me that represented the power of mobile platforms and uh if two years from now if uh whatever three college kids you know uh 20 years old are able to build a brand new application that uses the things that we know for sure whether it's web servers or databases but also language models in a fundamental way and say like wow we never thought of that um you know that feels very possible that is what is really um exciting about where we are um yeah in in the meanwhile super excited for where we are able to take search with Neva and appreciate all your wisdom and support I'm counting on that to happen actually um but but I think a lot of the things it will too sweetheart uh incredible conversation as always thank you for joining us on the podcast we appreciate it thank you Sarah thank you Elon thanks foreign