No Priors Ep. 49 | With Shopify VP of Core Product Glen Coates

Key Takeaways

hi listeners and welcome to another episode of no priors today we're joined by Glenn coats the VP of product at Shopify where he leads the core developer platform including all of their AI products that we'll get into today before he was a leader at Shopify Glenn was the founder of handshake a B2B e-commerce platform that was acquired by Shopify in 2019 we're excited to talk about how AI is changing e-commerce and Entrepreneurship as well as innov at scale and leading at Shopify welcome Glen thanks for having me so lots of fun stuff today let's definitely cover your personal story quickly since you're also a former founder one of several at Shopify now can you give us some of your background I I was a compsite grad I spent the early part of my career in video games and then I had a weird left step in my career where I moved to San Diego in 2008 to run the US operations of an eco-friendly Shopping Bag Company you know like you take your own bags to the store that whole thing anyway this company did a whole bunch of wholesale they sold a lot of bags through stores and through that I ended up um getting to the idea for handshake and handshake was a initially a very like sales rep focused B2B e-commerce product but then became both for sales reps and for customers to buy online so handshake eventually became basically a B2B only version of Shopify like a wholesale only version of Shopify so it's not hard to join the dots from there to after you know building handshake for better part of nine years here in New York the opportunity came up to uh join forces with shopy and then I've been at shopy for almost F it'll be five years in May I started out very much focused on on B2B and wholesale which was obviously the point of the acquisition I spent about a year doing that I then moved on to focus uh I I ran a code red on checkout in 201 20 which is the first year of the pandemic and then since the end of that um I've been leading the kind of core product group at shopy which is um you could think of it as all of the built-ins of Shopify the online store the checkout the back office the developer platform the App Store um uh and yeah that's that's what I do at shopy lots of lots of good stuff what's a code red look like a code red is when Toby sends an email to the entire company saying this thing is the number one priority and he means it from an actual operational perspective what that really means is if the team that's working on this code red asks you to help please drop what you're doing and help this is the number one priority so that's how they work but usually a code red is a symptom of some other much more systemic problem that's gotten you to that point at least in my case checkout code read in 2020 was hey the check checkout is failing in I don't know three four five different ways and the checkout is a pretty important part of shopy obviously so let's go fix that so we me and you know somewhere between I think it at its peak it was probably two 300 people working on various parts of the problem so we all like scrambled for a year and like did what it took to um fix those issues but then at the end of that year Toby took a step back and said okay well why did that happen like how did we get to the point where those problems even happen in the first place and then that led to some of the reorganization of the company around less like there used to be 12 to 15 of these kind of fairly small fractured business units and now there is actually only like three or four which is actually truer to what the product is but of course each of those units is bigger than the ones that were before so it actually requires you to lead in a slightly different way once you start grouping that many much of the product and the people together I guess like two related questions from your background I guess number one is since you were acquired into Shopify one of the things I've been very impressed with by the company is a degree to which it's been able to retain and grow the roles of Founders that it's acquired because as far as I understand it the majority of the management team at this point was acquired in at some point or at least a very large fraction and then secondly as you buy a lot of things you end up with a lot of disparate products disparate platforms disparate Integrations and all sorts of things can right or wrong based on that and so I was curious a little bit about the broader concept of integration number one from a team and individual perspective as a Founder coming in and why is it such a good home for Founders and then number two like once you're acquire in the the tech and the the product how do you integrate that as well yeah it's really notable so like when I look at the so the the the sort of exact team of shop fight contains a number of Founders some of whom were acquired in and some some of whom just were Founders and when I I look at like my management team the the people who report to me who each of these sort of nine or 10 people run R&D orgs that are like a couple hundred people each and almost like a very high percentage of of those people are also Founders right and so you ask okay well why are all these people ending up essentially running the main parts of the product of Shopify and Shopify is a product first company so that's sort of the same thing as saying running Shopify I think a lot of this comes from uh Toby himself who has extremely strong opinions and has I think learned for the better to no longer be shy about like saying I want to aim in this direction right I think there was like a big culture of Bottoms Up decision- making power to the edges like whatever you want to call that School of Management philosophy which is like you know delegate and Empower I think a lot of former Founders are actually coming back to the place where they say hey what makes what what am I uniquely good at I'm uniquely good at like having a point of view being willing to smash my head against a wall and like do whatever it takes to manifest that point of view in the world through building things building teams you know saying the hard thing at the right moment and I think Toby has kind of re-embraced that right in the past few years of saying like hey if I'm good at one thing it's being extremely opinionated and it seems that in this domain at least for Toby in this domain of like the internet and commerce I'm right more often than I'm not like I have a higher batting average than usual therefore if I can help accelerate this company through like being decisive that's going to be good and so he sort of sets the example and a lot of us you know see that and we're like hey we we remember being Founders we remember what it's like to have all bucks stop with you and if there's a way that we can apply that level of decisiveness here and accelerate teams through like the otherwise like you know designed by committee [ __ ] that kills a lot of companies then like that's a net positive right the other piece of of buying in companies is sort of integrating different systems and platforms and often you see people do one of two things they either let things run independently that was what's app for a long time at meta for example or you integrate in all the infrastructure and usually that makes things more performant you can have features that cross over Etc but the flip side of it is sometimes you see these giant projects that just stop a company from functioning right like at Google there was a famous sort of identity layer that was built and for a year and a half people just didn't ship things right uh in other areas and so how do you all think about integrating and acquired companies and acquire Technologies but more generally as you build and build and build and build there's a b a big munge of stuff how do you sort of consolidate that down in a performant way that's actually Speedy to execute and then I guess later on maybe we could talk about whether AI has played a role in that or not yeah it's it's a really good question it the answer is it depends like most problems usually when you encounter these problems you sort of have to look at the stack you usually don't want duplicates at the same layer of the stack right um and the funny thing about this is sometimes you actually don't notice the two things are in fact at the same layer of the stack until much later on right so as an example inside Shopify there is a uh there's the checkout which takes you know someone's shopping cart and says okay let's take the business rules of the store and convert that to an agreement for a sale taxes and duties and shipping and all that stuff so that's one engine and then at different points in shi's history other things that have been built that are sort of like adjacent to that but not quite the same thing right so like a thing for creating like an invoice or a thing for like even editing an order after it's been placed and a lot of these things kind of get built cuz because you start with a small problem example I just need to be able to add a discount to an order after it's already been placed because you know someone calls up and says hey I forgot to put the discount code on can you please enter it right and so someone starts a project over here which is oh we're just going to support editing discounts onto orders and you don't realize that what you're actually doing is creating a a second version of the negotiation engine right it's just that you started with a very very small part of it and then that thing kind of grows for a few years and you're like oh it's not just editing discounts in it's also editing the shipping cost it's also editing another product onto the order that they forgot to add and before you realize it you're like oh my God we're Reinventing the checkout over here right and by the way in the metime all the teams that work at Shopify are uh finding themselves having to build to support both of these things right so someone builds a new feature and they're like I'm going to add I don't know International pricing and then that team is like oh [ __ ] I have to build it into the checkout but I also have to build it into that order editing thing oh my God and I also have to build it into that invoicing thing and now it's like to your point it's like okay now I every the thing that should have taken a month to build is now taking six months to build because the layer of the stack that's underneath has five things where there should just be one right and so sometimes it's hard to know what to do right because sometimes the right answer is let's collapse those five things in the stack into a single thing which is it sounds like what Google tried to do but then you're like oh my God it's like trying to take the foundations of a house and like rearrange them while the house is still on top and like that's oh God this is going to be really hard because I can't just make everyone move out of the house right the real answer to how do you solve this problem is one is is um developing extremely good intuitions for when you are accidentally creating duplicates at the same layer of the stack so you just never get into the problem in the first place and then the second part is being willing to actually notice when you have created the duplication and eat the vegetables of saying like oh [ __ ] whoops we did it okay now it's time to eat the vegetables and collapse these things together because if we don't every team that exists above Us in the stack is going to pay this duplication cost like in perpetuity and the problem is it's not even just your R&D teams that pay the cost the saddest thing is that usually it's a combination of your R&D teams and your customers paying the cost because the reality is that they you not everyone will do the thing of building the five versions they'll probably build like the top two or three most important ones and then all the customers who depend on numbers four and five will go to use the thing and they be like oh this is super weird this thing doesn't work why doesn't why is this thing so different to the other thing I thought they were the same thing and then they look under the covers and they're like oh I know why it's different because actually this part of the stack is all [ __ ] up have have you approached that from an AI perspective so I think like um Shopify was a very very early adopter of AI and in general whenever there's like a really sharp technical founder driving a company they've been an earlier adopter right and Toby obviously fits that mold but it's an area where I think you kind of have to play around with it experiment Etc to really understand the cap ability set but at the same time to your point you want kind of centralized infrastructure approaches so have you thought about how to do that adoption as well as um how how have you thought about that in the context of launches like sidekick and Shopify magic and things like that I mean yes this is obviously the space where everything is moving the fastest right like the world of invoicing is not moving at the pace of the world of AI right um depends on what you're invoicing but yeah um yeah it's it's been an extremely interesting last 12 months of like looking at where can we apply different models to parts of Shopify I think the high level idea is you know shopify's real focus and real mission is enabling entrepreneurs right and the meta strategy of everything we do is how do you simplify the process of starting and running a business so that more people who would otherwise not get over the hump get over the hump and have an opportunity to try building whatever that business is and like the historical strategy for that Shire was build really easy to use software use software that is super simple when you start and it kind of reveals complexity as you need it like as your business grows and you have harder problems like the tools sort of like appear at the right time I mean the world is littered with the corpses of you know know the companies that tried to serve SB and Enterprise at the same time and this is kind of the hard problem of Shopify how youd be simple but also powerful and scale up AI has this really interesting moment where it's like okay well shopi built its position in the market largely by being the easiest to use thing as you build a business in scale but in imperative mode right it's the I'm going to give you all these switches and we're going to give you the best switches and we're going to show you the switches at the right moment time but these are this is the best switches right and AI brings us into the world of like okay well what if we can um give you the the driver in the car who knows how all these switches work and can help you along without having to learn all the switches it's probably one of the most powerful opportunities for more people who would otherwise be daunted by having to learn the switches even though we try to make the switches as good as possible there's all these people who can't quite figure out the switches but they could talk to the they could talk to the co-pilot and they could explain what switches they want hit but they just can't do it themselves and so um I think we have an amazing opportunity to increase like the amount of entrepreneurs in the world who actually get a shot right and so that's the kind of headline and then from there you go into okay well which models do we apply to which problems you know you look at individual problems like okay how many people get stumped just writing the descriptions of their products cuz they're like they're not good copywriters and they're like oh this is embarrassing I can't write a good description for my candle I'm not going to launch my store you know and there's so many instances like this where it's like okay well can you give someone that extra bit of skill that gets them over the hump where they're ready to hit the Big Green launch button you know yeah that's really cool Glenn it's it's really exciting how many obvious places there are to apply that to make it easier for the merchant or the wouldbe merchant but I think one of the things that has struck me that that you and Toby and the team has done is actually shipped a lot of that quickly when a lot of very large organizations are like oh that makes sense the outputs are non-deterministic um by Nature we have a process for measuring and evaluating quality of our products generally that process does not apply like how did you get to this is good enough and we can ship it well well one of the principles that we uh currently hold this might change in the future if confidence intervals go high enough but one of the principles that we have for all of the shop Five magic features today is that they're allowed to propose changes but not commit changes right so they can like generate text but they're not going to save it without you actually reading in and saving it we might suggest a reply for the user that's you know someone writes in like hey what's your shipping policy we can like suggest the reply based on us knowing what's in the store and like running that through an llm but you have to hit enter right right and so one of the things that like human is in the loop essentially right now is one of the um one of the ways that we're kind of mitigating risk here and obviously human in the loop is great because it gives you the feedback cycle you actually get a three-part signal you get which suggestions are accepted clean which suggestions were accepted but then with minor edits and then which suggestions were outright rejected and that's an amazing Loop to be able to improve things through so we're always getting better but that sort of human in the loop human must clicks save thing is a is a big part of the strategy do you think that Shabah High has a different like Risk orientation than other companies of its size you still need guard rails and even if they're suggestions like it's not it's not perfect right it correct yeah I think Toby Toby and Cass myself were all like fairly risk hungry people I mean I think that's just a little bit of founder culture is like you want to take risks like you don't really want to be safe you get bored you get annoyed when things get too safe but there's always this balance of like we like taking risks that are risks to us we don't like taking risks that could hurt like break someone's business and so if there's a way to do a thing that's like very risky for us but but is the risk is mitigated for the actual Merchant like we're very for that and I think that's where that kind of human in the loop human must click save thing is is part of that right and who knows like maybe a year from now maybe two years from now like we get to like you know hey you're you seem to be accepting our suggestions without edits 99% of the time do you want to just go into like full auto mode like maybe we get there eventually we're not there yet but like you can sort of see that the hill climbing will eventually take us somewhere close to that right can you just describe for listeners um some of the stuff that you think is coolest at additions if that's um you know image gen or semantic search or or even any of the foundations work the the foundations work is the thing that I'm most excited about but I'm a Commerce nerd so like this is going to seem like very boring to the audience but like just humor me for a sec it's a pretty nerdy audience so it depends yeah the data model that is the most most most Central to Shopify is the products data model right like how do you represent the products that you're selling and shopy has for you know basically like 10 15 years had the product data model hasn't changed that much it's like fairly simplistic It's actually an amazing Testament to how big of a company and how big of a you know the fact that you can get to like 10% of all us e-commerce on like a fairly simplistic data model is actually kind of amazing but you know like shopy has um been a little bit on the weak side for dealing with like very large and complex products so products that have a lot of options a lot of colors a lot of sizes when it gets complex we don't do as well and so we're updating our data model with support for just a much larger number of variants and again this is the nerdy part this is one of these crazy things about tech the reason this is so hard is because it forces you to go from unpaginated to paginated apis when you start making the variant counts very large and it's just a breaking it's just a breaking change for like the entire app ecosystem and it's just breaking changes are hard right so I'm really excited about just the fact that we're unlocking that data model we've actually also embedded in um not just like hey you can have more variance and more options but there's also now a standard product taxonomy that comes with standard categories and standard attributes so that like when a merchant creates like um a t-shirt we're going to Auto recognize okay this is in the standard category t-shirts and that category comes with all of these standard attributes and we're going to use AI to one detect what category it should be in two detect what the values should be so we're going to order to we're going to try and guess okay seems like the colors here are green and yellow yellow based on the images you uploaded seems like it's made out of cotton based on the description you had and the great thing about that is when that Merchant takes those products out to their own storefront all of that data is going to make the search experience better but more importantly when they take those products out to like Google and Facebook and Amazon and all those places having that structured metadata around their products makes them way way way more discoverable which is going to lead to them ranking higher in search results and basically making more money so this is one of those weird things where like get the data model rate and having the data quality be very high actually causes these very like important effects up funnel for the ACT for the business so that's really cool um uh we are also uh releasing um the first version of our uh image uh image editing with AI thing um so you can take product images you can replace backgrounds basically you can take uh uh product imagery and get to a very very very professional uh standard uh on it without having to go through like very expensive photo shoots basically a lot of what we do is like how do you help someone who's like actually just you know my mom at home trying to start a business how do you help that person look like as professional as like the world's largest companies and then um yeah what else am I really excited about in the addition I think you just mentioned something Sarah now I'm having a mental blank uh I mean I just have like I'm a Search nerd and so I think Rel yeah so here's a here's the crazy thing so I just learned what LBD means like a little black dress exactly I didn't know what LBD meant until recently um but uh very soon shopy stores will actually understand the meanings of things as you so today shopy search is very very literal keyword searchy there's a video of me going around internally being like infuriated when I go to there's a store that I actually buy things from and I went on there and searched for um LBD not LBD I searched for sweater and because that store uh only lists their thing as sweatshirt I got like zero search results I was like I can't this cannot still be true in 2023 like this is insane that this is happening so that's a really easy example but like you know if you go to a store now and they haven't put little black dress in any of their keywords if you search for LBD it'll actually do the right thing you can even do things like um I saw a crazy example from the team yesterday they were like um Christmas themed shoes and it would actually correctly go and find all the red and green shoes in the store so that's kind of an edge case example but it's like you know searching for things like um something to wear to a wedding something to wear to the beach will actually do the right thing whereas before would only would never do the right thing unless those keywords happen to be in the product descriptions which will never happen so that's I think again that's one of those things where it's like how can this search experience on someone's tiny little storefront actually be as smart as like what you would get on Google that's really cool and are you doing like rag or embeddings or specific technical approaches like to that or yeah yeah yeah so a lot of what we have to do is figure out what the right embeddings are to use how to fine-tune the models like obviously there's categories in Shopify that are um more Pro like apparel fashion gift home wees um so spending the time to um assemble the data sets and do the uh fine-tunings and pick the right embedding models that are best for this category but and it's also been really interesting with like especially some of the multimodal models that have emerged in the last I don't know 2 three months um experimenting with how much um how much weight to give like the text descriptions versus the images versus the um taxonomy attributes and figuring out what what mix of those things generates the best results has I mean we're still working on it honestly but it's a pretty cool and exciting space that's really cool yeah cuz I guess like uh a lot of people increasingly I feel are adopting gbtv or the vision model which allows you to upload images or understand or interrogate them and so it seems like there's a lot of information just resident in like visual imagery that people aren't really making use of that now you can translate into a textual understanding and therefore tie that into everything else it's associated with the mer Merchant that sounds very exciting and and it's it's an amazing thing for the buyer experience I mean like to your point like the buyer experience is really cool when you search for things to wear to a wedding and it does the right thing but actually think about it from a motion experience point of view if you're like hey I've been running a store for 5 years and I've got like 10,000 products in there and now you want me to go back and backfill like 10,000 products worth of five attributes for product like Jesus Christ being able to actually apply the models to do something that turns out to be like a 90% correct guess is an amazing way to help bootstrap people into like the current moment you know what do you think it's still missing from a capabilities perspective to substantiate your perfect AI world for Shopify well look I mean I think the the exciting and frustrating thing about llms right now is that you can get an llm like agent thing like a sidekick esque thing you can get it to like 75% in like 10 minutes and then it's like this brutal Hill Climb to get it to like 95% like over time right it's just like it's like catnip for like hackers cuz they're like oh my God I just in 10 minutes I got to a thing that's like pretty good and then you're like yeah yeah wait buddy wait wait for the next bit the next bit's real interesting you know yeah the last 5% I think is the next five years of tech or something right right right right and and the irony is is like you know people talk about they're like oh yeah this lm's hallucinating right now and you're like yeah yeah you know that's its job right you know that every time it emits a token it's hallucinating right it's just that you like some of the hallucinations more than the others right um and so I think that's that's been the challenge for us like it's even in the name right like Sidekick is is supposed to feel like your companion on the journey like your coach your assistant your the person who's maybe seen seen this movie before a little bit getting it from 75 % to 95% is is the process we're working through and it's and it's so exciting because you know we're doing stuff on our side where we improve the training sets we you know we're building more and more conversations with feedback from users that help us dial it in more but then every 5 Seconds you look around and like there's a new model that you can take all your training set and all your evals and then it might be another step function so it's kind of innovation really rapidly happening both inside and outside our our R&D team and you really know you never know what we each week is going to be bring really you know and you can be pretty optimistic in that um like especially if you look at any one of these problems I think there's a lot of focus on the core model capability and there should be but like you're like oh well embeddings models are getting better right and um and people are working on like synthetic data generation tools and working on tooling for the entire rag Pipeline and so I I think it's it's yes it's like you hit the the pain of reality of like oh no I have to go beyond the demo but a lot of other people are um working on some enabling stuff so I still expected a lot of the experiences to get better fast counting on it from Shopify yes yeah yeah well I mean we're we're at The Cutting Edge you know yeah is there anything external to Shopify um that you think is especially interesting right now in the AI World be it's startups or things that people are working on or projects or this is literally every single person has probably said this but I think the like the r thing at CES was pretty interesting um I mean I'm I'm literally wearing the shirt right now like I'm a teenage engineering nerd so like I was I was hyped on the hardware but I think one of the interesting problems with like llm based applications and agents in particular is like what's the actual interface they're interacting with right like in the case of sidekick right there's a couple different places like like what is sidekick using right is sidekick actually using the admin API under the hood or is sidekick actually reaching into the pixels of the web app and clicking around in the web app and doing stuff right like that's a pretty important question like what is the interface that the agent is actually learning and interacting with and um I thought the really interesting part of the rabbit presentation was that they decided to treat the actual goey as the interface and to try and like have a model that became very good at interacting with essentially web apps if it actually works the Strategic Brilliance of that is they instantly have the world's largest app store because the world's largest app store is just the web right yeah um now who who knows if it actually work but it's it's an incredibly interesting strategy I was talking to Jesse from rabbit about this and I was like it's a very it's a controversial interesting strategic decision to say like I'm going to interact with the applications themselves eles versus use apis and his last company tried to unify actions on apis and his view was very much um like well their implications on the uh like as you said reach and ability to use different types of data and like whether or not you own your destiny if you're relying on those interfaces and so as you said if it works that's super interesting yeah and it's even philosophically and from first principles it sort of rings true to me right because you know a lot of these experiences are especially with like when I actually think about the sidekick UI the side sidekick actually runs sidecar in the admin next to you right even the name Sidekick is like hey I'm here with you you know we're both here together and we're both using this Shopify thing to try and build your business right and so the idea that sidekick would literally be interacting with the pixels in the same way that you are it's sort of almost like smells right in a weird way um and I mean obviously the nice thing about it is as long as the model can actually learn the UI then every time you ship updates to the UI if the model's smart enough like it's just like oh yeah my smart Budd keeps also can use this new button that just appeared today right so yeah I think it's an interesting approach we'll wait and see what actually happens I also know people in the industry who are like yeah yeah it's a cool idea it's going to be insanely hard to pull off but who knows again this thing changes every week the other thing I think is kind of interesting which is actually like Shopify relevant is I I guess I think I'm curious what you guys think about this but like the the sort of the the triangle of like perplexity Google Chat GPT of like it's basically like llm native search and like chat GPT really good at the llm interface thing not amazing at search yet Google amazing at search not really figured out the llm thing yet perplexity is kind of squarely in the middle and saying like we think llm Native search is a really interesting like Hill in the middle of this thing I'm sort of watching this from the sidelines and kind of interested to see where it plays out I think embedded in that strategic tussle is the question of what do people actually want from search do they want hard facts do they want opinions how much lossess are they willing to tolerate in order to get the compression of the llm took 10 search results and summarized them for me but maybe they hallucinated the summarization right it's it's a really interesting place like what do people actually want you know yeah I think it's a really interesting open question and related to that too you start I've started to see um the degradation of some performance of some of really early players in the market where there's a lot of qualifications or safety or you start to see the llm go out to try and gather information and it just really in some cases either slows down or makes the information worse because to some extent the reason you're interrogating an llm is you're you want to get to an answer you don't want to web search or you don't want that you know poor synthesis of bad data from the web you want the good synthesis of bad data from the web and so yeah it's been fascinating to watch and to your point I think a lot of people um have started to adopt perplexity for that specific use case because that has that middle ground of some form of ir plus l in a traditional sense so yeah I agree with you it's it's it's going to be fascinating to watch how all the directions this goes and I think the reality is people almost forgot that people from search they actually want an answer they don't they don't want to do a search process they want to get to a result in many cases or a list of results it's almost like people forgot the user need in some sense yeah and it it's funny when like I mean some searches are so specific but some are almost like shared knowledge of humanity so it's like one one thing I found myself doing a lot like a year ago um when I was looking at okay how can we improve search at Shopify both on the storefronts but also on the shop app and just kind of playing with this idea one of the tests that I found myself running over and over again so at the time I actually had this happen I had a a friend's uh daughter was her birthday and I was like she was I think she was six at the time and I was like I want to buy a great class classic book for a six-year-old girl and as an experiment I would go to Google and I would like say classic children's book for six-year-old girl and like the search results you would back get back on Google shopping would just be like terrible like absolute disaster right and then you would go to Amazon type same thing in you would get like nonsense back right you went to chat GPT and you said give me 10 great children's books for a six-year-old girl what would come back would actually be amazing aming like it would be like really like the best 10 recommendations right and so I was like huh there's something here right now like chat GPT terrible if you ask it for hey I want a pair of Nike af1s in size 10.5 in a store in New York City but really really really good at the general knowledge question of like great children's books you know yeah it's funny because I worked at Google years ago and then when I was at Twitter one of the teams that worked for me initially was search and um you know you tend to segment these things as you undoubtedly do it Shopify into uh types of search right is it navigational is it certain types of information that you're looking for is something else and so you know you could imagine that in the llm world you effectively want to do almost like one boxes like you have a Google where you trigger off a certain types of keywords or phrases and therefore you end up with a different result and you should be able to especially have Integrations with search engines be able to serve the I want these specific Nikes and where should I get them right because that's almost like a form of navigational query and sense Rel to a Commerce objective versus uh just give me some knowledge so it feels to me like a lot of stuff will come as long as people don't actually think that llm has to do everything unless it can also function as a router or something so it's really fascinating to think about yeah and and then that's when you get into the world of like things that are like matters of fact and things that are matters of opinion right like is this shua Nike is not a matter of opinion right whereas is this book great for a six-year-old girl that's a matter of opinion and how do you construct search systems that can actually sit correctly at the midpoint of the world of facts and the world of opinions right I I think also going back to alad's point on um use cases like one version of this um like from a scenarios of the future perspective is it fragments right sometimes I want facts and sometimes I want opinions especially opinions like my own already I mean like date on how people consume media definitely looks like that right yes and so I think it's very easy I mean I was an investor in a search company called neeva I think very highly of the perplexity team I think it's very easy to look at the Chokehold on distribution that Google has through its Partners uh and say like that's very hard but we're at this very I think special moment where the technology and perhaps the like user Behavior the the expectation if we take out like slices of it where the expectation is very different and get people to think about pieces of search CU it's not not all clear to me it should have been one market to or it's permanently one market right you guys probably think about owning Commerce search um and so right but even within Commerce there's so many different um like the way people search for for clothes and the way they search for industrial parts are like not the same thing at all right so it's even within that world there's there's so much Nuance of the way that people Express what they're looking for Glenn awesome thank you for being here that was great thanks guys appreciate it good to see it thanks so much find us on Twitter at no prior pod subscribe to our YouTube channel if you want to see our faces follow the show on Apple podcast Spotify or wherever you listen that way you get a new episode every week and sign up for emails or find transcripts for every episode at no- pri.com