The rituals of great teams | Shishir Mehrotra, Coda, YouTube, Microsoft

Key Takeaways

i generally value the reference check over interview signals you know if i had the stack rank in interviews what is the best signal the uh reference check is the top of the list those people you know they work with this person for you know sometimes for years like their knowledge what you're going to get out of 30 minutes of artificial scenarios it's just like never going to compare with what a good reference check will give you shashir marocha is the co-founder and ceo of koda before starting koda shashir led the youtube product engineering and design teams at google where he spent over six years before that he spent six years at microsoft he's also on the board of spotify as you'll hear in this episode shashir is an incredibly deep and very first principles thinker on all kinds of topics and in this episode we cover growth strategy specifically a framework that he calls blue loops and black loops we talk about the rituals of great teams something that shashir has been passionate about and has been collecting from all of the best leaders in tech for the past two years and which will soon turn into a book we talk about eigen questions which is not a german game show he shares how he evaluates product talent and gives some really great advice on doing reference checks we go into so many other topics this is the longest episode that i've recorded yet and you'll see why shashir is so full of wisdom and we could have kept going for at least another hour and so with that i bring you shashir narotra hey casey winners what do you love about coda coda's a company that's actually near and dear to my heart because i got to work on their launch when i was at greylock but in terms of what i love about it you know i love loops and coda has some of the coolest and most useful content loops i've seen how the loop works is someone can create a coda and share it publicly for the world this can be how you create okrs run annual planning build your own map whatever every one of those codas can then be easily copied and adapted to your organization without knowing who originally even wrote it so they're embedding the sharing of best practices scaling companies into their core product and growth loops which is something i'm personally passionate about i actually use coda myself every day it's kind of the center of my writing and podcasting operation i use it for first drafts to organize my content calendar to plan each podcast episode and so many more things coda is giving listeners this podcast one thousand dollars in free credit off their first statement just go to coda dot io slash landmark that's coda dot io slash lenny hey ashley head of marketing in flat file how many b2b sas companies would you estimate need to import csv files from their customers at least 40 and how many of them screw that up and what happens when they do well based on our data about a third of people will consider switching to another company after just one bad experience during onboarding so if your csv importer doesn't work right which is super common considering customer files are chock full of unexpected data and formatting they'll leave i am zero percent surprised to hear that i've consistently seen that improving onboarding is one of the highest leverage opportunities for both sign-up conversion and increasing long-term retention getting people to your aha moment more quickly and reliably is so incredibly important totally it's incredible to see how our customers like square spotify and zora are able to grow their businesses on top of flat file it's because flawless data onboarding acts like a catalyst to get them and their customers where they need to go faster if you'd like to learn more or get started check out flat file at flatfile.com lenny shashir welcome to the podcast thank you for having me i don't think i've actually shared this with you but you're actually the very first ceo that i've had on this podcast i actually have a rule of no founders or ceos on the podcast at least at this point and you're the first person that i've let break this rule and so how does that feel i've always been a rule breaker perfect yeah so bio for listeners just briefly uh so you're the founder and ceo of coda you spent six years at google where you're a vp of product and engineering for youtube basically leading the youtube product team spent six years at microsoft here on spotify's board of directors you're also a prolific online writer and that leads to my first question which is i hear that at coda there's a contest internally for people who actually so maybe a little context you encourage people to write a lot of stuff externally within coda you want people to be writing and you have this contest of who gets the most views and likes of people internally and so is that true and who's waiting but by the way yes it's it's sort of true we did a similar thing at uh at youtube and we and youtube creators i mean obviously kicked our butts but we had a good way to make sure we understood our tools and learned how it worked and you know i think for a while at youtube i had one of the top videos it was a really cute video of my daughter taking everybody's words when she's like three or four years old and it's not fair yeah super cute kid is uh is it easy trick for youtube but i get to learn all the tools and so on you know here the equivalent at coda is you can publish go to docs they show up in our in the gallery go to you know cardio you can see lots of them and um you know but at this point thousands of doc's been published from lots of different people gets millions of views and like like at youtube the most popular ones are not ours but it is sometimes helpful for us to make sure that we understand how the whole system works in order to do it so i think that the current wedding doc at oda is um is one in a pretty deep niche from a guy named kenny wong on our data team and it's an orange theory workout doc so it turns out that orange theory has this like deep subculture that they all hang out together on reddit and so on and this stock just took off as uh i'm probably i don't really understand the dock because i'm not in that subculture but guys it's similar to youtube in some ways it's like that the the the niches are actually much bigger than than people think you know of the ones that are you know more work-related i think lane stock on two-way write-up still outranks all of mine but uh i have a couple of good ones on there too and i would say this competition exists in my family too i don't usually even win without my own family my older daughter annika who also had that great video on youtube has a has two docs that do really well one is family quarantine olympics which is a thing she put together at the beginning of uh coveted like a fun game with families and play and the other one is a score tracker for a game called ticket to ride i don't know if you ever played the game it's like i heard it the most complicated scoring system on the planet because like you played the whole game and you spent 10 minutes scoring it and she built this whole scoring calculator and that turns out to be super popular too so anyway but my docs do okay but yeah so the kind of interesting variety of what people end up doing i love that which of your docs has been the most successful hopefully we end up talking about it i i think it goes back and forth between uh two docs i've written one called eigen questions which i think you intend to talk about so we will get after that absolutely and the other one is one i wrote a while back called four myths of bundling um that's all about how subscriptions work and kind of how i ended up on the board of spotify was daniel and i geeking out on bundling theory which is like a super weird hobby but you know i have normal hobbies too but i liked it i like to explore bundling it's a kind of fun hobby and people enjoy that doc as well yeah you've shared a lot of good thoughts on that and we're not going to cover that because you've covered that in depth in many places but just to clarify did you write that document led you to being bored on the board of spotify or is that after the fact the conversation led to so that goes it was a napkin sketch um at youtube that turned into a really fun lunch i had with daniel daniel like the spotify uh founder ceo and then he encouraged me to write it down and for me i mean i don't you write prolifically and i i have uh i have writing for me is actually surprisingly hard okay i feel like i have to think about it saying like like you make it seem really easy but the uh um you know for me like he's like can you write that down it's like great now i'm gonna like take a year to write this thing down right there because you like you like think through each part of it and you kind of come up with the right framing i have a little review process i use for my docs that that allows other people to kind of help me make it better which is which is always really helpful but yeah so that's how that started so it had got written down after yeah so we were talking about writing and content and things like that and that's a really good segue to the first thing i wanted to talk about which is black loops and blue loops which to folks that haven't heard this before might sound like some kind of ultimate fighting nightmare scenario but it's something that is really important to you and coda and so to set it up you just talk about what black loops and blue loops off yeah and it's probably worth mentioning i think a lot of businesses have a diagram that describes their ecosystem and and how it works sometimes it happens a little later in a in a company's journey for us we're probably three or four years in before uh matt hudson's uh um runs our data finance teams here he came up with this this diagram and it really stuck for people but i highly encourage drawing a diagram like this for your business i'll i'll flash it up on screen for a second and i'll describe it but this is this is what the diagram looks like um black loop blue loop and it's basically the two different ways that our product spreads the black loop is someone comes in they make a dock they share with a group of people some subset of those people turn around and make another doc and the process repeats itself over and over again the blue little bit someone comes in makes a duck and instead of sharing it with with the team or or with the collaborators they publish it to the world and in that process expose it to they can choose how it should appear what publishing encode is a lot like building a website so you pick a url you tell us whether or not google should be able to find it it's open gallery and so on and what what ends up happening from that is they turn into broad promotion of coda but really it's about that person what they're trying to get trying to get done and they'll stop sharing so i can talk a little bit about the dynamics so they the i sometimes refer to them as the the microsoft loop in the youtube loop because those are like two inspiration for it to the black loop feels a lot like how documents naturally spread like the viral actions of a of a document platform are share create share create it happens over and over and over again you know the best way you learn about office or google docs or so on if somebody shares one with you and you're like oh that's pretty cool i bet i could create one and that loop can happen very very quickly and it really drives for us a lot of how we think about how we work mostly within companies and teams but sometimes across them as well and so for example it led to our pricing model so our pricing model is um a little different than most most companies in the space that we do what they call maker billing so basically all all document products are all product of the document metaphor have have three personas people who can see things people who can change things and people who can create new things basically everybody charges for the top two they charge for editors and makers if you can make changes then you have to pay and that's like every document product you can think of including ones you know do drawing or so and they all they all kind of do the same thing and we decided that we're only going to charge for people when they make a document so the the uh you think about it you get a code of doc only you only need one you know if you're using any of our paid features you only need one paid license uh uh for doing it and the reason we do that in terms of that diagram is i wanted no friction on the share end i mean the share edge for us is like that's the moment of hey look i i'm doing this thing it's so cool like and that's the moment where the the the line i gave to the teams i want no dollar signs in the share dial going to that that's your every product has its moment of how you how you how he's for growth and like you know going back to youtube imagine you had to pay for people you shared with like nobody would ever share anything but that's how basically every productivity product works is the moment they charge you is when you share with somebody the blue loop i i often call the youtube loop because the emotions of publishing a doc are are incredibly close to that of publishing youtube video and people have all sorts of reasons why they do it i mean sometimes people do it um you know there are people who do it for money but a lot of people do it for exposure for brand building they just want to get an idea out in the world they want to get feedback some people do it for fun some people do it as a as a charitable contribution there's lots of reasons why people do it but the net effect of what happens is you know for youtube the vast majority of how people found out about youtube was through a video that was shared that's sort of the the the uh the uh impact book but it changes the dynamic that allows everyone who publishes a code like now it's a very natural incentive to go share it with the with the right population yeah if you're if you're an orange theory you share it with the orange dairy population if you're into and playing ticket to ride you share it with the right population but the um you know if you're into bundling you try to find a you know small group of people that care about one like difficult about that but the um and and what happens for us is that then becomes a loop that um means that most people's exposure and then almost a third of our users come through this this loop they're not actually exposed to coda they're exposed to a great idea for you know how to run an off site or how to how to you know how to win ticket to ride or whatever it might be and in that process they learn about the product and so then they come in uh through this vehicle and you know one one reason is very important is because what for products like ours that are very horizontal you get you know you get um you get different types of users there's some users that i call the building block thinkers they like to kind of build up from scratch and the blank surface of coda is really amazing for them but for most people that's intimidating i don't i don't really know what to do most people in the world are are are problem solvers and so they start not by do i need a new document they start with i've got a problem you know my we don't make decisions about enough in our company or you know why family can't figure out what to do on the weekend whatever it might be and then when they find a solution that problem they then pick the right tool and so the blue loop allows us uh to go after that and it kind of changes changes how the motion of the ecosystem works as well but that's what blue open back awesome i have a bunch of questions i want to ask the purses how did you how did for founder listening to this and they're like oh man what am i what are my loops what's my flywheel how do i think about my business can you talk a bit about how you kind of came upon this way of thinking about the company and then also how do you structure your teams to kind of work in this way if this is the way you're thinking about growth maybe on the first part uh i think you just hosted casey winters he's like pretty famous for talking about loops not funnels and i do think there's a a very natural thing when you're building a product building a business to think about your funnel and you think about things as being linear that somebody comes in you know they go up to your sign up process and then they they see you're on boarding and then they get exposed to the first magic moment in your product and the second magic mode and so on but the truth is doesn't really work that way i mean almost all products have some form of loot i mean that that person turns around and maybe sharing is built right in your product or maybe it's not or maybe there's a way that it happens through advocacy and and so but understanding that the way products actually grow and spread happen through some type of loop not funnel it's i think pretty fundamental so the first piece of advice i'd give is you probably do have a loop whatever the product is there's probably something about it that causes that loop and understanding how that that works really important i mean in terms of like what what it is you know in our case i'd say if you take black lip and blue loop you know the black look was you know every product in our category has that when it's on it's actually been invented you you you build a document you put a share button on it you know every product does that but sometimes it's just recognizing what's there so it's not that it's not that interesting the blue loop on the other hand is not something that every product or category has it's not really a thing that you expect to do with google docs or office or so on it's a it's kind of our unique take on hey we're going to build a publishing platform that isn't just for sharing ideas and building things with your team therefore putting things out in the world i mean one of the best compliments we hear about the koda gallery is i had this user tell me this line i really love it said that dakota gallery feels halfway between medium and app store and you can come and you can read about anything interesting in the world and you can go shopping and say i need one of those i need one of those and i need one of those and it's my view that this category we call the all-in-one doc category i think this is going to be critical because i don't think that there's enough people out there that are looking for like a horizontal new blinking cursor i mean they exist and you could get through your first million users that way but i think to get to like the level of of impact we want to have we've got to find this problem uh seeker so you probably have a loot not a funnel and it might be hiding in plain sight or it might require invention that's that's the bound or dance and the the fun that it is but but uh you know finding it writing it down i think is really helpful um there's a second party oh how do we organize that team yeah but let me ask one quick questions he said this uh i think it was a data scientist that kind of first imagined and kind of diagram this out because i'm i'm curious as you know as a founder listening they're like oh how do i how do i find some like this i imagine part of this was oh this person brought you this interesting way of thinking and there's kind of this process of oh wow this is cool let's think about this what was that like kind of just over high level that process he currently runs day in finance for us he's actually one of the early founding uh members of the team matt hudson so he's getting kind of every job here really good working team yeah he's very insightful but honestly the idea can come from anywhere i mean it's a you know it's a really famous loop diagram or for uber that i think i think one of the board members drew it or travis right build it early yeah that's right okay yeah that app can sketch you know who knows how true that is but the the uh probably lots of dispute on that um uh i think airbnb had a similar diagram right you know we tried you know yeah some way some sketches right right so they um you know it can kind of come from anywhere i do find that the most natural place to to see it is just when you're talking about your business to someone you're pitching a customer or a candidate so i actually think canada they you know i think we're gonna talk a little bit about energy but i think talking to candidates is one of the best ways to hone what your business is about because those those people are in some ways even more critical than investors i mean they're investing there they're investing their time not just their money and and so your ability to get across to them why this thing is going to be interesting and how it'll grow and they're kind of the most discerning investors out there in a lot of ways and they're actually not that easily confused by metrics and so on that could be temporary and and you know they're you know they had they sort of put themselves in that picture of like can i see that happening and so for us like the blacklight part is pretty obvious for the part like you had to like squint a little bit to think yeah will people really do that will people come and you know publish these documents like you know some hybrid between websites and blog posts and templates like what what you know what are they gonna what are they gonna do and why and and so it required a little bit of creativity which forced me to get better and better at pitching why that's going to happen and you know what what that world is going to feel like and you know this analogy of halfway between medium and an app store is like that like help people criticize crystallize what that what that promise has to feel like so i think that the additional idea can come from anywhere but if you want to mine for your own loops go look at what you told the last few candidates you talked to i like that and just take some take some attempts at drawing some kind of diagrams that's kind of how i thought about that when i think about it do you find that the quality of user is different amongst the loops i imagine one is like 80 of the growth but maybe the other is a different type of user maybe higher quality i think about a little bit with airbnb referrals drove higher quality hosts even though it was still a small portion of all hosts and so it's a really lucrative and interesting channel do you find anything like that so i mean quality and activation are a little bit different i mean they they blue loop definitely the so there's actually kind of three entry points on that diagram those people come through the blue loop because people get shared through the black loop and then people come through the top of the one like they have to be like you're you're your seed population right they have to somebody starts with blinking cursor nobody shared anything with them either a blue loop a temple to document or or black blue bait and you know the way the team is running you know if you look at activation retention so on certainly the the the highest is the the black sherlock i mean that that if you kind of somebody shares the documents that hey this how we're running the staff meeting you're kind of you're just going to use it so the uh the job of retention there is um is all different and actually one one thing that is actually going to come back to that observation second second best to see the blue loop and then the third you know the worst the hardest is activating to the very top and you know from from there you know some roughly one and five people make it to what we think of as our activation moment which is hard right it's like you're gonna hand somebody a new product that they didn't start with a problem on and nobody handed them a document to say just work in like that that's part now you got now all our nuts flows are really important and so on but if you think about these three different dynamics from how you how you ask value instructor team they're like incredibly different mindsets right because that you come into the top of that diagram we get to own the conversation like we we tell you we we have our opportunity to tell you what the products about what you should do with it here's the here's the minimal set of things you need to know in order to be productive and gradually reveal the other things that you might you know being the order of those things wrong real trouble and so on but it's actually a minority of how people get exposed to coda because in the black blue you know the person who owns that conversation the person sharing the document with you like if that person does it and like mispositions it or doesn't like just makes a crappy document or something like you know we can't we have to like help them on board their users which become our users and save thing in the blue loops you know that conversation is now owned by the publisher who's you know they they're really not that interested in teaching anybody about kodak they're mostly interested in here's this like really cool way to do orange theory or here's this there's this interesting way you know to run a meeting and and so one of the interesting things about building platforms which i think is a little bit different than than uh products that they get to be sort of direct you know this for better or worse most of the products i've gotten to work on are platform products and and i find that there's two very different kinds of people that like that challenge so some people and i think steve jobs was the the quintessential example of he didn't really like being a platform like you know the iphone ship without an app store you know they locked down the screws on the back of the on on all the devices that nobody could open them and you know his viewpoint was i'm going to control every element of what my users see and on the other hand you know platform thinkers you you sort of assume that my connection to my eventual user is through someone else like youtube you know i regularly come to you they come to work at youtube and somebody say well here's what happened last night you know and and uh sometimes it's heartwarming like oh my gosh this person you know this kid bit this other kid's finger and it took off like crazy and you know this uh korean pop star just like broke through the billion view mark before everybody else did and sometimes it was not heartwarming and you don't get to control that because that's still part of part of being a platform and so it does change you know how you think about the the way you run the team because if you have a loop where your community ecosystem users saw and control that narrative then you have to you have to incorporate that i mean the other close analogy i think is like you know airbnb in the famous story of them you know taking pictures of people's apartments it's like you know they had to they had to reach out and try to control that and eventually you can't do that you have to you had to sit back and let people you know market their market their hotels and you know thankfully the ecosystem got good at it but it kind of has a similar dynamic i think absolutely with airbnb pricing is even more of a challenge where a lot of hosts don't really know what to price they think their place is worth a lot more and we can't tell them the price there's laws around that and so hey maybe you want to price it at this rate you know what's good for you so it's a lot of encouraging so yeah i've seen that in action so you talked about teams and how you think about structuring them a bit and that's a good segue to our second topic which is around a book that i hear you're writing called the rituals of great teams and i think what you're doing there is exploring rituals that have emerged at some of the more successful companies and so just a question that one how did you get interested in rituals so much so that you decided to write a book which is such a trudge and endless amount of work and just like how yeah where is it at how's it going and i'll ask you a few more questions there you know the the writing a book somebody's books called rituals of great teams and the uh when i signed up to do it i thought it was going to take six months i'm now almost two years in uh i mean i know my manuscript in four months and i am probably half done so there's a lot of work uh ahead of us in in um building but it's one of the most fun projects i've ever done so so the history behind this was it as in many cases is a lot of you know sort of odd luck and and happenstance i got hosted right at the start of the pandemic i was interviewed for a different podcast called masters of scale by reed hoffman and the way reed records which is not i'm not sure i would recommend this but he does it a little bit differently than that than you do you sit down with you know no idea what you're going to talk about you talk like three hours and then he has a group of editors the same group that actually had its head and they come in and they pick 20 minutes of it and they turn into one of and you have no idea what it's going to be see it's talking talk talk and gets 20 minutes out and they're pretty good at getting to a nugget so they picked out of this whole discussion this part that i thought was like really small and it was um reid had asked me for one of my favorite quotes and i talked about this quote from a guy named bing gordon people don't know bing bing was one of the founders of electronic arts he's a famous investor from amazon zynga so on lots of great companies and i happen to sit on a board with bing and he used this line i think being one of the best non-linear thinkers in the valley uh always learned something with uh with bing and he he used his line that really stuck with he said uh great companies has a very small list of golden rituals and there are three rules of golden rituals uh number one they're named number two every employee knows them by their first friday and number three they're templated and and he has great examples you know amazon has six pagers and google has okrs and salesforce v2 mom and there's all these different rituals that that that people do and i ended up sharing on the podcast a little bit about what coda's golden ritual is if you were to ask a set of code employees on the first friday what code is golden ritual is it would almost certainly tell you about this thing we do called dory balls it's it's sort of the key of how we run meetings and do write-ups and so on it's pretty simple idea is that in our write-ups instead of um and in our meetings instead of you know going around the room and hearing what everybody thinks we we do this thing called pulse everybody writes down what they think and we hide everybody else's until you're done writing so you kind of you force yourself to to be eloquent about your opinion on the record about it and unbiased and then the second thing we do is called dory which is instead of sort of randomly asking questions we ask people to put the questions on the table and then we take a round of uploading and downloading them to actually figure out what we're going to talk about dory's named after the fish who asked all the questions it's something it's a tool we use a lot of google that we kind of turned into this this mini tool if you were to ask instead of code employees on the first friday about kodu golden rituals with bing's three tests they're almost certainly talking about dorian both and it's not because they're meeting wonks it's because it's indicative of the culture of the team and so the the uh you know i'll regularly hear employees say things like yeah i just joined koda it's been a week it's amazing the culture is so open that i got to ask a question in a meeting and it outvoted the ceos um or they'll say i got asked for my contribution to this this really hard decision we're making and it was thoughtfully presented i had space to be able to do it well without bias and it was actually rad and considered as part of the as part of the decision-making process and so this this the the podcast on um your wreath podcasted pretty well and i got all these questions about rituals and so i decided to do a uh a dinner which turned into a dinner series and we basically every third wednesday we would host a group of people to share the rituals with each other and i learned a bunch of stuff in this in this process i've now interviewed over a thousand people for this book and and you know there's a lot lots of really interesting rituals that come out of it well first thing i learned is people love sharing their rituals i get i you know i've interviewed people from many companies that everybody's heard of you know nike disney new york times you know it's on all the way down to many that you know any startups that maybe people haven't heard of or or companies and industries that people don't uh talk a lot about um book authors uh pundits lots of different people that come to this process everybody loves sharing their rituals everybody's little secret to how they run their business but for some reason the how we work part video everyone's very willing to share people also love hearing about them and i was kind of like is this going to be interesting for dinner like you know geeking out about how a team worked and it turns out not only the people like hearing about it it's the the littlest details that matter it's like oh yeah we kind of do that too but we have this issue how'd you get past this and you start discovering that actually those little details are what make would make or break a thing that you know you can't quite do it the exact same way and then the other thing i realized which is probably the most important point is that rituals are i like to say that there are a mirror of culture that i mean one of the attendees there is darmesh founder of hubspot and very thoughtful uh person he talked a lot about this thing that is originally presented as something called flash tags which is really really uh cool example but uh darmesh talked about how we when we're building companies we actually build two products we build one for our customers and we build another one for our employees that's actually that how we work part of it that's the term he uses for that is culture that's the product we built for employees i think it's a very like interesting way to define what what culture is interestingly when you ask people about their culture hey what's the culture of google or airbnb or so on the way they'll answer the question is through rituals and say you know here's here's what we do and the way you know this is what we do is through this ritual that uh that's in place so i i thought that was pretty interesting um started off just building a little listicle like here's all the great rituals and then i started realizing that actually the comparison between them is kind of interesting and so i started sort of filling in the gaps between them of like when would you do x versus y and what did i learn through that process publisher asked if i would turn it into a book and i i um uh agreed without really contemplating how hard it would be and it's just it's become what most of my evenings end up being on this i have a wonderful co-writer erin dame who's uh who's incredibly gracious with her time and uh um and for helping me sort through the best ideas and the worst ideas but it's a really fun project what are some of the more wacky or and or impactful rituals that you've come across that you can share oh boy i did it sorted down the list i'll tell you some that are um interesting and recognizable one of the most fun ones is from ariana huffington and she shared a a ritual called reset and there's a bit of background on ariana i mean but well known for huffington post she now runs thrive the um she had an accident a few years back and ended up going through this period of doing a lot of research on how the brain works and ended up coming to this set of conclusions about how you can affect your own brain chemistry and one of the things she does as a sort of personal ritual is a thing called a reset it's basically you make a the the rituals you make a one minute video that is um personal and it's uh they have a they have a little template for doing it but it's a breathing exercise so it's like you you're supposed to like play it while you do this breathing exercise but it's personal so it's like her video you go search youtube for ariana's research you'll find it it has uh you know it has pictures of her kids it has uh it has quotes she loves it has you know videos of her hometown in greece and so on but the way she brought it to the team was they start meetings by randomly picking someone they call spin the wheel they randomly pick someone and they play their reset and the idea is you get this like like two for one where you uh everybody gets you know a little bit yeah the the the brain chemistry rewiring of 60 minutes 60 seconds breathing exercise everybody gets back into that sort of zen state a little bit and you learn a little bit about each other and the she was saying that you know the pictures people pick and so are interesting but actually the music people pick is one of the most interesting like a lot of people pick con music some people will pick something they rock out to some people but you know everybody does their reset a little bit differently so that was a really fun one um another really fun one that i was surprised by gusto does this thing in their hiring call so you get an offer from gusto and apparently when you get on the offer call of you know congratulations you got an offer instead of just meeting the recruiter which is what most companies do they have the entire group of people that interviewed you join the call and they all say something about why you're amazing and you should join gusto and like i said it's such an interesting ritual in so many ways i mean like it's a for the candidate obviously like what an amazing experience it's like you know the uh uh to use an airbnb term that's like a level 11 experience of uh i don't know what that what that feels like but also for the company it's like hey we're like we're gonna get you know i have one of the questions to ask what if i voted no what if i said this versus the no hire and i said it doesn't matter you're on the call you're going to work with this person you're going to help them feel welcome and you're going to help them understand you know where they stand i just i think it's uh obviously takes much time but it's you know it also is a signal to the company of how important hiring is and something that obviously we all we all prioritize those are maybe a couple of the different kind of uh maybe different ones that people might not have heard of before those are amazing examples have you integrated any of these rituals from other companies doing this research into into coda all the time and it's the cheapest form of research i mean i get to like borrow all these great ideas from all these companies we just added one to our decision making process this is like this is a good example of a little detail that really matters is coinbase has a ritual that's formed around this is a decision making ritual they call uh is it amazing how many companies have a decision-making ritual with a name with a with a verb right so like at uh um uh at square goku called them spades this is like he he kind of verbified it there's a template there's a you know but you you use bing's three tennis right it's got to be named every employee knows by the first friday and it's templated so coinbase does a thing they call rapids and rapid is you know uh framing around what the um uh the the roles in it are the the responsible approver participating informed and decider but their their technique of doing it was really interesting they they have this like subtle nudge thing that we weren't doing that i've now incorporated at dakota so we at coda we have dory impulse like a very common ritual we do it spread through a lot of different companies that use coda you don't really have to use coda to do it but you know i think code is pretty good at it but one of the things we were facing was that you would do this pulse and so you have a meeting and it could if you did it wrong it could feel like voting and it could feel like consensus building and so we would get this you know people would talk about it and every ritual has it's like pro and con but people would look at it and say very open culture you're allowed to share whatever you want but on the other side for the person that's trying to make a decision it can feel like oh my god i now have 30 pieces of feedback am i supposed to wait for like all 30 of them to be yes am i supposed to wait for it to be a majority and like how do i know what am i supposed to supposed to do here and so coinbase had this really simple idea that we sort of smushed together which was at the top of their rapids they named who all the people were and then next to each one they put a little box that said what is the decision from that person they just organized it's very similar to pulse but they kind of organized them and said like yeah everybody that's in the inform bucket can like they can comment it's totally fine but we really care about the approvers the responsible and then of course the decider the the one that really matters and so we just added a column to our pulse which is what is the role and we group the table by that and the other thing that coinbase does it's like sounds really subtle and small but really in detail that really matters is the person running the meeting pre-fills that with what they want from that person you are an improver uh you know maybe i have three approvers because i have a budget approver and i have a marketing approver and i have a you know i have a sales approver whatever it might be and i need you to give me this answer i don't i don't need you to comment on everything i'm doing but i need to tell me do we have the budget or not i need you to tell me you know am i am i authorized to make this change in this part of the product that we generally don't change or can i change the onboarding flow or whatever whatever it might be and we took a process that i think was like doing a pretty good job of getting getting rid of groupthink which is really the part of what we were doing with pulse but had this danger of being overly leaning towards uh you know consensus consensus building to a fault i think consensus building is a good thing but consensability to a fault is not and we sort of stole this one from coinbase and we switched it in and it like got better and i think that's a good example very recently i feel like you have a clear best seller on your hands here and i can't wait to read this i almost feel like you have an unfair advantage right now having all these insights before you share them being able to execute so much more efficiently well yeah it's interesting i don't think it's um i'm obviously not trying to keep any of it a secret so the whole point of publishing it is because i think other people uh will will enjoy it um and and can get can get benefit out of it but you know if people are interested uh one of the other choices i made in writing this book is i decided to do it somewhat in the open so there's a what i call the rituals the great team's brain trust and so if you go if you just search for me and rituals of great teams you'll find it and i'm sure we can add the link to the show notes but uh you can sign up and and basically as i finish a chapter i put put it out to the group and there's now a few hundred people that are helping me co-edit this thing some of them just because they like someone come in and give give me help on storytelling grammar and so on but a lot of them are contributing they show up and they say hey you missed this one like i we actually do that but we do this other thing a little bit differently and you should really talk about that and and because i kind of view it as a um you know it started as a dinner series right it started with a you know everybody's going to give to each other and so i kind of wanted to bring that into the writing process it's also a cheap way to get some pretty good editors um and it's pretty helpful i love that that is really smart this episode is brought to you by epo epo is a next generation a b testing platform built by airbnb alums for modern growth teams companies like netlify and tenfold and cameo rely on apple to power their experiments wherever you work writing experiments is increasingly essential but there are no commercial tools that integrate with a modern grow team stack this leads to wasted time building internal tools or trying to run your experiments through a clunky marketing tool when i was at airbnb one of the things that i loved about our experimentation platform was being able to easily slice results by device by country and by user stage epo does all that and more delivering results quickly avoiding annoying prolonged analytic cycles and helping you easily get to the root cause of any issue you discover epo lets you go beyond basic click-through metrics and instead use your north star metrics like activation retention subscriptions and payments an empo supports tests on the front end the back end email marketing and even machine learning clients check out epo at getepo.com geteppo.com and 10x your experiment velocity how have you found these rituals form for someone that's listening and are like we need some of these we need some rituals we need to move more effectively how do these come up are they just organically kind of organized do they come in from other companies and they evolve is it founder driven what have you found so far i know you're still working on the book but curious there's a whole section in the book on i get this question a lot like wait you know what how do i find the rituals we have how do i adjust the rituals to match are they supposed to change you know there's lots of lots of things you see because some rituals are like great when you're a hundred people and they're terrible when you're a thousand one that and you should actively change those things there's some some are great and what sometimes people call peace time versus war times because you should you should do this during this time but you should like actively not do it what do you when you're when this other time and actually that in itself is the rich woman every company has some form of a war room ritual that is uh i guess facebook i was learning we did a dinner last night so i learned a little bit about facebook apparently i called them lockdowns which is a term i hadn't heard before but apparently like it's well understood when they say lockdown everybody knows exactly what it means it's like we no longer do the goal setting process we you know you drop this type of work and like everybody just knows this is what this is what it means but i would say that you know when we talk to people about rituals there's a set of rituals that happen organically i mean those are the easiest it's like dorian pulse for us was one of the part commanders running a meeting just thought it was we're a we're a uh distributed first culture but like i didn't really adapted to it properly and you this this partner just annoyed at waiting around on zoom for everybody to go around and say they're beasts it just took forever and by the end it was like it's everybody's like my son can we just pause it we just write down what you think and he just happened to do it in the thoughtful way and it just took off and so some rituals grow organically and just you just gotta like wait for them to to do it but there are cases where companies actively form a ritual to to drive a certain behavior and you know the best advice i've given on this topic is um to read one of my other favorite books this is um it's a book called switch it's by chip and dan heath um all their books are amazing um but they also wrote decisive and made mistake and moments and and so but this one is my if i could recommend five books this would take two slots on the list and the subtitle the book is how to change things when change is hard and the basic idea of the book is they use this analogy of an elephant a writer on an elephant on a path and when you're trying to change things you have three options for what you can do and actually kind of map the bing's analogy so you can direct the writer so you can tell people what to do you can motivate the elephants you can kind of give this thing a kick in the butt and it's going to move you don't know exactly where it's going to move but it's going to move and you can shape the path shape the path is i'm going to set this way up so that you can only do these things and so you know the way i think about it is you know direct the writer tell people what to do that's why you know if you look at bing's tests that's that's why you teach employees this and before the first friday because you tell people like this is what we what we do and so some rituals like how do you get this ritual to work you put it in your new iron onboarding and you you make it work that way motivate the elephant a lot of that's about branding right so what do you do with rituals like it's an amazing number of rituals where i'll tell people that seems like a great ritual i would highly encourage you to give it an a get give it something that lets people anchor ideas to it name's a very powerful thing if i if i said oh yeah kuda we do voting and sentiment writing um or some i don't even know what you would say it would sound boring it wouldn't sound like something you could brag about something you could form identity around and so it's very important to give it a name and then and then finally like why do you shape the path where you like set things up you you tempt us make it easy make it as easy as possible to follow this ritual and make it just a part of what we do enough then you know if you're at gusto and you're like i don't really understand that's hiring cult thing where it's like guess what you're gonna be invited to one soon so you're gonna see it and then and you're gonna have to do it or or if you're at square like you're gonna see the spay template and you're gonna you're gonna learn how to do it so i think switch is um i i recommend this book for lots of purposes but as you're thinking about rituals for your for your teeth it's a it's a particularly relevant frame awesome by the way that's a little plug for the youtube version of this podcast which is now a thing we do so if you're like hey i don't see what you're talking about uh just search for lenny's podcast youtube and i think you'll find it and we'll probably link to that yeah in the show notes i'm also reminded of airbnb's rituals which you probably already know about but they're kind of all hilarious and weird one is formal friday where people dress up in suits and gowns on fridays another is a human tunnel for all new employees where every new employee has to run through a human tunnel and jump into like a bean bag or something yeah and then and then there's a new hire tea time where new hires drink some tea with some veterans and chat about where they're from and things like that there's a bunch more but there's the ones that are my favorite those are great then i uh there's a i also included some of the uh the the sort of level 11 thinking as well i think that's also a good chess key favorite brian also has another one that's in the book that's about how to rank your to-do list by finding leverage that's a that's a really fun one as well like don't write your to-do list a lot of people do like importance versus urgency or or so on and i guess he sorts his by which of these most likely to create leverage um like getting rid of the rest of my list which i thought was a very like yeah i started doing that in my to-do list and it's very interesting and impactful so that's actually an incredible segue to our next topic which are eigen questions oh yes and so again questions one of your most classic posts you mentioned this at the top may be your most liked post other than maybe bundling it sounds like some kind of german game show i get in questions can you tell us what i can questions are and then i'm going to ask you a few more questions around that i had thought about the german uh gameplay this would be a very interesting game but we can we could try to create one okay i'll describe what eigen questions are but maybe i'll start by telling you a little bit about where the concept came from and um this is actually youtube-ism that maybe just to place ourselves in history so 2008 i joined youtube and um many people don't remember this but youtube at the time was seen as a mistake it was seen as google's first first bad acquisition everything else had worked but this thing kind of seemed like a disaster it was it was grainy videos we were losing lots of money we had billion uh billions of dollars in lawsuits it's like none of it really seemed that obvious there's lots of discussion on how to reorient it fix and so on and so everybody outside that's what they that's what they saw if you step inside a youtube staff meeting in 2008 actually one of the toughest questions we were answering was this question we called the modern family question and and um it may sound small but it actually was kind of very perplexing the question's pretty simple the question was you know people if you looked at our search traffic at youtube one of the top five queries basically every week was for a show called modern family and modern family was number one show on television the time you know by far the most popular and we were the second second biggest search engine in the world behind our parent company google so search traffic was very important there was one big problem people would come search for one family one big ball we didn't have modern family on youtube and so we give them some pretty crappy responses and the question was you know oh and and by the way abc.com had decided to post every episode of modern family live on their on their on their website which nowadays is kind of typical but in 2008 that was not typical those is it kind of a big gamble they made and they're gonna post all these all these shows so the question was you know should we answer the ques the query modern family by linking off to abc.com do we link out or not and the company basically divided and so there's like half the half the company mostly the blockchain damage team's gone kind of aligned around a viewpoint that was that's what the user wants link them off to abc.com you know we're not owned by google google tries hard to prioritize do right by the user the rest will follow like that seems like the right thing to do so let's do that and in the other half of the company most of the business functions sales marketing especially content partnership said please please please don't do that if you do that and you start linking off to all these other places nobody's ever going to put good content on youtube and we're just going to get the stuff that doesn't deserve to live anywhere else that's like not a very good path to be and you can imagine that like those two mindsets it's almost like it was like good versus evil debate which is like dude right by the user of the business so like these are all like like almost impossible to solve problems and this would like happen meeting after meeting after me what did we make a decision yeah modern family what are we going to do so we had this off site we said we're going to like spend the whole day and we're going to figure this problem and we went up to this this hotel happened bay and the executive team all sat down and i was asked to frame the discussion go collect everybody's opinions and collect all the data and just like ground it all in fact so then we're gonna have a discussion about reach a decision and so the night before i'm sitting and thinking about like how you know how long am i gonna have this discussion not just be like a shouting match of yeah of this good versus evil position and i happened to read a uh analysis that was being done by a different team at google the google shopping team where they were facing this interesting challenge of um they were in this deep fight with amazon and they were getting their butts kicked and they were trying to figure out why and it the walking in theory was yeah the google shopping team's view was why would anybody ever be going to amazon like you could come to google and we had indexed all of amazon and the entire internet why would you ever pick amazon and the feedback that was coming back from users was they would they'd say i picked amazon because i value consistency over comprehensiveness they would say things like you know i i really value that you know i go to amazon.com and i understand how the reviews work and how the ratings work and i know that the returns work the way i want i understand shipping is going to work and and it just felt consistent i know it doesn't have everything but it has enough and i value consistency over comprehensiveness and so the night before this this youtube meeting i decided to reframe the question and say let's not have a discussion about linking out at all we're going to start by having a theoretical discussion about in a decade from now is the video market more likely the online video market more likely to be about consistency or about comprehensiveness and that is a question that you can have a very reasonable debate like what are the what are the reasons why a market evolves towards consistency of comprehensiveness and we basically have this discussion and we all came to a conclusion this market is gonna value consistency over comprehensiveness and by the way i think i mean now almost 15 years later i think we're right i mean i think the if you go look at the the video market obviously has exploded there's so many great video properties out there none of them are comprehensive there is no one-stop shop or a place where all the video exists and so i think we were right but what what happened was by answering that question the link out question all of a sudden became super easy like we value consistency over comprehensive we definitely don't need to link out in fact we should make a whole bunch of other decisions as well so we went and you know at the time we used to this is the days of flash players and so on we embedded other people's players on youtube we stopped doing that as well probably the most famous decision we made was with the iphone as i mentioned earlier the iphone when it first shipped had no app store and so they built all the first first few apps including youtube and so here we were a few years later iphone most popular for the planet and the youtube app on the iphone was built by a team at apple and they had not been able to keep up with what we were doing almost half the catalog didn't play back on on the iphone they were missing a bunch of features and so on so i drove down to cupertino and i sat down with with scott and phil and said hey we're gonna have to take back the youtube app and they they said i don't understand why would you do that like you have default distribution on you know of course they were concerned the most important operating system in the world why would you do that you're gonna have to like rebuild all this from scratch and you know uh just seems like a really bad choice and i said no it's like actually quite an easy choice we value consistency over comprehensiveness we would much rather be on fewer phones or the more consistent experience than beyond all of them with an interesting experience so what this is the choice we're making and it worked out fairly well so this decision the modern family question ended up becoming named as the example for a term called item questions so i open questions it's not a german game show it's not it is a made up word and it's named after a math concept called eigenvectors and um you know the math is not really necessary but for people who are curious you know go back to linear algebra eigenvectors are in a multi-dimensional space they're the most discriminating vectors of the of the vectors in that space the dimensions of that of the space it's a concept that gets used a lot machine learning and and so on but actually the math doesn't really matter the the iq question the simplest definition or eigen question is the question that when answered also answers the most subsequent questions and it's a very simple idea that when you sit down and you say hey here's all these questions we ought to answer how do we all usually rank like sometimes we write them just by yeah what order we came up with them sometimes we write them by importance like which is the most impactful decision we're going to make but this methodology says don't rank them that way rank them like you said to us he's thinking about leverage same idea rank them by which ones would eliminate the most other questions on the list so you take that list you said should we link out to modern family should we own the the the youtube iphone app you know should we do live those were all like really hard questions to answer but it turns out if you answer just one question do we value consistency over comprehensiveness yeah it's for all the others they all all of a sudden become very simple and so this this idea of i questions became part of her vocabulary it became a clear ritual for uh for youtube that you know what is the eigen question here it is for code as well it sort of spread to other places but that's the basic idea amazing what a what a baller move with apple and uh it's pretty scary room yeah yeah but i was just gonna say it i think youtube's probably in the top 510 most downloaded apps so it worked out you know that we go through that meeting i bring along our uh the the the product manager for the iphone app game andre darwin shift and he's the one like having to explain what we're gonna do and so on and it's like a hard contentious meeting and as we're leaving the meeting he says hey can i get a selfie with bill and scott audrey what are you doing it's this great picture of him like with us asking for you know take this back and he's clearly he's very stark striker like there's a lot of people who view was like apple would do a better job of building a youtube app than us like who are we to tell them to not do that um of course you know in retrospect that was that was a silly way to think about it wow i would do the same thing and it's amazing who's this person because that's awesome i love that as a leader you bring the the pm of the team working on it versus just you know the big big shots at the top yeah andre he's i mean he's now a founder he started a company called optic which is basically building content id for nfts which is a you know much needed thing in the web 3 world yeah i mean the team building it i mean it was uh that meeting i brought him i brought my partnership's lead and i brought the engi lead that was covered in area too and yeah i think that some of it is you know if i had to be honest some of it is like they really wanted to come and meet with apple so i was like you know for my own sake like i kind of wanted some backup i'm about to make this kind of bold ass and it could have been uh i mean to apple's credit i mean they could have been you know pretty bad about it it could have like not allowed us on the store and and and so on and they they said okay well you know we don't like it but we understand your choice you have to know that you're gonna start from zero we're not giving you a single download uh uh for free you're gonna have to start from zero and we will break the current app why did you agreed upon date and the negotiation was can you please just tell those people that there's a new app and so that's what we negotiated out of it and they eventually did that and it was uh that that was that was fine and in the end youtube is now you know one of the top apps on on iphones i think i mean it was like six months after launch we had like eighty percent share i'm like almost like everybody downloaded the app visited and said it kind of ended up not being that much of a comprehensiveness uh uh uh choice but it was a yeah clearly hard decision made much easier by asking the right argument question first wow speaking of biking questions are there other examples of any questions that come to mind to make this even more concrete in people's minds i don't know if that's the right way to frame it or is it more just when you have a list of questions look for the one that'll answer the most how do you how do you kind of operationalize this concept you know there's lots of them i mean for coda the eigen question um the sort of most concept lagging question for coda was yeah we use a line a lot for coda that could allow anyone to make a dock as powerful as an app you could reverse that statement and say allow anyone to make an app as easily as a doc and those two sound similar but they're not they're they're actually quite different statements and so our our most commonly debated question is you know are we more committed to being a doctor being an ad and you know which way we want people if people are going to misunderstand good at would we rather than perceive it as a document or perceive it as an app and we decided on doc which is actually and the way i the way i cemented that decision when we made it was i named the company now where coda is a dock backwards i said well well that's uh it's a this is this is going to be a we're definitely not revisiting this one this is a uh code is a dot first that's a good example i mean by the way i would say i didn't questions is a is a term that um you know a guy could resonate with someone but it's a hard technique it's not it's not always easy to know how to do it and one of the things you know i get asked a lot it's like is it a skill you can learn i absolutely think it's a skill you can learn it's a thing that once you observe it get better at it you can learn it but it's not it's not easy to learn it and one of my observations about learning skills like these is you want to learn them in non-pressure filled environment like to use a an analogy you know if you were trying to learn a sport or learn an instrument sean imagine if you never did practice like every time you played basketball was in a real basketball game and every time you played the piano was in a recital you know you probably would never get better and i think one of the troubles with the concept like eigen questions is we tend to only practice it in real world scenarios that are high stakes and so one of the one of the things i encourage people to do is to practice eigen questions and completely almost frivolous situations so i i have an inter interview question i ask which i think maybe we'll get to this a little bit uh uh later as well but it's a very it's a very simple question but it's a and it's a coded eigen question test and the question is um a group of scientists have invented a teleportation device they've hired you lenny to be their sort of business counterpart bring this to market what accounting for what this question actually worked well for anyone but say you could be the product manager for this thing bring it to market um and what do you do that's the whole question right people usually people will start asking a bunch of questions and say well tell me more about this device like what does it do how does it work and you know is it is it big is it small is it fast does it like does it disintegrate things or not does it need a receiver and a sender does it you know it's safe if all these different questions come out and at some point i'll just let those questions come out and at some point i'll say okay nice uh nice job generating all the questions turns out the scientists they kind of hate talking to people and they're kind of annoyed by all your questions and so they've decided that they will answer only two of your questions and after that they expect a plan what two questions do you ask and interestingly all of a sudden like the sharp you know product managers engineer so like basically everywhere they very quickly find what are the two one or two eigen questions on this topic and there's no right answer but i'll tell you like one of my favorite ones is as a product manager said okay if i had to ask two questions the two questions i would ask one is is it safe enough for humans or not and that was a like a very like crisp way to get to like just safety how reliable is it didn't ask how reliable it is how many bits in the middle is like just tell me it's safe enough for humans or not and the second one is is it more expensive cap extra optics is it more expensive to buy them or to run them and then he took those two questions and he said like jeff with those two questions i can perform these quadrants you can say oh it's safe enough for humans and it's cheaper to they're very cheap to buy but expensive to run then you probably run them like human fax machines like you put it everywhere you can and you say hey look it's expensive to use but like you all have the ability to teleport anywhere you want and this is how we're gonna run it if the other hand they're very expensive to buy but very um but cheap to run you probably have to place them very strategically in which case what you probably do is replace air forms like airports are pretty strategically placed in places where people are trying to trying to get around places if it's not safe enough for humans then you've got a whole different class of use cases where you go value what goods are transported in very costly ways and you know people come up with like you know do you do the most expensive things or do you do like you know is is teleporting you know people's replacement marks is that like a like a really demanding thing but these two questions kind of kind of get to the the part of it the question's totally made up like no palpitation device exists at least not yet and i find that people's ability to learn the method is significantly higher if it's low stakes that question by the way if you ask a kid that question then you know a new teleportation device you know you you get to ask two questions almost every kid will like quickly get to two pretty good item questions again kids are incredibly good at simplifying these things down it's actually a skill we like remove from ourselves like i'll see all your candidates tell me things like i guess i would ask them what what size it is and like why would you ask them what size would what what what decision is that going to allow you to make to to know what size it and you know sometimes i can explain it but sometimes not and don't get hired but yeah but actually the thing i'd say about it is there are iron questions kind of everywhere i mean so you could you could take any product out there i'll do it with my kids a lot they'll say you know i was just writing with my younger daughter and she said you know how come there's three gas stations like in the same corner so why why why did why do people do that yeah uh that's a really that's really insightful observation what's the question how do you place the gas station and there's like a burn yes if you could you can almost take anything and say what is the question that really drives judge's answer i love that do you do you actually still ask this question because you're sharing it and all the answers no i don't and i have a new one that i can't share but it's a and we've written about it in fact one of the big debates about publishing the i prevention thing is in order to bring this to life i needed to answer your question like how do i how do i test this how do i practice this and like it's much easier like nobody can repeat the youtube one because like they yeah nobody has that choice sitting in front of them so it's kind of a useless it's it's entertaining but it's as a teaching tool it's kind of useless because you can't you can't really go reinvent history and decide because yeah versus conferences and to sacrifice one we sacrifice one yeah to pull on that threat further and kind of dive a little deeper into into evaluating talent and product talent i hear this is one of your superpowers and so i'd love to learn from you and what you've seen around how to evaluate talent so you talked a little about interview questions you ask um so maybe we could either go in that direction or just like what do you look for in people that you're hiring interviewing that maybe other people aren't i have a technique for it let me show i'll show a quick picture and youtube plug yeah youtube you can uh i mentioned it before the call you can you can put video on spotify now too but they do the spotify plug all your platforms that you've done now you can multiply my videos that's right now so i'll talk through this diagram that has two axis scope this acronym pshe and this line so i'll stop sharing and describe it and we can come back to it but that's a little bit of this technique sort of changed how i think about evaluating not only product talent but it actually turns out you can use the same set of rules for evaluating basically every role but i'll tell it from how you asked it about product talent so 2011 larry page took over at google and he made a bunch of changes to the company mostly quite positive and one of the ones he did was he he moved us from being a functional organization to being a business unit organization we call them product units but roughly the same thing and there were eight product units at google you know youtube and search and ads and chrome and maps and so on and it's very positive it's like hard to believe that we were already like really 20 000 people were still functional like all of engineering all the products weren't reported into the ceo just seems like totally crazy with the breath of products that we covered one of the downsides of it was like in any functional business unit switch if you lose some of that that what does the function mean and in particular things like what is a good product manager was a question we were at risk of losing so a group of at the time i was running product for for youtube the group of the the eight product leaders around the company got together and said hey we need to like keep some level of um consistency amongst how we think about what's a great product manager or it's all going to diverge and it's not going to mean anything anymore and um actually it's a fun aside we did a ritual that i've repeated a few times but it isn't done often enough is uh we said so who's going to like drive this process and we did an election i don't know why we do elections in like the public world but not in the private world but it's actually like quite effective we used to do them on youtube where we would like we would elect into certain roles and like you got a one-year term you gave a little speech you're like we uh uh we did we did an election anyway so i i got chosen to be uh the uh the first sort of leader of this of this this this challenge could give me the keeping the product management function together at google and the the most obvious job we had to do was come up with a speech for the calibration committee so google google does um calibration a little bit different or promotion a little bit different than most companies most companies your boss decides to get promoted or not at google there's a committee that decides it's supposed to be a committee that doesn't actually you know work directly with the person so they can be a little unbiased and so on and it gets done the ritual was to do it in a hotel near the airport here in san francisco and everybody get in these different rooms and there was always a a speech given at the beginning that used to be given by jonathan rosenberg who ran a product for google for many years and now had to be given by somebody so i'm going to give this speech now i got to figure out what the hell am i going to say like what's a good what's a good product manager and as i was going through this i decided to run this little exercise so this this group of eight product leaders we took the level guides we had a level guide for product managers every company does and we took it we printed out a sheet of paper cut it into little slices one per level and we cut off the title and the number and i handed them out and i said can you reverse identify what level you're holding okay then turns out nobody could do it then you know and and it's it's not easy to do like the these love the level guide had been like you know kind of added to over time and not really that refined and so it's full of all sorts of things that were you know whatever that at that time was the priority of the team they stuck it in the level guy and so it would say things like this person can manage a medium-sized project and they interview at least three people a week and they always send their notes out on time and uh their expense reports are always fine it'd be like oh that's a director and it's like yeah and and you know it's just whatever we were trying to incent was sort of stuck in this thing so but we noticed that if you took all these sheets of paper and laid them out side by side you could order them by exactly one statement which is what was one that corresponded to scope so there's some word in there that was an escalating adjective that mostly correlated to how big is the thing you run and so we decided okay well we're going to standardize we're just going to focus on making that clear and so we said we're going to find scope and so that we all kind of use it the same way and we came up with some some stopping points and basically said you own a feature you own a group of features you own a sub area of a product you own multiple sub areas of product you own an entire product or you own multiple products in a product line and that's going to be how we think about scope go forth and and evaluate your teams we have the next meeting couple weeks later and everybody comes back upset like this didn't work at all like what would happen and said well the search team is super mad at the ads team because search is like one product like the entire thing is one product the ads team they went and invented all these products because there's like every little thing you do in the ass platforms has a skew has a p l as it like it's like it you can you have lots of products so that doesn't work the second issue was they said the scope is actually an input not an output like when we were talking to our manager and said well this person should get promoted they'd manage this a huge goat and then they would say but you gave them that school like i said we should be judging you not that not the person how do we judge what they're doing with the scope like it's actually it's actually very different and the third issue was in almost every team some of our best people were working on things with odd scope that we didn't they were risky projects we didn't yet know is this thing going to work or not work it might cancel and if we put in place a system that only rewarded scope we would heavily disincent people for working on these riskier more creative things so um they're kind of stuck and then we ended up seven we went through lots of frameworks we ended up telling those one called pshe and it comes from old mentor mine button clark who's my boss at microsoft for a number of years and it stands for problem solution how execution bshe and i will say it's a um i've tried many times to come up with a better acronym and i have not been able to come up with one so it's push that's the word that's as good as i can dance all right here's how it works good enough right so push that's all you have to remember the um uh it doesn't roll off the top but it's maybe i did better with eigen questions i think but this um so here's how it works so if you're sort of a junior product manager what happens you get handed a problem you get handed a solution you can handle the how you know go talk to this person write this document run this meeting so on and all you have to do is execute run that playbook and that's all that's all we expect getting you can become a little more senior we hand you a problem we hand you a rough solution you figure out the how you figure out the how are we going to organize this what are the milestones what you know how we're gonna get it to market you know how are we the meetings what are the rituals like all those things show up in the age at some point become a little more senior we hand you a problem and you come back with the solutions you come up and we judge you on the creativity and the effectiveness of the solutions and at some point you're senior enough that you tell us the problems and you say hey i know you told me to go work on activation but actually i think our issue is brand or i think our issue is quality or i think our issues whatever it might be and that's sort of the the pinnacle of this way of thinking about it now just back in this picture for a moment one of the interesting things that happened was that the teams went and they evaluated their teams on these two axes and there's they end up with a sort of curved line between them so it's not linear as you as you work your way through and what happens is early in people's career they mostly sit at that e point you get another problem and the solution handed to how and you just execute and they gradually grow in school later in people's careers similarly you know you're at that p level you just do bigger and bigger products and it's like you know the the job of being an entrepreneur ceo or an owner or someone is just kind of doing bigger and bigger projects but in the middle the the slope changes and all of a sudden it's not really about scope it's about psat and there's a circle drawn here or what i like to call the trough of dissolution and the reason i'll start sharing so we can talk about it but the the the what happens in that phase as to the calibration companies about this the reason we called the top of the solution is for the for the employee for the person this is a confusing time like everything about leading up to this moment from high school and college has been about scope and like at this point you're all of a sudden told we're not judging you on scope anymore we're judging you on this psh you think that's very confusing to the calibrator to the manager it's also very confusing because all of a sudden the difference they put is the difference between a level three and a level seven may not be scoped they may do the exact same job it's how they do the job that matters and here's some language for how they do the job and so psat became a very sticky way of thinking about it it turns out that this way of evaluating people is actually not that specific to product management it's really easy to see how why you do the exact same thing for engineers and designers and and so on but to pick one that may not be as obvious i'll pick salespeople like a very common thing people do with salespeople is they evaluate them based on photo attain it's like the easiest thing to do is take the sales people and rank them by who hit their quota who didn't you go ask the sales team who's the best sales person and what you'll realize is they'll say code entertainment it's just a signal for how good you negotiate your quota and pick the right territory the really you want to know is about sales person they say well you know so and so i mean she can sell anything and she can be in the region that's growing or the region is shrinking or the new product or the old product or and if you think about that terminology it's very similar to psat thinking like this is the person who can come into a new space identify the right problems and solve them that's what makes a really great sales person so it could become my framework for evaluating talent in internal all sorts of ways and you might recognize a pattern of like being a great p thinker is very correlated with being good with item questions like can you spot the right problems it's very similar to can you spot the right questions you decide what's important and so that's been my main framework for value wow there's so much there a couple quick questions is this basically your calibration ladder framework for pms at this point and then this is also just like your interview guide just like interviewed each of these pieces to level the person yeah so it's definitely how we do our version of leveling is you know bshe and the the um we use a uh a set of radford levels right radford has this like really interesting way of describing that as you grow in a profession he uses his analogy of uh of someone um of like a sailor and that a junior sailor is like learning to tie knots and then you gradually can tie all the standard knots and then you can tie the advanced knots and so on and someone you kind of work your way up and at some point the way you're judged is you invented nylon and there's like this like but it's like list between there it's like a way to evaluate every world it's very similar to pshe and so they we sort of look at a kind of similar list but yeah basically all of our all of our roles are evaluated on something that corresponds with pshe and in terms of interviews yeah you look for the same thing and so you uh and by the way i should say interviewing is one part of this and you talk about interviewing talk about calibration there's one other really important one which is reference checking and i think the best way to assess pshe is actually through references instead of the most important guide we write isn't the interview guide it's the when we call this person's references what do you ask to actually get at these questions because your people can often confuse them like i said just to pick product managers as an example we all know some really amazing age level product managers and one of the reasons one of the homeworks of an html product manager is that their counterparts usually love them and they'll say things like oh my gosh a person runs such efficient meetings and like everybody all communication is always clear hassle is always always buttoned up you know exactly what we're doing the market knows what we're doing sales you know we're doing like that that's like this great age and then you'll ask them a question of okay so when you're deciding which problem to solve who is the leader of that and are they picking the right problems or when there's a hard problem who is regularly coming up with the best solutions for them who do you turn to who is the most obvious person to turn to to say like this is really hard problem what's the right solution an amazing number of people will tell you no they run a great meeting but like i actually saw the problem designer does that or like that or the what are the problems now the ceo kind of tells us what to do that's like like not one person is really good at yeah i'm not sure we're solving the right problem but boy we run a great meeting and it's a and it's not meant to diminish any of that i mean we spent a while talking about rituals which mostly happen at that h level so i don't think it's unimportant but it's it's actually quite hard to assess an interview but incredibly easy to assess in a reference check and i think getting good at that is really important you may have mentioned this but what are what's kind of the question there is there a question that you could recommend the absolute ideal case is you get to the person that you do the russian check with and you don't even tell them who you're asking about right you say when you think of your teams who is best at that this obviously only works in cases where you have a pretty deep relationship with the person you're getting referenced from and and so on but you can't always you can't always do that but ideally you want to kind of mimic that behavior one of the things i think that's hard about reference checks is people have you know they have sort of perverse incentives in a reference chain they're not really nobody some of it is is uh for good reason some of it is for bad reason you know people you know generally don't like you know criticizing people and they also like they'll judge themselves and they're you know they're they're they don't want there's legal reasons that they see blow back and and so on and so what i try really hard to do is to draw contrast so you you try to say things so like there's a couple techniques i've used for this in the best case you say i know the per i'm not gonna even tell you what i'm asking you about but when you think about this team who regularly identifies what problems i should focus on who is most reliable at coming up with the solutions to the hardest problems and you're kind of working with do it the other way i like to do it is to provide contrast to give the person an out to not make it obvious to them that i have this ranking and so i'll say things like when you think about this person and i i'll give you i'll give you four different personas you know someone who's like regularly coming up with the problems that the team should be focused on someone who like you know given a set of problems is constantly like solving them in this like really creative way the person that is just really good at getting a team moving and like and or the person who can take a playbook and execute it with high precision and high quality and i will tell them that i have a qualitative judgment that one is better than the other but you want them because you just want your reference check to talk and you want them to say what the what's on their mind of you know you want to give them an opening to not feel like they're judging obviously like another question i always ask people is um would you hire this person again or how excitedly would you hire this person again i always ask them what what questions should i have asked that didn't another another key technique for reference checks is you know you just need people once they start talking they'll reveal what they really feel and often like the little things will come out but for this particular thing if i want to know where they stand on this axis don't tell people what you value by the way i will say i value p over s over h over e i've seen many companies that would have reversed that scale and there are biological industries where it's very required like you don't want a bunch of people running around and constantly telling you to solve some different problem like i just need people that can do this job that we give them super super well and it also depends a bit like what your personality is and what your company's culture is and so on so it's not actually that unbelievable that i might value the e or the p and you know anybody who knows me that's probably not true but you can for a reference check you can probably not give that away wow that was uh gold i hadn't heard these reference check insights and so i'm really happy we got to that maybe one last question on reference checks how often do you find that a reference check leads to you not hiring someone just like ball parker or does that hurt oh all the time and i and i will say um i try to do them as early in the process as practical if it's if it's possible because i think it's actually the worst feeling for an interviewee to go through a process and then get dinged at the reference check stage it's like it's a really crappy experience for the candidate and obviously you have to be careful about not ruining anything for them you can't always do the reference check as early as you can when you're inside a company when you're inside google or facebook or so on you can generally do them even before you enter it's sort of an expectation of like we're going to hear a little bit about people uh in that process and i generally value the reference check over interview signals you know if i had the stack rank in interviews what is the best signal the reference check is the top of the list those people you know they work with this person for you know sometimes for years like their knowledge what you're gonna get out of 30 minutes of artificial scenarios it's just like never going to compare with what a good reference check will give you and then the second best thing i value is anything that feels like a real work exercise and that and and which is also like that even that is hard because some people like their skill set doesn't naturally lead to a compressed time work exercise but you know we do a thing for basically all our roles where at the start of the interview loop the candidate presents to everyone on the loop and we invite some other people uh in the company to attend to for a while let's open the whole company now we're big enough for that's not practical but the the ridge was very simple at the beginning of the loop the person presents um and generally for most roles there's a there's a exercise that they do but about half the time is spent on them presenting whatever they want i mean they can they can talk about themselves they can teach us something and then another half is like give we've given them a prompt something that we we want to direct one of the things that i think um when you're doing interviewing one way to think about it i call home quarter way court neutral work you want interviews to balance in all those different spheres so like you know a common mistake people make is they do all questions in home court hey you know you're joining airbnb what would you do about x and then you know what is the candidate gonna say they can't clearly can't be as thoughtful as you haven't bought they haven't thought about it nearly as much as as you have so really what you're testing is did they come to the same answer that we did which is a pretty crappy like way to judge someone that's not like 10 home court questions tend to be tough you have to be very careful my idea a weight court question can also be tough like this person you say how did you solve this problem and they say well we did this thing you don't really know was that like was that problem actually hard was it somebody else was telling them what to do um are they just like not telling you the whole story of what happened and so on so you try really hard to make most of our interviews are done neutral course so the vast director like teleported question is a very neutral question okay there's nothing you don't know anything about kodak i don't have to know anything about airbnb or wherever wherever you i i can just ask you this question but there's one thing the presentation is the away court question it'd be you now have an opportunity to talk about yourself in this in this new way and it's super interesting what people do i mean i've seen people use that time i often tell people it's the brag session this is like and you know for product managers we do for everyone but it's like recruiters sales people marketers we tell them you know we're going to go through this whole interview process and in most companies you talk to six different people and six little uh segments at the end of the day you say gosh i really wish they had learned this about me and i said i tell them don't leave this with that feeling what what do you want us to know about you like all our questions are are in some senses like we're trying to lower bound you like how bad could this be i want you to upper bound us i want you to tell us what's what's really amazing here and so we'll we'll have the presenter go through that process and what they choose to talk about is very important how they choose to present it is very important but i do it as upper bounding but if i had to stack rank in interviewing what do i look for reference check at the top you know work product presentation next and then all the interviews if when they when they disagree that's the order we judging so sure i feel like you have five books in you that you need to write on so many of these topics this is such good stuff i wish i could keep going i know you have to run so we have this final lightning round so i'm gonna ask you five questions there's one at the end i didn't tell you about ahead of time okay so it's gonna be a surprise and so i'm just gonna ask these quick questions let me know what comes to mind okay first question just what are what are two three books that you find that you recommend most to other people oh pulling out the bookshelf then pull up what's up very good one one i already said we talked about so this this would take two of the slots the next one on my list is a switch by um uh chip and dan he had a change thing to change his heart the other one this is probably a surprising one it's about understanding comics wow super fun book it's a comic book about comic books and you don't have to like comic books at all to love this book it's basically the the sort of starting point is overtime communication has drifted at two extremes so one is we've gotten very good at written form and the other is we've gotten very good at art single pieces of art and comics are the hybrid right they are drawing mixed with writing sometimes i think he calls it um oh god he had a good term for that i'm not forgetting but dude he describes comics really well and he goes through the actual reasoning of what why comics are structured the way they are and one of the reasons it's so important to me is and you could probably tell from like how we talked about different things is often a diagram that crystallizes something the art of of storytelling diagramming so on i think it's so critical for basically any part of life and this book is it just it's still provoking on how to do it understanding understanding comics wow good choice okay favorite recent movie or tv show okay couple that come to mind only murderers in the building is a really fun one and uh uh my family got really into the the marvel series during the pandemic and so one division if anybody hasn't seen that if you're not into superhero stuff and so on which i'm not really that like i'm actually more of a dc comics person i like superman but one division is like one of the best pieces of art that i've seen done in a very long time very very well done show i feel with that show i had to like i gave up and i'm like what the hell is this what is going on and then it gets it gets good it gets good yeah and on the flip side only murders in the building i don't know if you've seen the second season yet but i'm just like i'm done with it i don't i'm just tired of it now i don't know what they're doing oh yeah we're we're only one episode into the second season so we'll see okay we'll give it up for you the first season was so good so good yeah okay and it's all about podcasters it's like so meta around podcasters oh man okay good good segue okay favorite interview question you may have already answered this i the teleporter one is definitely my favorite my second favorite one if i was gonna give you a different one is i have a series of questions around um uh let me me pull it here so i give you the what the real question is uh but basically around a dashboard prompt and it's uh the starting point of the question is uh pick a product i just say your favorite technical product and the constraint is it can't be something that you built you know worked on or competed with it's got to be in the space that like you're not an expert in and i generally ask people why which is actually a really interesting passion test and then i'll ask people design the one page dashboard for that product you're the ceo general manager whatever you run that product what's on the dashboard why does that okay it's an interesting eigen question e type question of like can you tell what's important for this product or not and then i asked them uh to basically redesign the product and the way i do it is you've been hired by a competitor to design a metoo version of product i'm going to leave aside for a moment that why you would want to build a youtube project version what is the bare minimum of what you need to build and then i tell them you ran out of resources you get a quarter of the scope and a quarter the uh you get support of the time and a quarter the the uh the team what do you actually build and then i get down to the other side of like you've decided you can differentiate in only one place what do you do to differentiate so a series of questions that are basically a form of pshe but just formed around a new problem space that gets let's people wander a little bit i've seen some really amazing answers to that there's also a lot of biking question elements to this sequence yes excellent okay whilst in the industry do you respect as a thought leader who comes to mind other than you and i have to have a pleasure company by the way your newsletter is one of my top reads i think that's yours and uh and ben thompson from trajectory oh my god two of the two of the ones that i think they both have a very like natural instinct for for writing and synthesizing things that people are feeling without um you know with a with a clarity that's really helpful so i i i really appreciate that i mean i appreciate that i think the rituals process is exposed to some really amazing leaders talked about ariana i always learn a lot when i when i talk to ariana shresthoshi has contributed a bunch i think he's uh really helpful and is like is i i can't believe he basically had no twitter followers at the beginning of the pandemic and like his tweet fights are just total gold and this is all so insightful i've all put together uh fiji simo is someone that i learned a lot from she now runs instacart daniel from spotify daniel act i think is is uh he's got this really unique way of thinking about the world and he's also one of the few people that can like hold a very long-term view and a very short-term view at the same time fantastic ethics i'd also say my entire board reed uh v hoffman ayman taneja i think they're i think they're all amazing and i'm super lucky to have you know a group of people i can call with questions awesome someone's gonna have a lot of work in these show notes yep that list okay final final question what's your go-to karaoke song or dance move at a wedding a karaoke song is if i had a million dollars by the barenaked ladies and uh it's a uh people don't remember the song part of the reason it's my favorite is i'm a very mediocre singer and you don't have to be that kind of singer and everybody can sing along so you can kind of bring everybody into it and it's just such a fun song if i had a million dollars you're as thoughtful about your karaoke songs as your about everything else sure thank you so much for being on this podcast you've set a really high bar for ceo guests so we'll see who comes up next two file questions where can folks find you online if they want to reach out or learn more and how can listeners be useful to you okay i'll give the same answer to both well you can i'm easy to find one of the benefits of having a not very common name it's uh easy to find me on basically every platform so you can find me on twitter it sees that dm me she shared io get right to me but in terms of being useful to me and also finding me i would highly recommend joining the rituals great teams brain trust and i think it's a pretty fun experience to get a chance to contribute to a book like that and hopefully you've made it up this far on the episode then you probably are interested i think you'll find it interesting and i'm having a lot of fun with the with the people in the braintrust amazing i'm definitely going to join the guest here all right thank you so much lenny that was really fun thank you so much for listening if you found this valuable you can subscribe to the show on apple podcast spotify or your favorite podcast app also please consider giving us a rating or leaving a review as that really helps other listeners find the podcast you can find all past episodes or learn more about the show at lenniespodcast.com see you in the next episode