eBay Baby! How eBay Is Working for Developer Speed

Key Takeaways

hey everyone welcome to another pancake breakfast today we are with a team from ebay talking about how ebay is working for developer speed and joining us is randy shoup vice president engineering and chief architect at ebay mark weinberg vice president core product engineering at ebay and lakshmi dry vankatash excellent so it's a pancake breakfast day it's also april fool's day but i've got pancakes i'm not fooling around here there's no food around here so who's got their pancakes today or at least something coffee ah coffee right on and a spatula well does anyone have any syrup okay i'll just like i'll i'll go off camera to get my syrup i got my syrup i'm gonna have my pancake a little bit later but let's just get started right away with the discussion here so we're going to really look at ebay and we really encourage your questions for the ebay team here you can ask your questions in the youtube comments the ev the ebay team will help will gladly help answer them and so we're going to talk about developer velocity and this is something that uh ebay of the ebay team has been working on quite a bit and so you're all leading a tech led re-imagination at ebay and what do you mean by that and why are you doing that luxury maybe you could start all right hello everyone thanks for having us as part of this show ebay has always been a technology companies ever since its inception 25 years ago but over the past few years um our investments in technology is not as much as we would like it to be so as a result of this our engineering velocity is slower than we want so this year we have renewed our focus in our technology investments given the scale of ebay with about 185 million buyers 19 million sellers and over 1.7 billion listings it's important that we are able to move faster we are able to do things cheaper and better so engineering velocity is so critical to do that is not only critical for enabling our users and businesses and delivering value for them faster it is also important for developer morale hiring and attrition so we are very happy to be here talking about our technology investments and how we are doing what are some of the things that we are doing to improve the velocity for our developers great i want to remind everyone that you can ask questions in the youtube comments and the team here will gladly help answer them so what does velocity then mean at ebay these days mark i mean randy i asked i want to direct this question randy what does velocity mean to you at ebay how do you know if you're really making progress what are some of the ways you're doing that yeah great no thanks for that and like lakshmi said yeah we're trying to figure out how we can improve our ability to deliver value to our customers to the to the buyers and sellers uh so we're definitely so in terms of developer velocity we're definitely taking a page out of this book so nicole forsgren's accelerate uh book uh if you haven't read it uh when you're finished with this podcast go buy it and read it please because it's amazing um and it summarizes a whole bunch of years of the state of devops surveys and the state of devrops reports seven or eight years of um looking at uh all sorts of organizations 31 000 all over you know all over the globe uh and trying to figure out what makes high-performing organizations work well and the four metrics that they mention which we are using are uh deployment frequency so how often are we deploying software uh to our customers uh uh lead time for change so how often how long does it take when a developer finishes committing her code to its live on the site providing value uh change failure rate which is what's the percentage of time that we are when we uh make a change do we have to roll it back or hot fix it or something like that and then mean time to recover when we have one of those incidents you know how long does it take us to get back to a good customer experience and so we use those the four key metrics essentially to to measure ourselves um and uh and people who are familiar with the book are familiar with the research maybe know that um different organizations kind of end up clustering so uh the high performers and the elite performers tend to be really good on all the metrics whereas the low performers tend to be poor on all the metrics uh and for us we're kind of in the middle uh we're trying to get more get more high performing and more elite performing so like we're trying to get the deployment frequency down to be more often we're trying to get that lead change fail uh lead um change time uh down as well and then hold constant you know our pretty good ability to do uh uh make updates to the site without failures and to and to recover from them so that's that's what we're doing so straight up following the the devops research marker lushma would you add anything to that is there any other perspective you bring to this that you would like to add no i'll just uh you know echo what randy said like we're really adopting the accelerate principles and we believe in the research and we're excited about the the changes that we're instituting and you know the effect it's going to have on our on our developers and our customers how are you doing those metrics then are you know you say you're about middle of the road how have you gotten to that point what have you been looking at well we've been so we uh um we we tried to we looked end to end at the um the sort of entire product life cycle from idea all the way through to providing value and you can sort of think of it as like there's like idea to project right so how do we go from an idea to some so we start working on it and how do we go from project to code you can call that software development how do you go from code to uh you know useful feature you can call that software delivery and then there's a downstream like iteration with experimentation and you know getting gaining customer feedback and the honest answer like lots of big companies like we have areas to improve along every one of those you know kind of four phases but what we decided to do was put laser focus on that software delivery point and the thinking behind that is because if we're not able to we need to be able to deliver software like quickly repeatably and reliably to do any of the upstream you know architectural improvements we'd like to make uh security uh patches like all the things that we that we want to do ultimately end up uh you know requiring that we deliver software uh and then that's also you know um needed for being able to do better on the downstream you know experimentation and feedback loops and so on uh and so the key insight that we that we uh did by working with a bunch of individual teams and understanding what was blocking them was like focus laser focus on that software delivery part yeah i'll just i'll just add to that you know like we uh our general approach to was sort of looking at small small wins and long-term capabilities so you know if we could if we could save an engineer 10 minutes on a task multiplied by that engineer doing that task multiple times a day multiplied by every day of the year multiplied by every every engineer in the entire org that turns into actually quite a big win but then there's also some longer term capabilities that we know that we need to deliver on things like re-architecting some of our code um those are those are bigger changes bigger impact with bigger risk but we know we need to do that and and sort of the other thing that we did is we decided that you know we couldn't we couldn't just work on everything all at once so we're going to identify a set of pilot domains we called them which are areas of the product and then and those those pilot domains are really the the places where we thought these are the biggest opportunities for impact from this work and an roi and it's also sort of we picked the teams that were most aligned with the mentality of doing this the hunger for speed the teams that had great leadership support but then we just kept narrowing our focus even further we said okay within those domains let's pick a set of pilot apps that maybe are the most active apps the most affected by lower velocity maybe the the apps that had the most dependencies on them or were most dependent on other apps and then you know for each one of those then we asked the team who said okay well what are what are the list of impediments you have to going faster is it manual processes is it you know the tools the environments the knowledge and skills of the team and that led to us identifying a bunch of platform tracks to to go after things like builds build times our server start up our test automation our ci cd pipelines and and then the teams work collaboratively to kind of identify a problem area root cause it come up with a fix and apply it to that pilot app and then once it's working then we can take that that solution and apply it to the other pilot apps and domains then ultimately down the road we'll broad it to apply it to you know all the places in the company so mark i'm going to go back to you again and and ask you about that approach and how you really how do you decide what to work on you know and what order you know you looks like you have an idea on the teams and how and who you're approaching but what is the order of that work well it's it's sort of what randy was saying you know we we are you're mostly focused on development and delivery and we we have um areas of opportunity and outside of that but that's really what we chose is we said hey for each of these pilot apps let's really focus on our developer productivity make their their their tools better so as an example um in just the process of sort of root causing some things we found a place where one of our top test automation frameworks that uses a jvm was spawning a separate process for each test for each jvm and just a simple change to a config file took to our our pr validation time down from like well over an hour down to around 15 minutes so you know we're looking for those kind of small wins which ladder up into big wins for us when they're they're cascaded out through the rest of the organization and then like i said you know like you know we we know that we have places where we have to work on our architecture um just refactoring some of the code improving things like our our test automation for sure is a big area um so we've been sort of systematic of you know on a sort of a team by team basis looking at what is that team's biggest impediment impediment to going faster and just really attacking that what are some of the things that you found um i know you know i would say sort of the general theme is automation i think you know really attacking every place that we find that that we have a manual process any place that there's any kind of human intervention really sort of digging into those areas and and figuring out what do we need to do to automate and then within that i think uh test our test um as we improve our our ability to test the software faster better it's gonna instill more confidence into the team and you know how aggressive we can be about rolling changes out and so you know really digging into those areas and just more specifically we found a lot of places where we have inverted test matrixes where we don't have enough unit tests there's a lot of reasons for that we have some legacy codes some monolithic code as we re-architect that code we can shift from you know more expensive integration and manual end-to-end testing to you know higher quality faster running automated unit tests so that's a big area of attention for us those automated unit tests have become if i have become increasingly popular haven't they is that is that an adjustment for people to start getting used to those things or are they are they just are they hungry for it randy do you see anything there uh absolutely and you know what the best example is the re-architecture effort that lakshmi's team is is working on so if uh i'd kind of love to like pivot uh uh to that because that's actually because like one of the main things that uh is enabling uh lakshmian team to do to move forward is exactly like modularization and test automation but i don't want to take that away yeah great luxury i'd love to hear your perspective it's great so um as mark and randy said right this is a huge initiative in terms of bringing velocity so we're focusing a lot of the low hanging fruits where you're taking the existing code basis and see what we can do in terms of tooling automation processes and how do we increase the developer velocity while we are working on the long term which is re-architecting some of our core areas so starting with the critical one which is view item this is where our customers come and make their purchase decision so over the years even though these are all service space some of these services have become bigger and larger and more ethic so we are working on um componentizing it modularizing it and making it micro services based architecture and of course it takes some time to get benefits out of this so this is one of our huge investment that we are doing with respect to critical areas the second one is on our native um platform so our native foundational architecture again we keep re-architecting this to the latest patterns and frameworks so we are doing a huge investment in our native foundation architecture and also the experiences that sits on top of it we are also re-architecting those codes as well um and then our one of our strategic initiative is focusing on verticals so there as well whatever we are building new those vertical platforms and capabilities we are building it in a way where we can very easily and quickly expand to other vertical categories um and also the to other other markets so the huge focus on multi-dimensional long-term focus can you talk about that that that architecture that native architecture and a little bit about it and how it's transforming so um as as as i talked about the view item where it over years it has become monolithic um our native foundational code base has been there for like a decade so some of the foundational pieces have been like you know we started developing over and over on top of it so it has become very slow for us to make a change in those things and roll out faster so given that our market share is moving more to native um we wanted to move faster on that so we just cannot rewrite the experiences without fixing the foundation so that's why we are focusing on those and rewriting those more into componentized modularized way where teams can work independently instead of basically removing those dependencies between teams as well is that more than i guess that means a container-based architecture sorry i want to we we use native to talk about mobile applications just to be super clear which when when laxmi's saying native that's the ios app the the android app android app yep okay okay so then the the underlying architecture at ebay this is a separate matter then we're okay um then what what are some of the improvements that you've seen then lakshmi what are some of the you know the changes and improvements that have been apparent um in terms of the of the re-architecture we honestly we are still starting so we'll start seeing some of those benefits around mid mid of this year like around summer but then in terms of the um the low hanging fruits we are already seeing lots of improvements the one thing is right it's not about how many improvements we are pushing it is about pushing those small improvements where the developers feel feel inspired they know that you know what i can do things better it can we can move faster so it's inspiring engineers to do things better so it's about shifting that cultural piece of it the mindset the comfort of people okay i can do this as well you know and that's a big part of it you know so i'm i guess the question i have for you mark is what are you doing to improve eba's ability to deploy software what about that testing that mirroring the canary environments that you're setting up yeah like i mentioned earlier just a real focus on just test in general because you know the the rate at which we deploy goes up proportional to how confident we feel in the quality of the software so i mean we're doing lots of things again you know getting better test infrastructure in place we're using techniques like doing contract testing which allows us to sort of break break some of the tight team dependencies we've had in the past which which enables us to block a bunch of weight states and just better independence you know putting a lot of scrutiny on manual release processes that we have with these dependencies a lot of times a team will need to check whether code that they want to deploy is working well with other teams as code and so you know getting a process in place where all of those tests are automated so partner teams can run them independently and if there's no failures they can just deploy without you know a team having to sign off so a good focus on that we're we've got a staging environment that we use that we put a lot of effort in trying to make that as close to production as possible with better data that's clean data from pii perspective with better code where teams have a last known good component in staging treating it more like a production environment with sla so if there's a problem a team jumps on it and then like you mentioned you know using techniques like traffic mirroring where we can simultaneously route traffic um both to our live service and an n plus one service and then canary testing which allow us to take a new service and route live traffic to it um with small amounts of traffic and then monitor it and then if everything looks good to progressively roll that out to more and more traffic so um you know a lot of techniques like that but then also there's some cultural things of just sort of just committing to faster deployment saying hey you know we're going to deploy daily what do we need to do that or we're going to do weekly deployments on our mobile stack what do we need to do that and just you know the teams are amazing they find ways to solve problems once you set a goal like that and so we're already seeing changes and just how quickly we can deploy both the services and the mobile apps so how would you compare the confidence today to six months ago a year ago oh i think there's a lot of confidence in what we're doing a lot of belief in in the research that randy mentioned earlier um we're again we're you know a little bit of progress goes a long way you know you have these you know i was talking to somebody about this and when randy and i started this initiative it all seemed like theory and the comment was in our just sort of latest progress um that wow theory to reality so the the theory has become real and i think the more people see that it builds on itself and it's sort of taken on a life of its own now we we get reports all the time of teams sort of doing some things and reporting like the impact that it's had and i i mean it's not unexpected but it sure is nice when that happens i guess i would add that it's like think big start small learn fast you know uh so we actually you know we thought big we looked at the big picture trying to see the whole board of all of all the issues you know across everything start small so like focus very you know laser focus on that software delivery and a little bit the software development parts of it as mark mentioned but then also don't try to do our 4 500 apps and services and 3 000 developers all at once like choose specific uh pilot teams as mark mentioned specific apps within those teams and then uh that set of like hor and giving them horizontal help so like help as mark mentioned around build time around startup time around pr validation time around automated testing in the pipelines around again that mirroring capability around canaries around automated deployment so like all these things are our horizontal capabilities that we're using in conjunction with these particular you know domain level uh teams that we're working with and we were able to work in super tight feedback loops with them because we do it that way so this has a lot yeah i just want to add that it is just not about fixing the current things um but also a lot of the new developments given that these four metrics are super clear um all the new developments people are very excited about like building it the right way from the get go so mark and randy has they haven't seen it yet but that's what at least happening with all the new developments so i would love to turn it back to you lakshmi about how culture plays a role here uh what are you doing uh about you know enhancing the culture you know how do you think about share goals and collaboration and breaking down the silos right so i think in order for any of these investments or key changes like this to work it's critical that all the teams across ebay needs to work towards a common goal and a shared goal right and we are clearly seeing a cultural change that's happening where people are thinking about end to end and they are like consciously breaking down silos it's not easy but people are like talking about it and also we are seeing that everyone is thinking about okay what is the outcome that i want to achieve for this particular initiative and how can we work together towards that particular initiative so there's a lot of change that's happening and we are clearly seeing it and so then then to follow up on that you know people need to be filled again that but they have to have that confidence right so they need to be feel like they're safe they need to feel like they can raise questions that they can uh be confident about their you know dependencies what are these factors that also play into it lakshmi yeah so i think um as leaders it is important that we provide that environment where there is a psychological safety and people can raise issues impediments um if there is a problem that they need to raise with other teams they need to raise it so initially it is important that the leaders are there and then encouraging them to talk about it in broader forum then once we give that psychological safety and raising issues is okay and we want the you know we want to shine the light on the problem that people are getting comfortable again it's a journey but we are seeing that kind of change happening and the biggest thing that we are doing as leaders is providing that environment where people can speak up and is rewarded for speaking up yeah and if i could just add to that like the two thing just to echo the two things that lakshmi said which is shared goals right so it's not like lakshmi has some goals and mark and randy have some goals like we didn't talk about where we work in organizations but we work in different parts of the organization let's just start with that uh but we all have the same goal which is uh you know making in particular you know lakshmi's team a lot faster and you know other teams that we're working with as well so like shared goals from the horizontal organizations from the from the domains start with that um and then type feedback loops right so a bunch of people on my teams in the horizontal organization have embedded like directly with uh with lakshmi's team to help with the re-architecture to help with the improvements on the on the mobile apps uh and like making that investment again to tighten to tighten that feedback loop uh it makes things move faster and then to lakshmi's other point it's making making it safe for people to say so when we go we want you to go you know we want you to do daily deployments or you know 20 times a day or something like that and then people get all nervous we're like okay why doesn't that work tell us and like i want to hear it you know we want to hear it all so like oh well we can't do it because this reason this reason this reason this screen we're like great let's let's knock them down all together and right in a bunch of those things you know can't be solved maybe just by one team acting by itself but you know solved by working together and like that's the that's the flywheel the cultural flywheel that gets this stuff going i want to encourage people to ask a question here we have some real experts here uh you know and the three technologists from ebay we had you know randy uh mark and lakshmi one of the questions that that i always have is about the teams and you know the individuals are important the culture's important but how how have the technologies affected the teams and how the team's been you know affected the technology you can kind of look at it both ways where you know you have all these technologies now you can learn how to use them to automate but often you know you need the team has to have that right experience like to be able to use the tools and not all the tools might you know be something that you know someone just starting out may be familiar with so i'm curious about that you know that back and forth and how do you think about that team development yeah we we we really internalize this we really want to instill a culture of continuous learning and improvement for the team and so we talk a lot about that and make sure that people understand and feel like it's okay that you don't have to know everything like and and we're here to help you learn and get better so we started doing a series of technical talks for our engineers and just on a wide range of topics things like you know test driven development dependency injection small batch sizes and the importance of using them and separation of concern and they're like they're it it's really been fun to watch people really uh get get a lot of value out of that and learn and people are excited they ask like really good question and then things like like randy mentioned embedding our are some of our senior technical leaders within the teams doing things like pair programming where you get sort of that one-on-one attention and people really can you know learn learn these topics in a meaningful way and then we do things like we have an internal conference that we that we present every or we have hold every year and it's kind of like an opportunity for engineers to present in an external conference-like atmosphere um on a wide range of topics they choose the topics and so engineers can go and learn from their peers on on a variety of different things that that are outside of maybe what they're looking for um and so that's on the engineering side but there's like a real hunger for understanding these principles and problems across our executives our business people our design team marketing people and so you know randy randy and i are constantly doing updates uh for the company uh at all sorts of different um meetings we have a monthly operating review that we we talk at we do speaker series for the entire team and then various group all hands uh meetings on these topics so that people can learn and there's just a real appetite for understanding some of these things and why they're important that's great um that's so important we were talking to adidas the other day and they have a mentorship program uh there and they talk a lot about these things these these same things about you gotta think about the technical aspects of the cultural aspects and the strategic aspects and the strategic aspects then carry into the business side of the you know of the organization so i'm wondering about this issue though then about how you help people feel psychologically safe you know blameless postmortems for example randy how do you think about that yeah so i mean so uh uh one aspect of psychological safety is like we were talking about raising impediments like hey you're asking me to you know to go faster i'm i'm uncomfortable uh but like say making it okay to say uh you know even to people with fancy titles uh you know here's here's why um here's what i need but yeah the other thing is when stuff goes wrong uh and things go wrong because it's a complicated system and software's hard uh at the end you know once once we're out of the woods and we've restored service to customers like we want to have a blameless postmortem and you know that's not something you know we started that's that's something that you know has been done for the industry uh for for years and years and comes out of you know safety culture in a bunch of other safety critical industries like uh or uh like medical or fire safety or transportation safety there's just everybody has like co-discovered this idea even with different wording that rather than blaming people and you know trying to figure out what you know whatever who to make feel bad or what throat to choke instead of figuring out hey let's get everybody together that was involved or affected i'm like what could we do better next time you know i wish we didn't have that thing but uh but what can we do better next time all together and that again in all these in all these industries and again by the research uh that's what that's what unlocks the culture of everybody now thinking forward and thinking about what can we do to improve things altogether and you know all these all these aspects of psychological safety are are important and then the other aspect of course of psychological safety is uh you know respect for diversity respect for diverse opinions that's a critical that's a critical thing in every in every organization there's a bunch of recent research that shows that um diverse teams make like order of magnitude better decision making it's like 80 if you have you know diversity among along a bunch of different dimensions including gender and geography and age um you actually make something like 81 better decisions uh so you know that's the other that's the other aspect people of psychological safety it's like being able to bring your whole self to work and uh and um that benefits everybody including frankly the business so uh we believe in that too so luxury how is this a you know i just would love to end it on you unless we do get a question we think we have a pretty shy group here but i'd love to just end it on uh with you and ask about how this has affected your view on strategy how's this affected your view on the evolution of of architecture at ebay i think this this change um when this investment on technology was introduced and martin randy took this initiative around velocity um initially it took a little bit for us to kind of believe that it is true because in the past like any other company we start something we stop it and then we started and stop it because we want to focus on features to deliver for the customers but then after like couple of months we see that okay it is here to stay so there is a lot of um you know excitement um from engineers that i heard that oh this is amazing finally we are focusing on this and we also saw a lot of innovation coming from engineers about hey here are the problems here is how we are solving here is how we want to do like front and automation so there's a lot of ways people just proactively came and talked about these are the things that we want to do and they also come and present to mark and randy a lot of things say here are some of the new ideas and it's back and forth we get feedback from randy and mark so there's a huge shift in on the belief that we can do software in a better way and it is it's basically inspiring engineers you know right when they in the engineers are inspired they will like they will just give everything that they got so the key is like that here it is here to stay and it is not like just a thing for two months and so people are like doing a lot of things that will that we will start seeing probably from summer to the end of the year so it's amazing that we're focusing on this well i want to thank all of you for your time today this has really been enlightening to hear about how ebay is thinking about getting that velocity and improving that velocity so thank you very much for for joining us lakshmi mark randy appreciate it thank you thanks for having us thanks alex