Building Machine Learning Teams with Josh Tobin (2019)

Key Takeaways

okay so welcome back the main focus of today is going to be on research areas which which we have an amazing guest speaker to talk about but before we dive into that by popular demand there is a request to talk a little bit about machine learning teams and so I'm just going to spend a few minutes on that and one of the focuses of this lecture is on kind of what are jobs like in this field and what is hiring look like so can I just get a quick sense show of hands like how many people are you know interested in some way or another and looking at a job in machine learning engineering at some point you know next year next six months okay so a good number one thing we're thinking about doing is bringing in some industry people who are looking to hire people to the project presentations at the end of class to just give kind of short presentations on what they do and what they're looking for and people on their teams how many people would be kind of interested in hearing those presentations if we did that talking about machine learning teams and there's a few topics I want to cover the first is just what's the context that we're working in here and so what's and the context is mostly defined by the gap in AI talent and then I'll talk about the different roles related to machine learning that we see in industry often then I'll talk a little bit about like how those roles are typically you know formed into teams and then what the hiring process often looks like so the AI talent gap you know one question you might ask is how many people are there out there that know how to build sort of high-quality AI systems and there are different estimates that you can find of this want to submit the löwe assessment that I found was 5,000 and so that's people who are actively publishing research according to element AI so that's kind of like a lower bound on the number from the same report you know the people with the right skill set whether or not they're they're publishing is more like 10,000 Bloomberg estimated that they're more like 20,000 PhD educated AI researchers now not everyone working in AI needs to have a PhD obviously and so you know you can think about what's an upper bound on that number and element si estimated at 90,000 and a different report estimated at around 200,000 and you know but the the point I want to make here is that comparing this to the numbers for a more mature industry like software development you know that's that's more like 3.6 million people right so there's still a huge gap a huge difference between the number of people that know how to do AI and the people that know how to do software development and that number is even bigger if you look at the entire world so what is the implication of this right like what is this what is the shortage in AI developers mean for AI hiring you know in short it comes down to a fierce competition for AI talent and this is a quote from Bloomberg everyone agrees that the competition to hire people who know how to build artificial intelligence systems is intense it's you know turned academic conferences into frenzied meat markets for corporate recruiters and driven the salaries of top researchers to seven figures right and for those of you who have been to machine learning conferences in the last couple of years you know I think you would agree that it's like there's very much a feel of like massive competition to get the top people so we talked to a couple of people who were kind of running startups and hiring for them in the process of you know making the the full stack deep learning class that we did and a couple of quotes that stood out to me from a computer vision engineer at sort of a late stage startup you know hiring is crazy ml got popular really quickly there's a ton of demand not a lot of supply hiring for ml is really challenging takes way more time and effort than we were expecting you know we're only able to get a few people per quarter despite our best efforts and this is just kind of the the general story that we heard from most startup founders and people and big companies that we talk to it's just really really hard to hire for these positions so what are these positions the most common roles at companies that we see that involve machine learning are sort of DevOps engineers data engineers ml engineers mo researchers and data scientists right so these are all like these bugs word terms and you know what is the difference between these like what are what are these actually mean in practice and so you can break down each of these jobs by sort of what the role that they typically play in teams are so DevOps engineers you know their their job is really deploying and monitoring production systems and so what they're producing like what their work product is is a deployed product so something that's you know actually on your phone or on your computer and is being served to end customers and so they're using tools like AWS to actually serve predictions data engineers are typically focused on building data pipelines so you know collecting data figuring out how to store it figure out how to monitor it figuring out how to serve it to machine learning workflows and their work product typically looks like you know a Hadoop distributed system or something like that next we have ml engineers and you know their their goal like this is sort of a pretty broad role and what they're tasked with is most often training and deploying prediction models so there their goal is like you often they're given a data set and what they're trying to do is produce a system that can create predictions and then either deploy it themselves or hand it off to someone like a DevOps engineer to deploy it for them and so these are people who are working in you know tensorflow but then also tools like like docker because they're often pretty close to the actual deployment then you have ml researchers and you know this there's sort of a fuzzy line between ml researchers and ml engineers but mo researchers the the biggest difference is that often the prediction models that they're training are not production critical or they're not production critical in you know in the in the short-term but they're often working on sort of more speculative products or projects rather that could make it into the product at some point but are not going to be deployed tomorrow and often the work product for these for these folks looks like you know actually a report that describes what they tried and you know what the results look like and that might get picked up by the rest of the team to deploy and then you have data scientist and the tricky thing about data scientists is that this is really a catch-all term that can mean many different things in different organizations and so I've heard this used to describe any of their roles above and you know in some in some organizations and many organizations in fact data scientists are actually not doing much machine learning at all it's more of an analytics role and so this could mean many different things and so you know what you might might ask yourself if you're on the the job market or if you're hiring for these roles is you know what skills do you need in order to succeed and so on this chart we have sort of the level of machine learning skill that you need on the bottom and the level of software engineering skill that you need on the left side and the size of the bottle the bubble corresponds to you know how well you how good you are communicating and technical writing so how good you are at sort of convincing the organization of your results so we're talking about each of these individually ml DevOps high on software engineering relatively low on machine learning this is primarily a software engineering role and oftentimes these folks don't have any machine learning background at all or are sort of in the of learning machine learning and you know want to work closely with an ml team but their skill set is more on the software engineering side data engineer is pretty similar mostly software engineering but there's often a slightly higher requirement for ml knowledge because their work product is so closely tied to what the machine learning team is actually using ml engineers this is sort of I think like kind of one of the unicorn skill sets from a lot of companies that we talk to a lot of people said that this skill set is particularly hard to hire for because it requires sort of a rare mix of you know machine learning skills knowledge of algorithms but then also software engineering skills and so you know if you if if you're kind of if you're a really good software engineer and you're also sort of pretty far along in your learning of machine learning this can be a really interesting role to go for ml researchers you know this is your your ml experts this is kind of the one role we're still see most people have sort of a master's degree or a PhD in CS or stats or did something like the you know the Google brain residency but you know I think this is starting to change but you know of all these this is this is the one where kind of academic background still matters the most and then you know data scientists again since this is kind of a catch-all term it corresponds to a pretty wide range of backgrounds one common background that I've seen is for people who have you know some other sort of science PhD like maybe they did a PhD and you know in in chemistry or something like that and they want to move into the tech world this is kind of a common role that people with that background go into ok any questions about kind of the landscape of the roles and the skill sets that you need for those roles all right the next thing I want to talk about is like how does this all fit together into into teams so what do teams look like that are working on machine learning projects and you know the main thing that we learned when we you know talk to a bunch of people about how they're organizing their teams is that there isn't really a consensus yet on the right way to structure a team that's working on machine learning projects but there are some some lessons that we learned that I think are worth sharing so the first thing is that a lot of the people that we talked to mentioned that it's really important to not just have ml teams that only know machine learning like if they're working on stuff that's going to be deployed into production almost everyone thinks that it's very critical to have software engineering skills on the team where people didn't agree is whether that can be a mix of people who are really deep in machine learning and really deep in software engineering or whether everyone on the team needs to have a mix of both skillsets we talk to people who have both of those views different teams had a different view of machine learning researchers so we talked to some people who said yeah you know machine learning research is exciting but I don't really like to hire machine learning researchers because it's just too hard to integrate them with my software team in my product team and so I don't actually get useful work out of them other people who you know maybe are working on more frontier technology you think that it's actually really critical to have you know at least one or two people on the team who have deep ml expertise because you know the field is moving so fast you need people who can keep up with it and make sure that you're always doing things that are up-to-date there were also different views on data engineering and so this kind of tends to sit in different places within the organization in some in some organizations it sits within the ml team and so the rationale there is you know the the the output of this team is the input for the machine learning team and so these two functions should be as close together as possible right like the data team's work product is directly used by the ml team and so they should be you know their desks should be next to each other other organizations have it as a separate team which is often called data warehousing and then another thing to note is that a lot of organizations that we talk to think that it's important to have people who are dedicated to data labeling and this doesn't necessarily mean people who are labeling data themselves it could just be people who are managing the out sourced data labeling workflow because it's such a critical part of what the ml team needs in order to do their job well all right I want to say a few words about managing machine learning teams and you know that the point that I want to make here is is just to point out some of the challenges of you know why this is often more difficult than managing typical software teams and I don't really have all the answers here but you know I think it's helpful to to sort of understand why this is a particularly difficult problem and you know so the first reason is it can be really hard to tell in advance you know for this particular problem that you want to work on is this a really hard problem or is this a really easy problem and I imagine some of you are maybe feeling this way about your projects right now some nods yeah okay and I like these charts that that Lucas already showed you but this is from a a I think was a Cabo competition that he ran where they looked at accuracy of you know model on some data set and they looked at how performance is improving in the first week and you know it's like great accuracy has gone up from you know 35 percent to 70 percent in a week it looks like this problem is just gonna be solved you know in the next couple of weeks but then when they looked at performance throughout the three months of the project it turns out that this you know there's really fast growth at the beginning was actually just ramping up to a plateau and the accuracy didn't actually really improve that much beyond that you know despite the fact that the you know the amount of effort going into it was increasing dramatically right so you might have seen this first chart in the first week and thought hey this is a pretty easy problem and we can you know we don't really need to step up this team too much you know it's just gonna be solved in the next couple weeks and then been disappointed three months later so it's a it's a tricky situation as a manager a related problem I think is that machine learning project prior projects and the progress on them tend to be very nonlinear and so this is like kind of in stark contrast to software engineering projects right where it's you can generally tell on a week-to-week basis have a reasonable estimate of how much work you can expect to get done but in machine learning projects often stall for weeks and and it's often kind of very unclear exactly what things are going to need to try because you haven't hit the roadblocks yet so it can be difficult to paralyze things and estimate project timelines in the early stages of the project because you know you kind of need to just start trying things and see what works before you know exactly what you're going to need to invest in and so one way I like to think about this is you know even in an applied machine learning and machine learning that's going into production it's still not like in my opinion totally an engineering discipline it's still somewhere in between research and engineering another common issue that we we've seen in talking to companies about this is that there's often a cultural gap between people who you know whose background is research and people whose background is engineering and this can manifest itself in different ways just from you know sort of different values different kind of outcomes of projects that they care about different things they're excited about working on and you know in like the worst examples of this what this can look like is you know either side thinking they're better than the other like research thinking you know we're the ones who are working on the really challenging part of this problem or engineering thinking like oh we're the ones that are actually building things that are useful and so this is this is a problem that crops up in some organizations and it takes conscious effort in order to make sure it doesn't happen and then the last thing I'll point out is that we've heard from a lot of people actually is that you know leaders just often in your organization often just don't get machine learning and you know and I think this can manifest itself in them like kind of not understanding the difficulties that better listed above but this is this is a challenge that a lot of people who are trying to start machine learning efforts within larger organizations complain about and so I think it's just important to make sure that leadership understands why the problems that you're working on are kind of uniquely challenging all right any questions about machine learning teams managing them I think I've like pointed out problems here but not suggested solutions so also if you have any solutions I'd love yeah could you spend a little bit more I think for me personally like trying to learn there are limited resources on how we should do your machine on a project that how should to be in a testing there's like yeah I think a lot of the like much of the standard software engineering toolset is helpful like I think most machine learning teams are are doing a lot of building their own tools you know and there are companies out there that are that are trying to make sure this doesn't happen anymore but you know the current state is you know many people are still doing this and so if you're good at building tools that can be used by you know machine learning engineers or machine learning researchers that skill will come in handy I think the other big thing is like sort of distributed systems broadly like being able to paralyze the workflows and paralyze sort of data pipelines and things like that is just like really deeply useful that's like the one particular thing I think that people look for anyone managed machine learning team yeah how do you deal with these problems like have you seen these yeah sorry yeah is there anything in your experience that's been particularly useful like very a very reliable motion right personally everything but at least they use it eight points landed I think that so finding someone who can bridge the gap between the technical work and then the business outcomes that you're trying to get to how did you find that person and then we had business leaders did I use kisses got it so there's so you found someone who understood the technical challenges from previous life and then was willing to sort of translate that yeah and then where there are there differences between how you think that works how to tell a good story about a machine learning project versus a general software engineering project don't really fight that much of a difference in there yeah but you just need to be complicated many of these problems when we say that for us Alicia when prediction is telling something else but the common sense is come but it was hard for economy even in the trust this is yeah any other thoughts on this part I think this is like sort of one of the hardest challenges in building machine learning products right now I'm the data scientist and I feel he's really hard to make tickets in the classic gira style yeah we've shot a lot of things at opening I and what's tended to work best is just kind of like the simplest and lightest weight things like even just a Google Doc where people sort of talk about what they're planning to work on like on the timeframe of say a week and then check-in after a week to see how things go one of the challenges is that is that you know I think in a lot of people who come from software engineering backgrounds kind of expects if you said you're going to get something done this week then it's going to get done this week and it doesn't at least in my experience has not always worked out that way for machine learning teams because and you know often the thing that the thing that you tried doesn't work or you encounter some difficulty in implementing the thing that you didn't foresee when you decided to work on it and so I think one thing that's like one mindset that's been helpful has been to thinking think about measuring people's sort of inputs as opposed to their outputs so I think a big driver of how much progress machine learning organizations are able to make is just how many ideas they're able to try right because every idea has like some probability of success and like even you know the most brilliant machine learning researchers like are only right you know I don't know 50% of the time is that maybe that's generous probably less yeah and so you know if you're able to try 10 things in the tend the timeframe that it takes someone else to try two or three then that's that's a huge advantage yeah yeah so number of ideas that you've tried I think is a really important one I think there's like there's some time scale like under which you have to sort of be able to make things work and I think it depends a little bit on the context of the organization how long that is so you know like I think in most organizations if like if a researcher for example has not tried any ideas that it works over the course of you know a couple of years then seems like that's probably bad but you know maybe if it's three months then it's like it's pretty reasonable to not have any ideas that have worked in that timeframe another thing I think is that's important is just like how well they're able to sort of connect their ideas to the rest of the team in the organization so like I think one failure mode for integrating researchers with the broader teams is that they kind of work on their own things and you know even their best ideas never really make it back into the into the work that the rest of the team is doing so I think having metrics around that could be valuable as well great the last thing I want to talk about a little bit is just mo hiring so you know a lot of you are thinking about looking for jobs in the next year so some of you maybe are looking to hire people so I think a lot of the places to look for m/l and data science jobs are you know just the typical things that you would think of so you know applying directly to companies you're interested in I think a lot of people in general think that this doesn't work one thing that I've noticed is that in machine learning since these roles are so hard to hire for sort of the direct approach can actually work quite well you know if you're in school on campus recruiting if you're more experienced LinkedIn recruiters ml conferences are great places to go look for jobs a lot of the big companies are there kind of with booths and they're very actively hiring at those events and then and then lastly you know as I mentioned we're gonna bring in some some folks to do some work who are looking to recruit in the project presentations at the end of the class so connecting with those people could be helpful so what should you expect in the interview process I feel like software engineering interviews are sort of very well-defined at this point and you can kind of buy a book that tells you what types of interview questions to prepare for and you know you can Google interview questions and you can figure out exactly the right way to solve them the interview process for machine learning roles is much less well defined at this point there's a lot of types of assessments that are pretty common but it they are mixed in NASH in different ways here are some of them I think the so the the first few of these are let's see the first three of these are just sort of general software engineering interview type questions that you might get in terms of ML specific stuff a couple things that I've seen that are interesting pair debugging on ml specific code so sort of like the debugging exercise that we did a few weeks ago it's like hey here's some buggy code let's sit down together and figure out where the bugs are you see a lot of math puzzles a lot of linear algebra puzzles take home ml projects like here's a data set how well can you do on this data set those are pretty common apply to ml so you know here's a problem that like may be one that's related to the team is working on you know tell us how you'd approach solving it what data would you be looking for what type of model would you train it's a very common type of of question people will also often like to try to dive deep into your previous mo projects and sort of probe at the assumptions you made and why you made certain decisions about like what architecture you chose or you know how you know how you like what your goals were for the project and things like that and then m/l theory questions are also very common so you know explain explain the the bias-variance tradeoff what's the difference between overfitting and underfitting how do you know whether you're overfitting or under fitting you know asking you to explain specific algorithms that you're familiar with in terms of how to prepare I think it's especially you know this this advice is kind of focused on the machine learning engineer role I think it's still very important to prepare for a general software engineering interview because it seems like a lot of companies will ask those types of questions you know make sure that you are sort of brushed up and ready to talk about previous ML projects that you've worked on and and then I think the rest of it is around you know reviewing the basics so make sure that you remember how kind of like even simpler machine learning algorithms work like linear regression nearest neighbors things like that and you can explain it someone and then you know if you're interviewing at a company that does a lot of deep learning then you know reviewing the the Ian code fellows deep learning book is another really good way to just make sure that you're you've covered most of the basics great any other questions on hiring jobs teams yeah Ian's good fellows deep learning book oh yeah I can select it out mm-hmm if you're googled deep learning book it's also the one that comes up yeah all right so that's that's all that I wanted to cover in terms of homework for next week the there's a lecture that we want you to watch and I'll post the link to this in slack on testing ml code bases and deploying machine learning systems and then the other pieces just continue to work on your projects and you know again feel free to post updates and ask questions I'm happy to kind of go through those and make specific suggestions on places you get stuck I think we have two weeks left in the course and so you know if you're if you're at the point where you're getting close to having something that you feel like is working but is not quite there then you know that's that seems like right where you should be but this is a good time to make like a final push to have something that's working pretty well for next week so you can kind of wrap it up and make a good presentation the following week