Why Experiment Tracking is Crucial to OpenAI

Key Takeaways

I work on the robotics team at open AI where we try to build learning-based robots that can eventually do anything that humans should be able to do I worked on everything from figuring out the right algorithms to power these robots to building the equivalent of the sensory systems for these robots one of the things we've been working on in the project is to get a robotic hand to manipulate real objects so you can put a block in the hand and we can orient it to any orientation and this is a kind of a problem that had eluded the robot community for decades what that involves is programming computers such that they learn from the real world or say in arcades allowed in simulated worlds just as humans - you know as as children and adults - when we want to do new tasks if there's a learning process you won't get everything right on the first strike and I'm programming robots to have this more human-like learning based behavior before we started using weights and biases everybody had care of their own little setup of how they would get the results and so on like some people would be using tensor flow with tensor board some people would be using their own kind of homebrew version of some visualization tool and so on so everything was very fragile like if I want to share a piece of results with someone else the best I could usually hope for was a screenshot of my graph and like then paste it and send it to them in some some way over slack or over email what has changed now is that since we have like a common place where all our results are I can take the results of my colleague Lillian for example I can take whatever she has trained and I can compare that with what I trained we can create a quick report with that I can download the model that she had trained I can go in and look at other metrics very easily since I have all the raw data that I got asked her to make me a new screenshot it's reduced a lot of the overhead in communication to make us really focus on the on the communication that really matters about like what should we work on and what what are the most important things now rather than like what did your results look like two weeks ago that's a waste of time we use weights and biases with continuous integration a lot it's extremely important to see that your model don't regress you know it gives you a kind of sense of the pulse of the team of how quickly you're moving and so on but it's also an extremely good way of just having transparency in the work that you're doing with other people we have like 10 to 20 people working with our code base so at any point in time somebody could commit a change that breaks something the worst thing that can happen is that you find out after a few weeks that you have a regression and then you have like two weeks of commits to go through and figure out what went wrong then you lose easily a week or two of work thanks to Vice advisors I've just saved lots of lots of money just comparing results in general is much faster when I have all the data in one place in some ways kind of like a shared logbook for the team of our progress we do this a lot in our workflows comparing against old baselines and so on so we can kind of keep on having old runs available and compare against those over and over and over again it's a very transparent way of seeing how much your utilization is of your resources like do you use 10 percent or 90 percent of your CPU or GPU and you know we want to be at as close to hundred percent as possible so it's been a very very useful tool for us for just like saving money and you know it's you can call up your friend like why are you only using 10% of the GPU you can be running ten times as many experiments we're trying to build the robot brain the brain that could work with any robotic incarnation so I think it's it's kind of an enormous positive impact on the world to build general-purpose robots I want to be part of a figure out how to do that [Music]