Track your machine learning experiments locally, with W&B Local - Chris Van Pelt

Key Takeaways

today I'm going to talk about just a couple features that that I've been really excited about and yeah I won't i won't geek out too much but hopefully just the right amount so as most of you know we provide weights and biases is a hosted web service so we actually host our service on Google cloud platform and that's where app WBAI sits but when we go out into kind of the corporate world and talk to potential customers of our of our software it's often a requirement that they're able to run their own server or their own version of local so over the last year or two is we built this company we tried a number of different approaches to providing our service local and we recently relaunched W&B logo which is now a part of our python library so if you do a pip install W&B you'll have this and behind the scenes all it's doing is it's running docker and it's going to run our docker container WB slash local which contains our entire server and allows us to run a fully featured weights and biases service on our own machine in our own private network so I'm going to go ahead and just do that live now and I'll also sneak in kind of a new feature that I'm excited about as well so first time in this directory W&B demo and I've got that me be installed so I could just friend that me be local and I already have the docker image pulled if this were the first time I were running it it would actually pull the the latest image from docker hub and now it actually launched me into the browser and as this as this boots we're gonna see that we're actually loading the environment so weights and biases local and I'm seeing comments here I will increase the text size but surprises local is actually running a number of services inside this docker container so we use my sequel to store metadata and we use cloud storage to store all the artifacts associated with experiments and we've actually launched our own instances of those services inside the docker container the first time I looked local it's going to ask me to create an account so now the the install can be secured so I just created my account and now it's giving me my API key so I can go back to my terminal and now it will log me into my local instance so to show you the actual local instance in action I'm going to go ahead and from the same directory start a Jupiter lab instance I'll make this a little bigger or too big okay so if I are a to Peter laughs you put her lab not Jupiter lack okay cool so now I've got a demo notebook and inside this notebook I just have a very simple neural network so I'm pulling in fashion in mist and I'm importing @wv library and I'm going to go ahead and load the data and normalize it and then I've got a super simple convolutional neural net here that I'll compile and then lastly I'm going to knit my WB project and then use the weights and biases callback specifying that I have images so they'll automatically be logged with labels and this should be good to go so now you can see because I actually installed local my local system is configured to talk to local host instead of app that WBAI and it's gone ahead and put it into my default entity which is Van Pelt that username that I created on signup and now if I click here I can actually see the run before I need to step through my new features here and my brand new install of local and now we can see the the same weights and biases local experience that most of you have seen before but now instead of running inside of our Google Cloud Servers we're actually running directly on my laptop so we can see that we're logging some examples from our fashion data set here and all of our los metrics are updating in real time and if I were to turn off my network right now which I won't do because of course I would stop streaming to you all all of this would continue to function because all communication is happening over localhost just on this machine just for fun we can actually we can actually see what's going on inside of local here so if I list all of the processes that are running on my system we could see I've got one container name WMV slash local running on my system and now I can actually shell into it so I could say docker exact IT let's go into seven one nine and we'll run bash but now I'm inside of local and we can see that we're actually running a number of services so wits and biases itself is actually running my sequel and then many Oh for a cloud file store as well as a number of our internal go services that that actually make all of the systems work together so one of the new features I wanted to share was better version control inside of Jupiter so we just released a new version of our client version Oh 834 today that has much better port much better support for for Jupiter versioning so here let's say I changed my dense layer maybe two to six before and maybe reduce my convolution from 32 to 16 and then let's run for three ethics this time as well now if I redefine my model and ivory in it and call my call back again I can go to my runs and see now that I have two runs honest win one and resonance too if I jump into honest with one I can see the files that I've actually saved with this instance so here we can see things like our config which will contain the number of ethics as well as some metadata about my experiment and if I go into the code we'll see two two new files here so one is demo I pi and this is really a sneak peek feature guys so this currently isn't available in the in the cloud product but we will be making it available in the next day or two we now render Python notebooks so we can see the exact kind of Python notebook that was run but we also create this session history Python notebook so what's happened here is behind the scenes we actually tracked exactly which cells were run in your current session and we can see the exact code as it was run so here this was run when I hadn't changed these these values so we can see epics 5 indents 10 if I go to my other run which would be floral resonance we'll see that this actually saved my entire history including the cells that I ran when I actually changed the values so now this can really be a great way to add a layer of version control on top of your notebook so that you can go back and see exactly what code you ran to produce any given experiment now I haven't done this yet so this would be trying live and I'm worried this might break but it wouldn't be a demo without potentially breaking so we also have a code compare panel and you know notebooks behind the scenes are actually storing JSON but for our cases we we actually extract all of the code from the notebook and then can show you the actual lines of code from the notebook so that you could see the difference between two given runs and in this case we're actually looking at the source code which I hadn't saved so we're not going to see a difference if I had saved this I think and ran it again we would so with that I am more than happy to take any questions folks have about WB local or our new Jupiter notebook support and yeah one thing to know someone's asking about Cola one of the issues with collapse we couldn't get the source notebook itself because it's actually not stored on the collab instance but now we will be able to actually store the session history so all of the the cells that you run within a collab context will be exposed in that session type I and B and you could use that for reproducibility or sharing down the road so with that I'll see you guys in the in the slack room and I'm really excited yeah go ahead LaVon I can ask you some of the questions they're asking the fact sure let's do it cool someone asked a local instance be used completely offline after the container is pulled yeah absolutely so the the only requirement is going to be wherever you're training your models that needs to be able to talk to this container so if you were running the container in the same machine you could just talk to it over localhost but most of our customers run the container inside of say a kubernetes cluster on Amazon UCS but you could do that on a private V PC network so that you would never have any any data go out on the Internet it could all stay on your own internal networks well someone else asked are we different from Savoie or how's your it's different from you so there's a whole bunch of things that are different from from tensor board and I'll highlight just a couple so one is beyond just the the kind of metrics that you would normally see in tensor board we're actually capturing system level metrics as well behind the scenes so you can see how much CPU or memory that you're using we're also capturing a bunch of metadata about the code in the environment that you ran in so for instance you know I showed the actual kind of source code here but alongside any run you can have arbitrary artifacts another example of some code that we track as requirements txt which is showing me exactly what Python libraries I had inside of this Python environment when this ran so we have a number of features really around reproducibility that you wouldn't find in something like tensor board and then when we really start to shine is after you've done a whole bunch of experiments so we've heard from the majority of our customers that when using tensor board if they have more than a few handfuls of experiments that they're trying to compare that it tends to get really laggy and become difficult to use and we tried to make our interfaces really snappy when you have even thousands thousands of experiments and we've really made it so that you can collaborate with other people on your team whereas tensor boards really just for an individual practitioner generally the last thing I said is if folks really like tensor board you can use it with weights and biases and will actually run tensor work for you and you would see that as a tab inside of an individual run thank this happened more questions so someone asked can i push the runs I do look me within vices yeah absolutely so right now there's no easy sync interface but from our Clee itself there is a WB sync command so you can always think a local directory back up to our cloud or to another instance of local in long term for our local offering we envision having a way to just programmatically with a button inside of local say maybe move this to the to the cloud version or vice versa and so that's something that we we hope to come down the road right some nurse Justin's not expose your IP to the outside world by releasing the doctor now so the docker container will just listen on the IP address of the host that you're running the docker container on so you know most of our customers would instead of say go to localhost that create their own DNS that would point to that that host but if that host were on a private network that wasn't exposed to the Internet then that IP address would would only be known to that network cool and in this one last question how does the callback change for a torch maybe we can share the code for that and I'll answer that in the stack yes documentation here and there are examples for 4/pi torture especially if you're using lightning or ignite there's some nice callback examples that look a lot like that that Cara's call back to the person who asked the question about the IP site that my IP cement intellectual property oh I see yeah I mean although the whole point of local is to really isolate all of the data to be fully in your control so no data would leak beyond your networks or or infrastructure cool there's other questions yes thank you Chris nothing Barry you bet thanks guys looking forward to the talks