PyTorch Developer Conference 2019 - Panel Discussion

Key Takeaways

[Music] hi hope you had a good day with the breaking between being longer than expected I actually am glad that break was there because usually with the one-day conference people don't get to talk to each other very much by the time all the talks are done you just get tired so I was like this is great PG helped us out here so coming to the panel I think the point of the panel was to try to bring God some questions around tooling and using that tooling from the creators and users of this tooling and get some questions answered that will bring like make us build better tooling for the future as well so there are five great people who were willing to be part of the panel especially that the hard questions I'm going to ask let me quickly introduce each one of them the first person is Rachel Thomas Rachel is fast AI co founder and the director of the USF Center for Applied data ethics where she works on data ethics AI accessibility and bias and machine learning welcome Rachel [Music] [Applause] the second person is Clement de la noche Clement is the CEO of hugging face the company that built the library pi torch transformers which is now called transformers because they added TF to support and Clem o also has a background in computer vision he actually worked at French startup called mood stocks that eventually got acquired by Google so welcome Clemmie to the stage [Music] [Applause] [Music] Thanks the third person is Maria FOA she is a PhD student at CMU at the laboratory for molecular modeling and she's also the optic author of open chem a PI torch based deep learning toolkit for computational chemistry and drug research welcome Maria [Music] and next is our as mikela Paganini she's a postdoc at Facebook a research she investigates the learning dynamics of neural networks and she brings her physics background and scientific approach to machine learning research and she was previously a physics PhD student at Yale and also part of the Atlas project at CERN where she was doing particle physics research welcome mikela and the last person on the panel is Leisha Lee Leisha founded rosebud which is a stealth startup so she can't actually tell you what she's doing but she can tell you that they're building tools for creators for photo and video generation and Leisha was actually in the VC world before she was an investor in several startups and did her PhD at UC Berkeley welcome Lisa [Applause] thanks all for being on the panel I picked five a few because you are users of piperj and creators of things on pi torch and you have a combination of both between education and library creation two core contributions and so on and you know startup research - production which is the team of Fighters these days so I think I have a very simple opening question to set the team to set the context what are the central tools and frameworks you guys rely on and take for granted and use them day to day start with Rachel sure said it's a Jupiter note books is a key one and they're great for teaching but they're also great for software development at fast AI we're moving all our development and testing into Jupiter notebooks as well and I think that they've been really revolutionary beyond that I think kind of many of the obvious ones PI torch of course yeah or a fan of hugging face transformers thank you yeah I mean for us surprisingly even if we are builder of a tool we don't use so many tools company-wide everyone obviously at their favorite tools with big users of Pi torch obviously we released our NRP library on tf2 - as you shaded me a little bit earlier and can sound camp like a cliche but we're spending our days and sometimes our nights on github to manage all our open source we end up a lot like I received like thousands of emails like asking me like oh can you add that can you solve this problem and I always I have no to reply at this point raise an issue and github open Apple requests because that's very much what we spend our time on it's our kids abuse our OS in a way so even if it's cliche I think that's important or not to forget great oh yeah yeah for us we we use Python and PI torch for research and pi torch is great for fast prototyping and I've also used a couple of commercial software for molecular modeling cuz our lab is working on computational chemistry and part of that is doing like real simulation and yeah for we try to put all our code online open source to make it open source and for that we use github as well so I would say those are the main software main frameworks were using yeah likewise python pi torch of course and other deep learning libraries and then if I were to wear my scientist hat I wouldn't I would say all of the scientific Python ecosystems scipy matplotlib pandas etc and then again as a high energy physicist I would say a lot of open-source libraries that are domain-specific and that have been developed by software groups in the field iris have hsf et cetera and then a lot of tools in C++ as well because as a scientific community we have to integrate with either old archaic or highly performant code bases so there is that dimension as well so yeah we have a user suite of tools I mean obviously pi torch we also use tensorflow I think thing with PI torches that mean we're trying to solve for adopting research into like fast prototyping so not only you have to write new architectures ourselves but also like because the open source community offers so much in PI storage I think is very popular in the image generation literature and research area I also run a distributed teams so collab notebooks github totally a great stack for that so me and my engineer we sit different cities different time zones but having these tools that make it just basically I think software is the the most easily distributed thing that you can kind of run team wise and then more this is like a more recent thing we've also tried for cost reasons using spot instances to like run training so there's like a new top ad tech blog post that we wrote about how to combine that with kubernetes and Horvat on AWS nice I I heard a lot of Python I heard a lot of open-source I heard a lot of ecosystem /oh being able to use stuff from other people especially Rachel said hey titers transformers and and so on and then I heard from Maria about commercial software or you know things it might just be important and I heard hi torch which you know it's nice here I have a simple question I heard open source Python slash large ecosystem commercial support brand value which of these are the most important aspects of each of these that can you simply not give up like is it open source is it the ecosystem you know the network effects is it commercial support as a start-up founder would you expect that it comes with certain support which of these maybe I will because everyone looks like they are not sure sir yeah oh we just want to use things that work so I don't know that we feel strongly about a particular kind of whatever works I mean a tool that we use that's pretty uncool is Microsoft Excel is really useful and we actually you know have has a convolution implemented in it to show people kind of how it works it can be a great tool and so we do kind of use I would say like a rain of tools and it's just kind of finding whatever's best for the job we're trying to do yeah I mean to complement that we were looking for when we investing time onboarding ourselves or our team on the new tool is for it to be future proof right to make sure that it's going to get better in the next few few months few years that it's still going to be around and so when you think about future proof then it doesn't really matter so much if it's open source or commercial right if you have an open source as it's well maintained where you have like community behind it or organizations behind it the same way PI torch is for example it's as future proof of like your service that you're buying from a company that you know will be around for some time so I think your future proof is like the thing that you're looking at when you're thinking about adopting new tools okay for us I'd say the most important is whatever lets us do the fast prototyping thing to try ideas very quickly because I mean you don't want to spend a lot of time on trying the idea that won't work it's all right as a as a research scientist so I would say Python is great for that and this is not something we would ever give up on and also we don't care that much about the open source or commercial as long as the software does whatever we whatever we need it to do so yeah I would say that is the most important aspects for us I would echo both of the last opinions the future-proof for experiment design is really important because we're building software for the next generation of experiments and so software has to last we can't just go off of fads or Twitter popularity we need something that will be there years from now and obviously that often times causes some delays in really trying to evaluate and assess whether the software will have a lifespan of half a decade or half a month in terms of the support but that is key for scientists and then again the Python ecosystem allows you that speed of prototyping and iteration in terms of experimentation and research both dimensions are really important to us I think open-source matters and so far is I'm providing a good environment to attract good talent I mean this really matters but in terms of I think support I mean I think most of the frameworks now I mean halfway we heard it now it's about earlier today but like there's a lot of good cloud infrastructure support and right now I'm doing b2b but when I was originally focused on a consumer deployment you know I was experimenting so fast with the actual models that I couldn't like to have a you know serialized version to run on on device and so I had something that was deployed on cloud that I can constantly change and so like having a lot of good support for various frameworks which PI torch does it makes it easy to prototype and give it to my customers okay that's great I heard some very practical answers whatever gets the job done - you know it's actually important for an ecosystem to be there - you know Clemons answer as well so that's that's interesting so now I think you know one of the things that maybe we need to explore is how do you pay for tooling all right like you know the economics of tooling itself so I heard that I heard Excel being one thing you know and I should say we use it as a teaching tool and there's things that are useful to do at it but we're not like the fast a library is not in Excel obviously being interesting to us a library so I I heard a talk recently that that said something in the order of numpy has like what one or two full-time [Music] developers and that's the funding they have various non PI's universally used very as I presume if you have to buy something like Excel you wouldn't really bulk it buying it would just be like oh it's like $50 or whatever we'll just buy the license so I think there is a bit of a systemic issue and I'm wondering I'm asking the panelists like what would be a way to fix this like you know what are your thoughts about like the economics of tooling should we just make all tooling commercially and if it's open source should be like how do we have the most popular tooling also be sustainable does anyone have thoughts yeah that's it that's a good question if like if you have the answer to that you should start a company together I think there's like a big switch happening and if you talk to investors today I think interest in Orbitz released an article a few days ago basically saying that the first stage was software eating the world and now open-source is eating software where basically you have a new world where the boundaries between software and proprietary and commercial are fading in a way you know you used to have this open source which was mostly hobbyists not not in that sense of it but people doing that on their spare time or like very very small teams and you would have like commercial and they wouldn't be kind of like ways to go from one to another now I think it's more complicated than that and and tools like tight watch or transformers or testament to that where you can really camp like take advantage of it what is great with open source meaning that everyone collaborating to the same tool the power of the community the power of adoption because people can use it the way they want to use it rather than new telling them how to use it but still build a company around it and find a sustainable way for you to keep building this to and keep improving this this tool so I think the current evolutions are by themself games like finding solutions to the problem that you talked about which to me was more problem of the previous world than the new world where now you manage to find new models for open source I mean there are catholics examples from elastic confluence hype like you have now like tons of multi-billion dollar companies that have been built around open source so now it's a good time if you want to do that if you want to start a company around open source and can fly can make it a sustainable business I think you can today okay so I'd say I mean I think there are still serious gaps in that ecosystem though and so like one example is package management like there's definitely kind of need for something better there maybe with like continuous testing where nobody has stepped in yet to kind of provide or like you know the solutions don't have enough kind of support yeah and I I don't see kind of like an obvious or easy solution to that I mean just quickly I think I've seen there's a lot of capital that is willing to invest in popular open-source teams that want to do a prosumer layer on top of those so I don't think it's like a problem of funding it's just you know build something that people love I'm saying obvious stuff here I think definitely like the public and private partnerships are another really good way to go this hybrid model I think has worked in the past really well if we think even just a project Jupiter not for profit but with significant contributions from companies that lend some engineering capacity to the project in the form again of people sitting on the board and contributing real code while obviously taking advantage of the technology in their technology stack in the company as certain as well with certain open lab we have great corporate partnerships where again companies can can collaborate with CERN to to help develop open source code work on some laser focus projects as well and and mostly provide again that engineering capacity to to keep on developing in a sustainable way and giving back to the community in an open source way way yeah I agree a lot of commercial software software they offer academic licenses so you can just access it for free if you are like doing the research not for profit that I think is a good model to be but if you want to use something to actually like base your that will be used for your business I think that's fair to pay for for the software okay fair enough I think I heard interesting answers I mean Clemmie had a detailed answer because I presume your company has to think about AI transformers and you know how to sustain that I I still have some deep questions about that because well what Leisha and Clemmie said are someone like numpy could raise a Series A and then get a huge valuation figure out how to make money later okay great those are very useful thoughts thank you and I wanted to ask a question that is actually you know beyond software tooling we have educators scientists startup founders what are the current things you know you you said some of the software you have these are things that make you productive these are very useful helpful to advance your mission what are current things software or not that are making your mission unproductive like what are the things that are generally not letting you progress do you have any like broad thoughts but you know giving ideas out to people who might want to disrupt let's face I mean just like bouncing of the subject of tools for us what's really important is for tools to work together so like for example when we really is the tf2 version of our in LP library what was really interesting and important for us is to give users the ability to basically go from PI tours to tf2 to four teams that have some people working in pi torch and some other people working in tf2 to be able to collaborate right so it's sometimes frustrating with with with tools is when they don't work well together when you don't have like the interoperability between between them and on the reverse when they work well together it's like it's like it's light like for example the fact that you can now train on GPUs with PI torch even if obviously it's two different companies that are something some sometimes kind of like conflicting in their goals first it's like amazing because it means that you can start taking the best of both worlds and really make your workflow efficient and productive whatever tool or framework or language you want to use interoperability basically yeah and the lack of it's sometimes a challenge so if your bank could talk to your taxes directly yeah I would say access and accessibility are still issues even though we're making progress here but when it comes to cost or energy consumption those are still outstanding problems I would say and then in terms of tooling I would say the proliferation of it and with that the lifespan is she that we mentioned earlier will the stool be supported two months from now before I you know make the investment to integrate with it I would want to know if the community will support it and if I will still be able to speak the same language as the majority of the people in the field moving forward so so those are key yeah listen the main thoughts I kind of said this last time the package management is the same thing for us package management yeah I would say the the research reproducibility is an issue for us when we take a paper get their code and actually weren't able to reproduce the results for like this could be for many reasons but that's a problem great I think we heard and the research track from papers that code who are trying to improve it but yeah as a field I guess we're trying to make a concerted effort to fix that just curious how do you deal with that and something like chemistry which it must be much more complicated yeah sure and also I would say that something like culture in the in the community is slightly different from pure deep learning machine learning like statistic community so people often don't care about sharing the environment something like docker and you can just you take the code and it's very hard to to get the same results and also there are like different aspects different tracks in drug design so people working on small molecule design people working on rich earth into sis models and if we want to build the whole automated pipeline for drug discovery we need all these parts and sometimes we take take some results from other research groups take that code to build a pipeline for for our for our own process and another issue could be is that like the best retrosynthetic model day now they saw that you scare us and yeah and it's very hard to to to plug it in into my torch pipeline so this kind of thing is very outdated software and maybe they even have the better version but for some reason they don't want to share that open source they keep it to themselves I think that that needs to be a bit of a problem and yeah in learning community - and now it's like people who keep their code I guess just lose out in the long term yeah there's there's still a little bit the issue though like we're seeing it with the library like we're like dozens of thousands of people using it for like fine-tuning pre-training not all of them are sharing BAC into weights fortunately a couple of people do like for example with people sharing their weights or German and from for Mandarin within within the library and I think it's awesome but it still camp like the same way there's like one percent of the people online that are actually actively contributing to something it's still a little bit the same with its pruning I feel like yeah yeah that makes sense I'm actually surprised the majority of you being in the bay area none of you said commute so that's that's interesting like reproducibility being at the center of advancing your mission I think on probably you know the tooling topic as well I wanted to talk a little bit about a question I asked something similar in the last year's panel as well which is in in machine learning in the whole space in the tooling space you the entire panel said fie torch or answer flow you know that that's really about it and the whole tooling is consolidated around these two things which now look very much the same so I'm just wondering what your thoughts are on about the upsides and downsides of this and if you see any ways you want to use the upside or address the downsides any thoughts on this aspect of non diversity and the main tooling that's powering our field I will say at first AI we are definitely interested in the promise of Swift for tensor flow as a potential tool down the road and even included it in our last course and we're still using PI torch kind of fully in our library right now so I don't know that it's like fully consolidated and Swift swiffer tensor flow is you know not Python it's a new language so it's I feel like there is kind of still some openness about like there may be more options in a few years okay you're you're hopeful of the future yeah I think there's still possibility like it's not yeah right yeah I pretty much agree that we're super excited about like a switch for tensor flow to and and we think there are also more and more camp like vertical ization of the tools obviously transformers is a good example of that because you end up using significantly different kind of models and technique for every single field of machine learning I think more and more you'll see vertical tools emerging that are going to be amazing for robotics or like amazing for sales for having or amazing for NLP just because at the end of the day as deep learning is really camp like replacing software 1.0 with different ways of telling products different ways of dealing oops and that it's camp like spreading to any single field you'll need more specialized tools to really address every single field resident just having one tool and using it for every single field I'd say our life would be easy if them if there was an unified ecosystem for research experimentation because as I mentioned this kind of relates the reproducibility issue and building a single pipeline based on the results of other research groups so if like someone shared their code incident it citizen tensorflow and I want to use this model in a combination with my model which I develop everything on pi torch this sometimes becomes an issue I would be happy if everyone if for me for sure the upside of consolidation is that we don't have we don't suffer from the JavaScript syndrome when were there are hundreds of different ways of doing the same thing and instead as Maria said like we we have this opportunity to speak all the same language and build this ecosystem of tools around a agreed upon schema which could be pi torch or tensorflow or others the downside of that I would say that if that doesn't fit your needs at the moment then you basically have to resort to building some highly customized solution and then you're no longer part of the in groups at that point it's really hard for you to communicate your ideas and share them with the rest of the community because you're working on something that is very niche and the other downside I would say is the potential explosion in size of these frameworks so if I don't know I think one of the advantages of numpy for example is that I can fit it all in my head I know exactly everything that I can do with that more or less and there is a risk of you know creating huge frameworks that are extremely complicated and only a few power users can really be productive with them and then as as the size grows then the average user just becomes overwhelmed so we need to be careful with that and maybe rely more on this eco system system to break out all the different various ancillary tools a torch vision torch that's the thing torch has done a really good job so far at keeping things separated enough so that you can import only what you need and then you know branch out only if you need to I think both I mean I've not had to like make a choice between the two and it's like pretty I mean especially since now tensorflow is looking a lot like pi toreador makes it easy right and as well but given that we're sort of we basically have an API for all of our trained models and so if it's stalker eyes it runs in the same server so any you know whether it's a pie chart or tensorflow it can so it's really a matter of like how can we get this thing up and running faster and so if we're getting things from open source then you know a lot of it because the research community loves pie charge is going to be in PI George but then there's also like really good models that already you know written in tensorflow and so we can use it so I haven't really been I think the pros for me are if it's fast prototyping both frameworks are so well supported by cloud infrastructure that I've had no issue using both and you know personally have a little bit more preference for pirate torch but whatever the open source community kind of like loves what about user I feel like you're the best person to know why torch and maintainer of it how do you feel about this consolidation I am NOT on the panel but because this is the last question I'm happy to I think it's really bad personally I think it's it's it basically kills innovation in a very deep way so I would rather have competition brutal competition than being in a place where there's too frameworks were comfortable with each other so I think the panel was pretty interesting to me at least because I I think I heard very very unique perspectives like you know funding open source like with a VC model or like like various different things packaging you know if someone can fix packaging that would be saving the world in a lot of ways especially me and Rachel and Clemmie had very interesting thoughts about like the vertical integration and and I think that's something Maria and Clemmie and and Michaela shared an extent which is I just want things to work and in an interoperable way like like it's not so much as like their different tools but hey they're different tools maybe they do each of these things differently but I wish they'd just talk to each other better right like that's fundamentally what you guys were going for and yeah without further ado we could tank the panelists for being here and I believe the next thing is our incredible set of posters but I might just be wrong but Joe will probably come on stage and tell us what the what the next program is and we could walk off stage [Music]