Deep Learning with PyTorch Author Interview with Eli Stevens, Luca Antiga, and Thomas Viehmann

Key Takeaways

[Music] rate of change and innovation in the space that we're in is so high there's a tremendous amount of opportunity and there's a tremendous amount of different directions this technology is being taken awesome thank you so much everyone for joining our book club this month we're reading deep learning with pie torch and we're very excited and grateful to have all three author authors with us today um yeah would you like to introduce ourselves who wants to get started Eli first author sure why not hi I'm Eli Stevens um uh let's see I'm currently working in autonomous vehicles uh writing the book was uh pretty awesome Journey especially because it introduced me to Luca and Thomas awesome yeah I'm lucantiga I'm currently a CTO at lightning AI and uh yeah uh it was a great adventure and mostly for the uh or not mostly but yeah first played for the friends I made for sure give me an amazing opportunity and also kind of uh you give me the opportunity also to publish some of my little uh characters which I would never had the opportunity to publish anywhere so that was a the actual official excuse for lighting love it love it so did you draw all those uh drawings not just like we we all Drew but I'm particularly fond of the memories of creating the drawings I love the drawings it's so much fun yeah so Thomas uh yeah I didn't do any of the drawings I think well I kind of sketched some things and then look at the real artistic version of it but uh yeah so I joined the book late the third author is not just not just the order and the book but uh obviously it was awesome and look and I have been in very close contact ever since uh and it was great to do this with Eli and Luca awesome thank you so much for the introductions so we have people joining us today if any of um the book book club members want to introduce yourself feel free to speak up or type in the chat and also feel free to ask our questions whenever we have a few questions from the book club also from myself that I would like to ask if that's okay before we get into that I just wanted to say because we were talking about the art um one of the things we were trying to do with the art and I'm curious to hear feedback from the book club if this made it through was to kind of have that like people standing around the Whiteboard scribbling things talking to each other feel which is why we went for that like hand-drawn it wasn't actually hand lettered but we we attempted the hand lettered feel uh we actually used a comic book font uh for some of it um and I was curious uh just as a question to to the rest of the group if if that feel made it through or if it was just something that like we tried to pull off and it didn't quite come through I feel it looks a lot warmer when I first saw the drawing I was like wow that's cool I really like it I mean I like the book now I like it more thank you okay someone was asking a question okay so yes it is recorded someone was asking if it's recorded it is recorded So if you're not comfortable of speaking up you can always type in the message the messages won't um show up publicly when we publish it so yeah okay first question what is the story of the book How long did it take to write this book what are the challenges Which chapter is the hardest you write and and did you all know each other before the book how how did you guys meet okay yeah I'll jump into it so um from my perspective no I I didn't know either Luca or Thomas uh prior to starting on the book um uh when uh when so Manning reached out to me in the summer of 2017 saying hey would you be interested in writing a book on pytorch and at the time I was very surprised because I had worked with very pytorch very little um and uh while I'd always been interested in neural networks and AI I hadn't really done anything in the field until about six months prior when pytorch 0.1 came out um I was playing around with some projects and I submitted a couple minor pull requests back to the project is to fix little minor things or or do documentation and I think that they were just running down the list of people who had committed uh code or something along those lines anyway they got to me and I'm like I think you have the wrong person I like I'm self-taught I haven't been doing this for more than a couple months are you sure um and uh I was also concerned that I didn't have a whole lot of time because um at the time I was working at a startup we were getting ready to sell the company uh had very young children at home who's now big enough to run around and stub his toe uh anyway um and I was like I don't have any time and they're like how about a co-author and uh that's when I got introduced to Luca Luca what was your uh how did they rope you in yeah I was also contributing to pytorch and I think it was like steam Dynamics um I had been doing deep learning for a little longer um because I'm co-founder of a company called aerobics which is still very active and um and a few like I think three years earlier than Fighters came out or a bit started to do applied AI in various Fields like manufacturing Healthcare and um yeah so we I went through like a few Frameworks and then finally pytorch came out and I was very enthusiastic so I started contributing a bit and I think yeah that was the thing and my problem was exactly the same so small kid uh day job no time um and then probably the the doubt will I ever make it and uh yeah any two people that didn't have time I've came like a couple of people that didn't have time the project started very enthusiastically I remember uh it was summer I I was planning the the chapters and we had like plans that were greater than what the book was yeah or more comprehensive let's say and uh and then we started writing and the dynamic of the book was that we had a like a I think that the first chapter um was something that you know there was some back and forth we kind of uh uh set ourselves to right tone I would say because we had to find our own voices and stuff and then a bunch of chapters came out and maybe you can talk we can talk about it later but um at some point Facebook wasn't meta at that point uh decided to Stace Manning was publishing uh chapters in Early Access uh the uh they decided to make this book as kind of a an extract of the book the first chapters um something that they would like distribute publicly through the website and this made the book gained some popularity and uh asking some steam and so on but at some point there was a need for the extra push to get through the Finish Line because writing a book is a lot process and the editing but also the completion of it and and extra talent that goes back and kind of reviews critically and your availability is not constantly done like we had no time at the beginning and we have even less time moving on because many things were happening and uh and so this is when uh Eli and I Mike tried to you know touch on like some of the early contributors to Fighters or The prominent contributors fight Arch and to my surprise Thomas was saying yes like let's do it and I was very excited because Eli asked him he wasn't at the pythosh conference right so so the story is that uh obviously uh I've been good friends with Pierre who if you use the python forums uh he answers about I don't know one-third or maybe more of the questions I mean he literally answered ten thousands of questions there um and so uh we always or sometimes on on the slack channel for python we would joke that when we gave an answer well yeah this is this is explained in our imaginary pie touch book in chapter 27 something like that and so at some point Eli said well yeah would you be interested in doing a real book and so uh that's that's how I got in and uh I think it was uh it was really I mean it was a great great book project at the time when I joined and I'm I'm really glad and honored that I could join Eli and Lucas for that and to this day I think that there's a lot of things that turn out really well and some of the things many of the things have been done before I joined but there's also some that's where I think I kind of was able to add a little perspective that help up fairly well for for like this fast moving as subject as our book has this is an understatement of course yeah Thomas is the reason the book got finished well exactly let's not put to find a point in front there um it was it was one of those deals where uh you know we I think we've been working on the book for about two years uh and then uh we're like okay we really we need we need some we need some fresh gas in the tank and uh um yeah Thomas came along at the the perfect time help us uh author the last couple chapters and really give the entire thing that level of polish that uh made us really proud of what we produced so it was it was good timing yeah it was amazing the structure of the book is so interesting I wasn't expecting part two at all like diving to the medical field it was really eye-opening not only in the expertise in the field but also like the process of how actually do you do an end-to-end project um the different levels of tweaking and stuff is really nice to see I I think one of the book club member said that part two inspired him to look into the medical field so yeah thank you so how do you come up with this idea of like the two parts it's like I've never seen this before well so I Luca touched on this earlier um in the very first let's like when we're trying to outline the book I I think we ended up having like we originally wanted to have you know part one was the intro and then we were gonna have part the part two that you see now and we're gonna have another part three that that was like text like a similar idea with a text project and maybe even a part four with another thing and then we got into we were like oh wait no that's gonna be like a thousand Pages there's no way that the publisher is going to agree to a thousand Pages um uh but so it's kind of cheating so if you remember I mentioned that um uh uh you know I had been playing around with pi torch in the uh like the January time frame of 2017. at the time kaggle had had a competition um the the cancer moonshot challenge which had a million dollar prize pool and that was kind of my impetus to start using pytorch and so a lot of part two is actually my experience playing around with um uh with deep learning and stuff that learning process mine was a lot more Jagged and haphazard and all that kind of things but I was like okay well what what do I wish someone had told me at the start of this to kind of help get me through it faster um and uh that I think is why it was like why it felt real is because I like had literally done it like six months earlier but just all all over the place um and so so yeah I it has that kind of ring off of authenticity to it I think because yeah I just kind of wrote down what I did wow so you are not from medical field or are you so so actually my background um at the time I was working in um radiation oncology software and so uh consuming CT scans um like writing code that consumes CT scans was literally my day job at the time and so the file format and and like the uh familiarity with CT scans and their quirks and Oddities and all that kind of stuff I had been doing for seven years at that point it was the Deep learning side of it that I was new to and so I was able to that's why like chapter nine it's like okay let's let's just talk about what needs to happen in order in order to understand a CT scan so perfect love it so much um this book has been published for two years now and I love it so much I feel like it's a golden Bible for anyone who wants to learn pie torch but is there anything you would like to change or add to the book at this point or like another book or text part three could be another book so uh I'm not sure that I should talk about it too much but actually yeah uh so there's several parts to that I might answer here so the first is that I I mean I do try to watch teaching Consulting uh uh as a as a day-to-day job and so I'm still relatively fond of uh of like the structure of the entire book um in the sense that like putting data first right after you know what a tensor is you learn how to represent data and I still think that that is a very very good structure to to give that and I'm using that when I'm like doing beginners trainings just as well um of course uh and then the second part like uh giving you a real impression how a real project looks like the things you'll hit um I think that's something that either I think it's well you say It's relatively rare but then I think it's relatively important to me or it's dear to me because my vision is that just like statistics machine learning deep learning will like be a tool for things and not like deep learning a subject in itself but it will be a tool for Architects or people in the medical field uh in industrial applications like an all sorts of applications and so just like all of the empirical science to statistics these days I hope that this will be something that brings good to many fields of work and so I think Chef the the second part is is still very good in achieving that um but now things have happened in the last two years um and so probably one of the one of the addition would go in the direction of text and specifically uh I think today a deep learning book will have Transformers and things like that um and in fact I have a not very polished chapter on Transformers that might be the start of a second edition or of an Edition for the second edition foreign but so there's many things where I think it held up very well in particular the structure um there's some things entitled that have changed um in particular this deployment story for pytorch 2.0 meter decided to change that rather dramatically um so in two years time what is threatened in the book now will probably not be the thing to do um and uh and there's additions to the field just driven by the success of and the developments that came with the success of deep learning yeah I would say that if I can add um deep learning and yeah I have changed profoundly in the last couple of years in the last year in particular and the ability to have models that can do many things if prompted or tuned was something that was less dramatically so when we wrote the book in the first place and by the way we only touched upon convolution and not attention as mechanisms but that's that's I think yeah that's something that we would ideally want to explain with the same kind of hopefully uh Simplicity that we try to achieve with the convolution side part of things but also the some of the the general structure of how the workflow used to be or still is if you do like I don't know objective object detection or other things for uh with smaller models now would be a bit different because you have this like very large so-called Foundation models that you can then adapt to many tasks and the tasks they learn is not essentially related to the tasks you use them for but it's just tangentially related to to them and it allows them to acquire capabilities and so this whole story needs needs to be told and I think uh the book today gives you everything that you need to understand that but it doesn't give you that directly and yeah so uh you must be nice to have yeah stamina too to get there second edition looking forward to it you mentioned large foundational models like what are the challenges do you think uh are there of using pytorch with those large models like large language models um I'll I'll start just because I was speaking before no I think um using well most or many of the foundation models have been developed using pytorch so Fighters gives you the ability to scale up to the sizes that are required uh by by these large models of course these large models are so large that oftentimes they don't fit a single accelerator so you need things that we haven't touched in the book like different parallelism strategies like uh charted at the parallel or model parallel tensor parallel they are always breaking up your kind of model into so that no one accelerator holds it all in memory but actually when I say accelerator is GPU not one GPU holds them all in memory but the model is kind of shared across devices and this allows you to you know scale up inside because one of the things we've seen is that screen up inside apparently is very important for achieving capabilities although there is a tendency now to make the similar things happen with smaller models so it would be always like inflation and then like consolidation and inflation and consolidation at this point but I think pie chart gives you all these thanks um in in a way that is still accessible still has the a similar ux so if you learn base python you will be able to understand the code that is behind uh on our GPT which incidentally if you take away all the engineering complications related to parallelisms and so on can be written in like 100 lines of code for 200 lines of code that you will understand if you read the book essentially because the operations the underlying operations are quite simple wow really 200 lines of code yeah if you look at uh Andrei karpati's Nano GPT yes it's it's a code that contains the definition of a decoder Transformer model which is what it's used and then of course if you need to scale it up to Trillium or not Trillium but uh tens of billion or 100 billion or enter models then you need to do a lot more things but they're all related to making this model so large that it now you know your model code and your the engineering tricks need to play in order to have this model code run across many devices make some core thick in memory or use memory efficiently these are like all things that complicate the basis definition but the basic definitions there's where can people learn about all those complicated memory issues parallelism things do you know yeah yeah I I think there's uh lots of things that are scattered uh so one of the one of the fun projects where I have the fortune to continue to work with Luca is that uh he did like you mentioned Android kapati's Nano GPT and so he started a repository called lit llama which does that for the Llama models and so this also has like the nucleus of the very conceptually simple uh lava model and then there's or the or the engineering bits that like first enable you to run it uh on like commodity Hardware things like quantization so you'll get the size of the models down a bit and then also methods to like take a foundation model adapted to your specific task which goes here uh doing things like uh clever linear algebra doing low rank approximations of Matrix operators and stuff and so uh there would be that would be one of the things that I would recommend uh they like take one of these repositories and so uh obviously write a code with Luca always uh lets me look at good code too and so there's a similar thing uh they did for some of the other models uh uh called the turret and uh so we I mean I also try to do some simple tricks like getting the memory consumption down a bit uh crazy tricks amazing one yeah yeah it's for real I mean when Thomas comes like makes things happen like beyond the current like Theta VR stuff like it's crazy yeah but so I mean things are currently there's scattered around a bit so uh for the concepts I think enrich pathi also has a series of lectures which uh you know are very reasonable uh but obviously yeah we think there's something to be said there too uh that probably will influence what we we do in the second edition when we eventually do it yeah one thing sorry I keep talking about one thing that I think would respond this question as well to this question as well is um some of the material that uh Sebastian rashka is publishing he is also working at lightning and he publishes uh uh there's a course a free course video course there are feature lectures they're five to ten minutes long each and they're like 10 modules out uh they're looking at High website but um he will take you through all the things including the ones that are a bit scattered and they're very accessible and also in some of these like recent blog posts go he goes into like explaining some of these bits so I think it's a uh it's a very useful resource today awesome could you send us the link or type the link here like the Sebastian's course lip mom lip number yeah I will uh yes I will uh write in a chat I think Eli was about to speak right yeah you're like please you yeah so the this is not a direct answer but one of the things that I think is absolutely fascinating about the space that we are in is that we have hit this inflection point where and this happened several years ago but one person can no longer keep up on everything yeah just as a full-time job reading papers they're coming out faster than than they can be consumed and I think that's really interesting and exciting because it means that that the the rate of change in innovation in the space that we're in is so high that like there's there's a tremendous amount of opportunity and there's a tremendous amount of of different directions this technology is being taken and so it it absolutely introduces friction from the perspective of well how do I keep up with all the things that I need to know in order to be able to stay current in the space but it does mean that like things are changing so rapidly that it's the things that we have today are amazing even by the standards of when the book was published like the the large language models are unheard of in terms of like what we were thinking about when we were writing the book and it's not like we finished the book that long ago so I just think it's really cool uh about all the different things happening so what's your advice on how to keep up or do we just you not keep up not even try for me it's it's just focus on the things that are relevant uh to what it is that you're trying to do and underlying that is you have to have a goal if your goal is to keep up with everything it's not going to happen instead have a project have us have a task that you want to accomplish have a a you know a model that can do X that you're trying to achieve and what that allows you to do then is exclude like if you're not talking about medical stuff probably most of the medical stuff is irrelevant if you're not talking about tech stuff probably all the large language model is irrelevant although that's changing these days but it allows you to exclude huge swans of the space and give you a much more focused reading list um and that I think is really important is having a goal having a project um rather than just being like well I want to be an expert of all of it because honestly that's never going to happen I don't think unless there's some huge amazing simplifying uh thing that happens a decade out nobody's ever going to be master of all of it agreed agreed yeah I think in addition to that uh probably there's like I mean obviously all things are are exciting but there's also some of the things where it turns out half a year later well they aren't that exciting long term and so I think some part of it is like choosing what to look at in terms of not then jumping onto the next thing when a new paper comes out very relate and goes to Echo what's Eli says that if you have a project and you like pick a method and then uh don't go looking like should there be an upgrade somewhere because that will kind of get you on the on the wrong track unless of course you hit a problem that you can solve without changing something uh but really yeah I think there's there's value and and like also using the tried and tested things uh if they're possible in particular if you if you want something that I I also runs for a while right I mean if you it's like buying a new car the moment you drive it the first time it loses most of its value right and uh and so in in some ways uh and in particular if you want to run a model like for five years uh in the future uh maybe it's it's not the latest and greatest you need but something that is uh fit for your purpose yeah I think you need to pick your rabbit hole and be okay spending a leaks at least six months in it because otherwise I'll see you are not really getting to the point where you're actually creating volume for yourself and you're just not skimming the surface and which you can do because maybe you know it's it's an enjoyable thing but if you need to challenge yourself you need to give yourself the time and as Eli and Thomas have said like try not to try to stay on course and uh not get distracted too much um yeah and on the on the topic of uh Foundation models do you know that stable diffusion under underneath uses the unit that is described in uh Eli's second part of the book yeah it's uh there are some modifications here and there but the unit model is what power stable diffusion which is a you know one of the uh very relevant Foundation models that people are so excited about today so it's not just that of course but that's a key ingredient in making that work so it's nice that's awesome well I hope to see stable diffusion in the second edition we're getting like messages about the second animations here every few minutes one of the things I wanted to add also is to highlight a distinction that I think gets lost sometimes it certainly was lost on me uh when I first started is that there's this the split between uh you know state-of-the-art and research and actually making usable useful products and I think that a lot of people you know there's there's a lot of attention put on oh we got you know point three percent better state of the art results kind of a thing and all of that is absolutely essential for the field but one of the things that I think is really driving all of the interest is the ability to take these and turn them into useful products that actually have uh value in people's everyday lives or maybe not everyday lives but you know the all the work going on in medical the intent is to then have that have an impact on people um uh same thing with you know uh autonomous vehicles it's an incredibly challenging research project but it's also an incredibly challenging engineering project and I think that uh one of the things I would really encourage all of your your listeners and book club members to keep in mind is that there's a lot of really good work to be done in the on the more engineering side of things is taking these techniques and actually turning them into something useful that uh not to get too you know dirty capitalism about it but it can actually you can actually make money with and um you know again like what they're saying is trying to keep up with the absolute latest fad chasing is not necessary there it's instead that the you know strong engineering fundamentals and making sure that the solution that you're coming up with is robust for the space that it's in there's a lot of value there and there's a lot of possibility for impact on the world and uh uh yeah so I just encourage people to remember that that industry is there too and it needs ml practitioners quite badly that's all thank you so much for the perspective I think it's super important sometimes we just get excited about fun things and don't really think about what's next okay we have two more sets of questions from the book club one is a little more uh on the technical side I guess uh when we read about tensors like I just thought about it's just numpy arrays so why do pytor why does pie torch create tensors why don't they just contribute to numpy numpy pay torch compatible and and and people are another technical question people are sometimes using tensorflow Keras or like the two sides of um the Deep learning models like what do you think is the advantage or disadvantage over um tensorflow so I think the the first question has probably I mean probably two parts uh the first one is that obviously uh numpy doesn't do GPU uh or acceleration in general oh there's number well yeah but uh uh and numpy array per se is on the in CPU memory right and so uh if you I don't know if actually I haven't checked but if you look at the when I last looked at the front page of the title GitHub repository it said like uh pytorch is like multi-dimensional sensors which would be a numpy 2 plus GPU acceleration which isn't a numpy plus autograph which at that time wasn't in numpy there was obviously there wasn't autograph system uh on top of numpy uh but yeah and so this is one part the other part and maybe look on those more about that is that obviously pytorch evolved from a library called torch um and so all these all the functionality in particular for early Pi torch came from that Library uh torch 7 and uh or the like all the computational kernels and things like that and so in the beginning it was much more a python wrapper around these functions um and while it evolved it became more of a focus to uh also be rather compatible with with numpy because that's kind of what what people expect sorry go ahead yeah there's still small things like Dimensions or axes uh that are different which really comes from this historical thing that uh numpy uses access and uh and torch use Dimension as uh the name for the thing and then you you have to pick something sorry Luca no no no no sorry I I didn't realize you yeah and you didn't finish yeah so this is totally historical because most people don't realize that even before torch 7 there was torch sticks five four and before that there were there was another project uh called I I always remember the name and now I I don't uh something learn uh anyway sorry about that and at some point in my chapter one of the book I had the whole history because I digged most of the versions with the source code and so and then at some point it was edit it out but I think there's still somewhere uh that I I kind of traced the history there and uh yeah it was fascinating because the each version of torch which was uh released by the group uh initially it was developed in Switzerland uh and one of the AIS the early very early in the united institutes there uh where the whole like French speaking thing Hawaii comes from right and then uh and uh yeah and then programming languages change so at some point you see something written in C plus plus and then they tried Objective C uh uh which I also found at some point that my path you know my journey as kind of the one of the deal ways of mixing like low level C where the kind of high level orchestration and but it never caught on because probably because Apple always kept its like core library is a bit too closed so it didn't happen and then they landed on on Lua but the terminology tensor and the API is the fact that you have eyes and that shape was there all along and as well as Thomas was saying part seven was actually the project I started uh doing deep learning with and then I briefly touched upon tensorflow and then as soon as pytorch came out which was really just take the core see not C plus but it was pure C uh kernels and see a Cuda and they were made usable by a high level thing just like with the oh objective team that I was mentioned before were kernels and then you you wrote enough high level stuff to kind of yeah uh do competition there at first Adam didn't have an autograph by the way there was an autograph for tar7 developed by the Twitter uh team that was made available at some point in Lua always but it wasn't part of python or seven and then basically the first pytharch was just the same kernels with the kind of orchestration at the top done with python at the same time the API with with exception of one thing stayed pretty much constant like what you see today is very very close to what it was very early on so it was a great work of taste and the ux done when figuring out what this python bindings would be but yeah so in all this process numpy compatibility came a bit later in the process so because everything traces back to this history yeah and there was another question right I don't know Eli do you want to take the tester test flow or this one I don't know well yes sir I'll talk a little bit about tensorflow and I want to make it clear I have not used tensorflow extensively um ah it is one of the things and and I think that uh the the the torch leadership has done a really good job of this is like not trying to engage in uh tribalism or flame Wars or anything along those lines um I think that we as humans have a tendency to want like our choice to have been the best choice kind of irrationally um and I think it's important to to to realize that in all of these things like everything is built with knowledge of what came before um you know like torch pytorch wouldn't have existed without Lua torch um High torch probably wouldn't have existed without tensorflow and Keras and cafe and thiano and you know it goes back for a very long time um and so I think that that I think that torch did a very good job of balancing Speed and Performance with um approachability especially from the perspective of uh having things feel familiar enough to someone who comes into things from numpy and because numpy is such a juggernaut in the um you know this python scientific Computing World feeling familiar even though it's not exactly the same I think is a huge benefit um and and so that's kind of how I think of this you know you can see a lot of convergent evolution too with the uh you know tensorflow 2's eager mode and all of the work that torch is doing in terms of like you know freezing graphs and that kind of stuff which I will admit I have not actually kept up with the latest and greatest there so I'll have to defer to the other people on that one um but you can see there's just a ton of cross-pollination and and rather than thinking about it's like oh you know one versus the other it's just they're different tools and they have different strengths and weaknesses and it's important to not pick a tool because of the branding it's important to pick the tool because it's going to solve your problems well um I think the pytorch solves a lot of problems in the space very well but it's not going to solve every single one of them and so it's important to to kind of be able to take that step back and make sure am I choosing the right tool for the job or do I have a screw and a hammer and I'm just trying to make it work so yeah I just linked in the chat the version of chapter one that contains the history so yeah you can maybe take a look if you are an archaeologist like I like to be something oh this is so interesting you can see different versions of the book yeah apparently that's so cool we uh we had been releasing the book in uh EAP they call it I think Early Access publishing so as we finished a chapter it would get put up on the website and so that you know people who had pre-purchased the book could look at like say the first three chapters and then after a while we'd published chapter four and it would get updated but we would also be going back and like doing cleaning and editing on the earlier chapters and so you can kind of see I don't think this is worth anyone's time but if you really wanted to you could you could watch and see how chapters evolved over time um I don't know why anyone would want to do that but you could well see how the book was written I guess your entire writing history is online it's pretty incredible uh yeah so if you ever write a book make sure you know that whatever you produce and send like random chapters to people you may see the the light and you cannot take them down so be aware of that terrifying okay my final set of questions sorry I have so many questions and we have two more minutes left for people who wants to contribute to pytorch how do we do it and also like what do you see is the feature of pie Dodge I know this question might be a little bit um just feel free to touch on whichever you wanted to say yeah you find something that doesn't work you figure out why it doesn't and then you'll make it work and uh is that your story yeah pretty much I just like tinkering with things um yeah the uh then the other thing might be uh get a friend and do it with them uh because it really is something uh the apps a lot and I've been fortunate to like meet quite a few people uh through Pi torch and I've met quite a few people in person but I've also met some of my best online friends if you want yeah and I think the the ecosystem is a place where contribution is really a bit a bit easier maybe you know these days a very large prob uh project and uh it continues consists of a core and the core uh requires some dedication before you can actually contribute to it I would say um also because it had a long Evolution he had like time to fix things so that now the most obvious thing will work the way you expect and maybe the a bit more advanced hkcs or Advanced things that are coming out uh may not work as you expect so but it takes a bit to get into it I would say um but the ecosystem is huge and uh like I'm cqi like an AI fighter sliding about this one of the is one of the leading framework with which you use pytorch um and so there's a lot of contributions you can do in a project like that or other project I don't know for I don't know uncertainty estimation or um uh so I think it's all related to speaking the same language so to speak which is by torch itself but all it enables is so cross sector that if you're willing to spend a few nights on problems and so on there's plenty of things you can do there and also I think in the last few months uh or a year or so the accent or on Frameworks has been as important as the access on individual repositories that enable people to do stuff so but I I would say before it was more framework it's like that that like gather the interest of people and now it's also implementations like we were mentioned in lit long and lipaired uh before but of course there's a ton of others out there and I don't know tweaking a model so that it can run on your laptop for example so you can fine tune it on your laptop and stuff uh it's pushing people to kind of uh make optimizations that are not obvious that deal with pie charts you need to know it but at some point uh you kind of live within this like end goal repository which is a model right and so on so there's a lot of like open source going on right now rather than in Frameworks uh in in this model repositories which is very interesting I think I want to piggyback off of that um because I think you you kind of hinted at this but I really want to emphasize that I think that contributing to Pi torch is it's a means to an end it shouldn't be the goal um I think that that uh this is kind of what I was saying before is like making sure that you have your goal clear in your mind if are you wanting to contribute to pytorch because pytorch doesn't do something that you want it to do well figuring out how to make pytorch do that thing that's a goal um you know if you're wanting to to be able to optimize the internals of Pi torch okay well that's a goal now you know how to shape what it is that you're doing to contribute to Pi torch so that you can get better at low level optimization um so again I would I would I would encourage everyone to just be really crisp in your goals make sure that your goals have clear success criteria so that you you can know when have you achieved it how are you actually able to move towards that that kind of thing um and I think that that uh you know what Luca was saying about there being a lot of places to contribute to the torch ecosystem um that aren't the pytorch repository itself um in terms of where torch is going into the future I'm going to have to defer I have no idea it is uh it's an exciting place to be but I don't feel that from where I am I have any and special insight into what's next um but that's because I'm over here in dirty industry trying to get things out the door so yeah I think at some point probably some of the things that Luca mentioned earlier uh as the challenges and for example running foundational models on local hardware there will be something that probably where there is too long but it's currently it's like an add-on to cry torch and obviously at some point this will become more natural and maybe less error prone too and so I I think that python might have like a future similar to numpy and that has been going on for a long time and I think we'd see that like a lot of techniques will move from being being uh being external or being add-on to uh getting some some more first class attention if you want and uh and you can already see that uh and this is something where like also things have changed a lot is in hardware and so the gpus got a lot faster and they got a lot more faster than the CPUs in the same amount of time and that shapes a lot of Python's internals um and so I think that kind of we're going to see new things come and maybe some of the things that are hard today become more easy to yeah the the way the hardware is evolving is dictating what the future uh this technology is and how it looks like there are other Frameworks like like jocks and so on that are um that have some some different things from some different angles But ultimately I think uh pytorch has become uh one of the lingua francas because of the of the ux and so the challenge is how do we leverage the hardware with all its intricacies and the specificities that the hardware today has and imposes on whoever writes like low level code to really make use of this huge computational capabilities that you can waste the moment you don't do things in a certain way how do you pair that with the grade ux that you have in in pytarch and pytearch was born at a time when the the balance between how fast you can compute things and how fast you can move things back and forth from memory was kind of you know certain place and now it's in a different place so we see things like torch compile coming up that are addressing this specific problem so avoid going back and forth between like computer memory but try to compute as much as you can for how long as you can and then put things back in memory if you want to really use compute without impacting on the field that the user has in writing programs and this is a big problem and so compilers in general are probably going to be more and more important and at the same time compilers could make the front ends a bit more flexible where the pytorch ux becomes kind of the front end and then there's a backhand that is computed in ways that well were not originally there in byters so it there's a relatively a silent Revolution going on in the in in Frameworks in general and uh yeah we'll see it's a very exciting time awesome thank you so much I love thinking about the future as our ending message um really grateful to have all three of you here I feel like I have learned a ton and thank you so much for everyone who are joining us um yeah any partying messages for us well first of all thanks for uh getting to the book and thanks for having us it's uh you know it's always very humbling uh to think that uh the the work you've done it's like somehow impacting others in their like even in small in a small scope but it's really humbling So yeah thank you so much oh thank you okay uh I'll post the video online um later today or tomorrow and I'll send it to you guys uh thank you so much and I hope to see the second edition of the book soon thanks Sophia thanks for having us yeah yeah thank you enjoy the week thank you