Open Source Office Hours

Key Takeaways

hey everyone er welcome i'm super excited to be hosting the first hockey phase a open source office hours i'm happy to host it with lissandra a core maintainer of transformers and lead of the open source team at hogan phase and quentin the core maintainer of datasets so if you have any questions feel free to ask them in discord or directly in youtube you can see the link to join discord at the bottom and yeah having said that welcome this andre welcome quentin i don't know if this android you would like to introduce yourself and tell us a bit about transformers sure thank you thank you omar so i'm lysander i'm i'm leading the open source team at hugging face i've been part of hugging face for uh close to two and a half years now so i've been involved in many open source projects along the way um and i'll be happy to to give a small update on first well transformers and uh and more generally uh also the course that's coming uh that's coming soon and then yeah so first uh maybe to just quickly mention the course we have the part two of the hugging phase course coming out next monday which is going to be a super super cool because we'll be looking at many different nlp tasks in the first part of the course it was just a single task whereas here we're diving deep inside question answering a token classification and complex tasks like these with the whole post processing and pre-processing in which we dive into so super excited about that and i'm really looking forward to to seeing many community members there for transformers more specifically there are a few uh maybe directions we can have general road map which i can share that would be interesting interesting for the community maybe the first big aspect is that up to now uh transformers was mainly nlp focused in the recent months it has changed quite a bit with the uh speech sprint with wave to vector and the xmsr this is going to continue this way we are going to continue working on the speech capabilities for transformers and for the general plug-in phase i think quantum content you can speak a bit about that regretting data sets too it's also the case concerning vision we've done some efforts to integrate fit the vision transformers bait better clip and other models are to come in the future so that's one big aspect of transformers that's going to to evolve quite a bit the second one is that recently we've refactored the pipelines a bit to fully use the gpus whereas before it was super hard to get 100 gpa utilization uh we've completely changed that for the pi torch side of the pipeline so they should be much more efficient on gpus now it's quite experimental still but we're really really looking for feedback here so if you can use them i would be happy to hear any comments you have and finally a super super cool thing that's upcoming it's not it's not yet completely released but it will be soon is the ability to push to the hub inside repositories actual code actual model code and tokenizer code which can be leveraged then in transformers so it means that if you're using for example you have your own bert model with a few tweaks with for example a layer added here a layer normally there you don't need to open a pull request to transformers to add it so that others may benefit from it you can just push it to a hub to a hugging face hub repo and alongside your weights and tokenizer files and so on and it will be used directly in transformers and other users will be able to use it too so this should unlock quite a few uh big features and just sharing capabilities across models so i think those are the three big points for transformers so maybe omar or carter you can you can take like the lead yeah that's quite exciting just a comment from the community is that looking forward to the new course i yeah and we are also quite excited about launching the second part next week i don't know quentin if you would like to go next yeah sure and i can already see some questions about about data sets so that's really interesting so just be uh before going to the question uh just to introduce a little bit myself so i've been working on the data sets library of hacking phase for a year and a half now and so in the beginning it was only uh it was mainly focused on nlp but now we are working a lot on adding more audio data set and more vision data that's exactly what we are working on right now and one nice thing that i also wanted to to show you is the uh the new preview the data set preview feature that we added on the hub so here if you can see my screen it's the list of like all the data sets that you have on the hub and just very quickly you can see that if we access the glue data set you can see on the top that there is a preview of the data set so that you can explore each data set this way and what's also really nice is that it also works with your own data sets so here for for example uh is a page of one of my own data sets i just uploaded the csvs on the on the hub and here i got the preview in real time so as soon as i uploaded my csv i was already able to to preview the data set so that's a really really big feature that we just added and uh i would love to have some feedbacks from you guys uh on this then uh two other things that i also wanted to show was first the new uh bigger turn software feature that we added to data set which is the 2tf data set which is now like the main entry point to use the data set loaded with the hugging phase datasets library to use with tensorflow in your training and you can feed it directly to the model.fit method like but in keras for example and finally the final change i just wanted to mention is that we we have now like really nice features or audio data sets here so here for example you can in the documentation explain how to load the common voice data set which is a pretty well known audio data set and in it you can get access to all the audio data directly and it does all the decoding of the mp3 into the arrays on the fly so don't you don't have to to do that uh yourself but so that's pretty much all i wanted to uh to say so i can stop sharing my screen now all right so that's uh that's a lot that's pretty much it awesome thanks a lot uh yeah so now we'll spend some time answering questions from the community feel free to make them directly in discord or make them in youtube yeah so let me begin with this question some from brown bandrai are there any plans about new products in the pipeline and what is the role of open source in those pro in those products because we love open source and we love it as well i don't know if lissandra would like to answer that one yeah i can say that um so i don't think there's any plans that uh about new commercial products that we can really share but there's the uh we're really going to push forward with the uh platform aspect of the of the hub so to have this central place where people where users and community members can just share their models not necessarily across transformers but across all possible machine learning frameworks so may that be a spacey team lnai and then or transformers um and so since this uh since this is definitely going to be integrated with uh all the uh like communities and open source software that would like to be integrated with um then we're really going to push that forward and of course the open source will be a core aspect of that platform that we're working on um so i guess something that we're looking forward in the future is just yeah making open source even more accessible uh by simple platforms very simple uh usage and apis um across open source awesome thanks er yeah then next question is how can i start learning about transformers and contributing to the hog and face ecosystem okay so i think those are two different questions the first one is how can i start learning about transformers well i guess here the the best place to start is is with the huggy face course part one was released a few months ago and part two is released next monday so i would definitely recommend this as the most useful for learning about transformers since uh it doesn't teach about transformers only but about the whole hugging face ecosystem with data sets with tokenizers with the hub so you get a full spectrum of everything that's going on and regarding contributing to the hugging face echo system um i guess here the easiest thing to do is just to go inside the libraries look at the issues look at the pr's we have some good first issues tags for issues where we think that people that are not yet introduced to our libraries can still contribute that's definitely the best place to start contributing in my opinion maybe content you have something to add regarding contributing that's uh that's pretty much it and yeah don't simply don't hesitate to uh open issues and to discuss on github uh we uh like we have a full team working on helping everyone getting started and uh if they have questions so yeah don't hesitate cool thanks uh the next question is from why this yeah uh he's asking or he's saying that i am curious if it would be interesting to have comments uh sections in spaces demos i think it will generate more interactions and energy yeah so this is something that we are discussing internally uh thanks for the product feedback and yeah thanks a lot okay uh i think you already answered a bit of this uh when you presented this android but maybe the two of you would like and also you're pointing with the audio a feature in datasets but maybe you would like to expand a bit more about the plans for the future right so um in terms of open source libraries yeah we're definitely opening up to new domains we started with nlp but now we're reaching to audio speech which is quite mature now but also computer vision in transformers we're on we're currently integrating the receiver model which will be the first module that can also support video video tasks we're not limited to any of those fields we're really looking forward to being as helpful to as many community members as possible and regarding the hub i think this is also the case where we will continue to try and integrate with as many different open source libraries that could benefit from having a hub integration in quarter maybe you want to add something about data sets well yeah but on the dataset side the thing is uh keep adding more android more data sets especially for audio and vision because for many many of them it's still really complicated to download them and just use them in a limited uh resource machine i mean so in datasets we are we plan to add more and more of them and we have all the best optimizations to load the audio and image data awesome thanks a lot i think grenting is the most prepared person to answer this question so the question is are there best practices for dynamic data sets for example a data set with a weekly snapshot how is an update best automated yeah thanks for the question christopher so it's really it's really interesting to have dynamic data sets you can think of wikipedia for example like each like every month they release a new dump and you can download the latest updates from wikipedia and usually usually the best practices i would say would be to uh if you want to have a dynamic data set would be to have a standard way for each release like to access the data so for example if your data are hosted on the hugging face hub you can have like each version in a different directory for example of your data set and then in the code you won't have to to modify anything regarding the way you load the data set except that you can just pass an additional parameter saying uh which version you want to load them and so as soon as the new versions have the same formats or stored in the same structure as your the old version it's uh something that that you can do awesome i hope that answers the question yeah and if you have any follow-up question feel free to ask directly in youtube or in discord okay the next question is a jax tensorflow and by torch are all supported in transformers which should i use which is holding phase strategy moving forward with having multiple laboratories that's a that's a good question i think here um jack's tensorflow and pytorch are all very different frameworks with um even if they solve some of the same problems they're each targeted at some specific problems like for example tensorflow being way older has an ecosystem which is way more mature for example with uh relation to serving and to efficiency in production pytorch is still progressing quite a bit here with a torch serve and jax is super efficient on cpus but you're basically going to use the libraries for different use cases and so i think here we're really going to focus on the aspects that make each library shine for example with pytorch it's super malleable it's super flexible with what you want to do with it so we're going to continue implementing very researchy type of models in pytorch as it makes little sense here and it's definitely designed for that and for tensorflow we're going to continue optimizing for uh while simple usage simple uh integration with keras and just all around uh high performance and production great setups and for jax uh here it's a bit different it's it's optimized for parallelization so we're going to continue with that moving forward especially focusing on maybe bigger models which fit on big tpus basically just trying to focus on what makes each library shine and adapting transformers to that so if you're looking for a general approach i'd recommend pytorch to start off because we have uh because departure side of transformers is the most mature uh but if you're looking into like very high parallelization very high performance i would still recommend jax and if you're looking to look for something uh very careless uh dramatic i would of course recommend tensorflow thanks there's like a direct follow-up question is the ucf tutor for yx xla is also available in by torch uh yeah of course we see a future for jax otherwise we wouldn't have converted a lot of models into jacks um there's definitely a future for jax uh maybe not as big as uh as tensorflow pytorch uh because right now it's more it's more niche but we really trust that it's going to uh continue working on itself and improve as time goes on of course there's a xla available in high torch and tensorflow as well um but jax's approach is still a bit different and you can do things in jacks way simpler than you can do them in python or intensify okay thanks a lot we have two very related questions so let me show you the two questions first and then we can answer them so from andrew mean in discord they are asking if that hogan faces currently looking for machine learning engineers and what are the top qualities that we are looking for mle and another question from discord as well from brandel 1507 is what should i be working on if my aim is to land a job at hogging phase uh so yeah maybe a quentin or lysandre and if you would like to share a bit more about what that ml engineer does at hoginface go ahead uh if you want to start okay sure um so i think working working at hugging face requires a bit of a different mindset to to traditional machine learning engineers because what we're working on at hugging face isn't uh actual models like we're not training models and we're not looking for the best uh accuracy here at the best performing model we're actually building tools to do that so it requires uh to to think of it differently than just working on the application and it requires uh understanding what you would be really looking for when working at hugging face is it to make software that's accessible to as most people as possible is it to make software that really revolutionizes the field rather than optimizing a model and working on data centric aspects of the job i think uh usually yeah the job of ml engineers at hogging face is why it requires the same skills as other machine learning engineers it's still a bit different and and it requires a different drive i think what do you think yeah i totally agree one thing that i would also add is that in addition to all the standard the machine learning engineer skills that you might expect you also need to have like to be very close to the community and to everything that is going on in the in the fields like nlp vision or audio uh to be able to uh to know what are going to be the the best tools and what are going the best ways to develop and democratize the access to all these models and data sets um so yeah if you are really close to the community to what everyone is either be doing and also to the research what are the the highest like the biggest trend uh recently it's really a big plus and and to add to that i think that the second question was about courses and uh and just machine learning courses and if that would help uh personally in my uh i took a computer science degree in a master a few years ago which didn't have a focus on artificial intelligence or machine learning i i got all my expertise in the coursera's courses which are still not beginner level but a bit so and still like because i was really really motivated by the field i learned a lot on my own i worked a lot on my projects and uh having a strong software engineering core with a huge drive and interest for machine learning can definitely lend you a job if if our values are aligned awesome yeah i fully agree with everyone everything that both of you said yeah going to the next question i think lysandre pretty much answered this before so the question is is the focus of hacking fees only on nlp frameworks and the answer is no we are building collaborations with other libraries in other domains and also transformers is starting to be applied since few months ago maybe more than a few months since er early summer in computer vision and audio we don't have a much work in 3g transformer right now but that's also like an area of interest to us and there's also the very recent widget that was introduced uh maybe last week about image segmentation which is really really cool they are amazing that you must check that out it's insane yeah we i'll share the link in in the comments in few seconds all right uh so this is a question from geeker decodes the question is i want them to understand if there is batch processing supported across attention heads in multi-head attention of a bird engaged by torch model i guess the question here is to is to understand if we do a batch uh batch processing of sequences inside bird and uh and yes we do we don't uh um like you can match up to as many sequences you want in parallel it just depends on the amount of memory that you have and it's not only for a bird encased pt model it's all models and transformers all checkpoints that's that's one of the core principles of each of our models awesome thanks we have a question from emmanuel hoover new architectures are going to be implemented in transformers or separate repositories as mentioned that's that's an excellent question um so up to now uh we've been mostly implementing well we've been only implementing new architectures inside transformers via pull request but as i was mentioning in the updates we're going to offer the ability to push your own code to a hub repository and then use that as the uh as the central uh definition of the model um well this uh well we're going to push for this we will still uh really motivate people to share their models inside transformers uh as much as possible offering this is basically to remove the constraint to have to open a pull request you have to go through the entire review and merging process uh if you want to get something out quick if you want to how to to share your model and and basically share it with your friends and just show how it's working but if you have like a research project or something that could be really helpful for the entire field then we definitely recommend opening a pr to transformers sharing your model with others this way we can have a thorough testing of that model we can test it our ci documented share it with our community members it's just the best way to have more visibility on your models awesome thanks a lot the next question is uh is there a forum it is like to discuss with hogging face members and ask doubts on how to get started on an issue i want to work on i think i can answer that one so first the link to the discord is below and there is also a forum but normally our philosophy is to discuss directly on the issues in the github repositories that helps having all of the discussion there and it's like more dynamic we can discuss directly over the code so yeah i think that's the best medium to discuss but maybe quentin or lysandre have like other suggestions i think you covered it well one more awesome okay uh so we only have like uh three more minutes uh so let me put the last few questions uh sorry yeah so a kicker decodes ask do you plan to support models with pytorch transformer attention encoder decoder module types instead of custom modules that's supported as of today so that's a very interesting question too which which is very linked to the philosophy that we have in transformers the philosophy being first we want the models to be extremely readable like we want people to be able to go and set the code inspect the code and just understand what's happening inside the model and the second one is that we want the whole processing uh to be done inside a single file so we don't want uh so for example we're never going to have let's say a attention layer that's going to be shared among all our all our different models because when inspecting it would mean going from the basic model implementation to the abstract implementation and just having a back and forth between the two files which would make understanding quite painful but regarding the question to exactly support python transformer attention or basically the standard modules offered by pytorch um we're definitely not closed to having support for those uh basically if if someone contributes a model leveraging those uh modules uh we'd be very happy to welcome those to the library for our existing models we're not going to refactor them to simplify them simply because that would be a super big change that would probably hinder comprehension of the models by our users which isn't something that we want and it would result in modules that are way less flexible than what we currently have for with our own custom implementations so the answer is in the future if there's some new uh models that are offered with those no problem at all but we're not going to reflect with the old motors let's hear about the news no worries uh yeah so let's answer like three last questions so this one is a quick one are we going to discuss prs from the red point office hours now the idea is to discuss more about the strategy new updates and high level questions and not the super specific things for discussions we usually discuss directly in the prs a few people were asking about julia so the question is to use julia or plan to use julia would you like to give us high level view of the stack um i think omar you have the most experience with uh with the julia related uh hub integrations though yeah uh sure so right now there's like this uh so right now we have a library called hacking facehub it's like a very nice python client that allows people to programmatically upload models to the hub download models from the hub it doesn't necessarily need to be transformer based models so we have mixins for keras for example this is being used by other 15 libraries this is used with spacey for example to push models to the hub and right now there's like this nice integration with julia that allows people to download models uh using this python client to run the python code in julia and to download the files directly from github so there's already like an option to upload and download models from github using julia and use a version control and all of the other benefits that github offers you with julia but for transformers library there are like no plans to have something like that with julia okay i think we can answer let me see yeah i think we can answer like this last question and finish for today this is mostly for quentin i would say what is the upcoming plan on data set to fine-tune the models and the chart yeah i understand the question yeah actually i'm also not sure i understand it uh hey i wish feel free to follow up in the discord uh let me put the discord here we didn't get to answer all of the questions so feel free to keep asking them in the office hours channel in the discord server and afterwards we can answer any follow-up questions or clarifications yeah anyways thanks everyone for joining thank you alyssa andre and thank you quentin for joining us today thanks everyone for being here thank you everyone for being here thank you more for organizing yeah thanks a lot see you all