MMF, a PyTorch powered MultiModal Framework

Key Takeaways

hello everyone I'm a man and I'm researching yeah that face okay I research today I'm going to talk about our research framework MMF which stands for multimodal framework which is used both for research and production inside Facebook and I was actually talking about how you can use them into your vision a language research what we've been driving for the less talk what what do we mean by vision and language terms here in specify an anonymous language and it has that involves a visual and textual component and requires reasoning over port to solve it we would call it original language tasks so let's say we have an image and text source which is passed through model order does is full of magic and runs back and output they see a scenario we will be limiting regional language tasks inside MF so I think that fits inside this particular scenario is a vision and anger task for M specifically we are not concerned about video tasks here we only concerned about image which tasks here we are blind when we get a sooner or later inside left so keep on keep an eye out for that so let's first talk about an example of within a language task we hold we dive in further so which in question involves answering a question there's about an image and text VK is a variant of week it involves answering the question but the question requires a written text in the image to answer that so let's take an example of bqa tasks we have an image and a question that is basically about that image is opening into the image we can see that we require reading text in this image to answer the question oh you can see where exactly the cars are to see whether the left lane is closed so not all that clean is closed so the Mirotic's in both image and text does raising over it and should return back left then the another task that is involved is very popular in regional language to me in its image caption we'll be taking an image possible through the model and generate a caption based on the image that because it is a variation of image captioning similar wrote xvq but which requires reading text in the image to generate the description then we have integrity classification tasks such as hateful means which takes in an image there you can see there's a meme it's look how many people love you and there's a lot of people inside this image which basically this means it is not hateful so the model takes in both image as well as a text in sign language and classifies it with notes it MF contains the reality of these tasks which are divided into different of means these domains of I know me is like restrictive so one task can be in like multiple domains like SN IV I can envy integrity classification as well as using a simple visual question saying it can be sighs we quest as listening so we have like various kind of tasks from with your question on singing image captioning to dialog and integrity classification inside a member and we keep on adding new and you task every day inside an image so MMF is our multimodal framework which we have built from scratch there's some fire coverage for first division and language research so Emily for used to call fighter which many of you might know about so we have a blended by T as FMF now moving from IPA to M we have made a lot of performance improvements the same imposing motive is used to take around sixteen hours to train on eight GPUs now takes only like five hours to train on the same amount of infant compute so we have a 3.5 x percent improvements inside a max speed when moving from five levels of 0.3 to the current and we're so why do you want to use what are the major reasons why would you want to use a mess so first of all MF involves less boilerplate so what do you buy that so running a derivative point Union from create a more I want to note some sorta models it just involves this much of code like specifying versus your world sighs which config which you want to run on the data set model and which particular Zoo model you want to resume from you had to only specify these many options everything is configurable so everything from distributed support to configuration to checkpoint into holiday shopping has been already equal to you so let's go through some examples of what exactly we have in the boilerplate you don't need to write this again again and we'll provide out of the box so common metrics and those is used in week all like regional language tasks something like you equate courtesy or something else then the distributed trainer will be provide a lot of sort of models you train models of you which you can just directly import for MOF and download them and use them directly you provide automatic downloads for the dataset we run the command dataset is automatic download for you and then you can go on with your main thing so and then we also got a white pre-training fine tuning approach that you can create in a model then you can take specific parts from that motor than fine tune them then you can do - parameter optimizations you can launch large sweeps and it's very configurable you have a lot of V and L encoders and modules which can you can use out of the box in plugging them into your model try different encoders see how they work with your system then we also have V and L optimizes and schedulers customized for our use cases as well as regular V and energy use cases so you can also like at your home optimizer schedulers and there's a lot of things so using my we want you to focus on what matters most test your model and we will take care of that is like you don't need to setup all these things all the boilerplate code again and again you can only focus on your model just be handle for you mmm is powered by touch which means you can basically use everything that should know about by touch inside a McGrath you can create your PI touch base data sets you can do models you can use all the five colors modules and whatever you want so powering by using Wi-Fi torch we leverage everything that white which provides inside and so all of the components in my touch power even basic modules and everything else is it about scratch using fight watch so you can extend them use all your fighters knowledge base really use them however you would like so 13 is that MF is modular and composable hazardous is extensive so what do we mean by that so mm feels like a registry system where you can register your custom models use basically this allows you to like use a manifest library you can beep install the package then use the use user register system till that time enough know about your model and then load your model use a strainer this will allow you to write custom libraries independent of Mme so you can register your custom model you can adjust your custom data side you can register your custom process so you can assist optimizer loss trainer metrics and lot of so MLS is like configure from ground up so you can override everything you can implement all form all how you want using whatever em with provides wherever you need so that allows it to be very modular as well as configurable so we have your design MMO from scratch in a way that people can easily use it however they like let's dive into some stressful details of how you can use enough in your custom projection how I'm gonna handle various things and how do models and data sets work inside element so let's take some specific examples again so for Bishamon saying you have a image under text it text is the code and then you pass these two possessors so you can think of the senses as torch regent transforms which takes in an input and an output so this key allows us to keep our data set modular while we using the same coding to extract data sets so all this is like on flavor from data from configuration system like there's a text versus of if you want to use glove then you say that I want glove text possess so if you want to use Burton coder you say you want but text encoder similarly go image you can Expensify all the transforms you want for fast X at which whatever kind of embeddings you want so these go through so you can make a sample sample is a custom custom object inside inside which represents a single element from the data set these sample combined in like that convert into simplest so simplest is like a temper of a batch which allows easy access to different kind of attributes inside batch this this allows us to like decoupling data set from models so let us return like a simple list and models take an example this so this allows us to use multiple data sets with multiple models and have a decoupling structure looting them so let's see how we can connect these data sets to the IMF model so I can take an example of bqa that we took in start so there can be cases like in this one like you can have a OCR text which are like other modalities you shall not the common thing inside you I decide something like OCR or something like face features or something like other embeddings which you don't normally use but this can be other modalities so image goes through image encoder which can be of different types and is configurable so it can be last night twice next for a scene and or something else similarly for word text you have text encoder which can be many times for example you can use power robot on low-cost xixo mod or anything that forgive this transformer provides we have a dependency on having first transformer which you can use which allows you to use anything from that package and these are all configurable to config system or family then similarly for other modalities you will have a particular mortality encoder which can be something specific to that molarity for the example for fast text you can have different kind of encoders which convert them into fast text or dr. seuss intent or you can do something with it so all these modalities modalities paths go through code get back a representation which is pass through some kind of fusion mechanism to return back a single single embedding so these fusion encoders can be also a different time for example recent regional language model involves transformers something like Wilbert which will go to something else then late future models element-wise models which basically take element-wise product of the encoding to generate a final embedding then it can be MC vo you can use a tension with fusion like the older idea base model you so finally once you have a fusion you pass it through multiple kind of heads and these heads gives out and output all those which you can use to propagate through a whole thing so let's take some specific task examples to understand what exactly we mean here so you use your question and saying we'll have a classification head which is to basically generate lab then a visual entailment which again has a classification head and the task involves an image and attacks on you have to tell whether the text is like entailment contradiction or neutral with respect to the image so you have 3-way classification and the conservation healthy is configurable for number of classes you want to output so in this case like there is a blood choke on the left or roadway and it's actually on the right side also it's a contradiction here then for image captioning you can similarly have a generation head which generates some kind of text for the in this case particularly takes in an image and returns back oh cap similarly for integrity classification you can take text an image and a being pass it through classification and then due to a classification of hateful and not full let's see how is that we can do retraining fine-tuning inside a member which is like a very major use case in current regional language scenario given all of the st. indentation of multiple transform based within a language models like we'll be sure about and others so you can take in like for example mass language pre-training where you mask a word and you have to predict like that word so you will just apply your pre-training head on top and predict the mass token that's left so you can similarly calculate the Lois here and just please back propagate through the whole all on so you can take the art heads before retaining head and just copy paste it inside here for your question and sing I had a classification I don't talk sorry placing this head is very easy inside here you just specify your pre train straight mapping and you you can just get the same thing back so on the side I show how it's possible through the contributing system in checkpoint you say that this is my resume files like this the pre-trained file to pretend file checkpoint and then you say I am presuming operating model and then I specify the pre train station having which match model to model so it won't load the head so after learning through this it will start classifying this answers this questions answer is left so the overall end-to-end flow of F is like you taking a image and text pass it through the motors and then you have like different kind of heads on top which you can replace and then you can use the same pipeline again and again five five by using pre training and fine-tuning so this allows us to do multitasking inside anywhere and you can you can pass in through multiple data sets inside I'm going through the same model and they can all use their own head and this way you can be allowed allow multitasking inside a member which is an interesting feature for use future use so it in models inside and we belong to multiple categories we have a lot of models inside so they can be like multimodal the transformers like visual world Wilbert pixel art and MBT and dynamic Anza space models like Laura and m4c which are specifically useful for text we kind of problems where they point to some my console space so distances like there can be like very different type of models which tells again about like how flexible is the car with regard to your model like we make no assumptions about your model so you can hardly do anything you want late fusion Jenny model as ancient version of this model that all that's why I think we small and then also database models so possibilities are endless so and then you can see well I'm done with your project and you want to publish here create a model we already provide a lot of beauty models inside Emma laughs so you can just do this three line code for classifying some URL and text into gates so an important thing is not a phase configuration system which allows amber to be as configurable as it is right now so it's according to like have a look at it like how exactly we do configuration inside a member so there is like a default configuration inside which provides basic defaults for everything inside for example training something like snapshot it as well all maximum updates best size checkpoint how do you exactly want to resume do you want to lose you from 310 file with zoo or something else the distributed set up then scheduler attributes and all lot of different things so this is like a base of our configuration so our computer system is on the accounts base which is what Hydra is based upon so it's very configurable in the sense that it allows you to do various kind of things with camel files includes and other things so then after the default configuration we have Perl data set and model configuration which defines the defaults for that particular models all that particular data set for example here for MB TV ID you find the hidden size at 768 number 12 and drop out of 0.1 simple it is you can define where the features are and then these configurations are used metadata second mode of builders to middle the model in the way the user wants so this is all very configurable so the user can configure based on their particular experiment they can configure all of these defaults and update them based on their own parameters let's say I have a particular version of mmvt where I'm using hidden size of 1 0 to 4 and I want of a size of 32 I will fix this inside my configuration and that will be my experimental experiments configuration which I can distribute with other users so that they can directly get the same setup essence then for running millimeter Suites everything is configurable to command like pops so you can basically parts training board size is equal to 64 that will upgrade update the user configuration and all the confusions before to have the best size 64 and then you also specify data set and model here and then you may use the MMF in the squadron commander on this so this allows us to live run hyper parameter optimization at enter last case so for example you want to like do it with size in l a-- so values of the bed size lie between one five hundred twelve thousand and twenty four and either lies between 1 e minus five and five e minus five so it will run a grid search over all four by passing them as a command line argument and overriding whatever it requires so we have a grid search and I cover my optimization utilities inside America including visualization you to reduce that can help you maintain launch and maintain experiments that very last MF is made for scale this is very evident from the decent anti report form ad hoc group where we conduct if I wonder plus experiment easily each on 30 which we use as you can see is a table there are so many combinations that each have it onto a very expensive a kilometer suite to get the best values out of it so n the models are very big including visa bottom bed apart so we were easy able to conduct more than 500 experiments on 32 GB using MF without any it comes so work is made for scale feel feel to use it like for any kind of experiments that you want to use before obviously it's inside the package itself so there are endless possibilities using these are all projects that will run inside they are using ever for the projects which whose implementation has been included inside MMF and IMAX has like made it possible to make do very different kind of projects and easily like Mpho see and it will use a very different kind of projects which are easy done and even the same codes is based on the configurable approach that we have it and so I work a lot of projects possible we hope that you will use Angra for your own project as well in future so we are looking for the community contributions and we are always open for any kind of contributions that community wants to do you want to use your favorite model inside and like I feel free to like optimal PR and we can also help you to learn that we are inside anyway so is possible due to a joint collaboration between a lot of teens and folks this includes but not limited the people listed here so this has been like a very joint big effort inside into Facebook and it's being used in production as well as the settings so here are the resources that you can follow if you want to learn more about unless looking forward for you to use anywhere in your next project and any feedback that you have regarding that thank you