Transformers for Vision Tasks

Key Takeaways

i think your next point is maybe closer to home for machine learning and computer vision and it's uh it's a big one that has a lot of people talking uh and it involves uh transformers not the the robots of course but the transformer networks that we all know and love uh what are you seeing there yeah uh i mean this has definitely been a highlight for 2021 computer vision we have seen transformers finally coming into computer vision transformers is actually not uh like in natural language processing nlp uh researchers that have been working with transformers for a few years now um but they have finally made it to um into computer vision um and it has been an extra exciting times because we now sort of are seeing this line of work where we are slowly replacing cnns uh with transformers and i do for you know various recognition tasks um and in particular we've also seen uh what is exciting about transformers and computer vision is their impact when it comes to working with extremely large-scale data um so it's still quite unknown to us whether transformers work well in the low data regime this is something that is the answer is not out yet but we definitely know that for large data we're talking hundreds of millions of images and more transformers are quite impactful and is the idea that transformers can create more robust representations based on lots of data than a convolutional networks yeah so i think that the um so again this is uh we don't we don't quite know you know and like theory behind these things is kind of always unclear but uh so the the main uh sort of highlight is that so cnns which is the predominance or the tools that we used before uh to represent um our visual inputs uh had were had inductive biases and this comes from exactly the structure of convolutions and the image grid now transformers take a completely different approach they actually treat images like a sequence of uh tokens so this can be patches of images so like like a small neighborhood of the image um and they uh and the only inductive bible that comes into transformers is through this um this serialization as you would say of the image and the pose embedding that comes in when we are processing these inputs but other than that everything all operations are sort of global with an attentions uh being cast into these representations so there is no other inductive biases in these networks whatsoever so that means that they have the potential to be a lot more powerful because you're constraining them less uh but in order to achieve that you need more data so um so it is a much more it is much closer to to to actually having a true function function approximators like we you know what we say like mlps are great function approximators but um with cnns that was a little bit taken away because of that particular structure and that transformers are bringing this back to life which is an exciting exciting and has proven also to work in for images and how would you characterize where we are are we in the you know just kind of demonstrating that it's possible to make it work stage or have transformers in computer vision demonstrated you know state-of-the-art results or um you know either better results on um you know known tasks or tasks that we were doing pretty well on or you know allowing us to uh perform tasks that we weren't able to perform well with cnns i love that question and i hope that my answer is not going to anger people but um you know i it's a great it's a great question and i hope and i wish we asked it actually more often um so you know before i answer that i'm just going to go a little bit to the nlp world and say why i think transformers are have had and tremendous success um i think it's partly due to two reasons first is that you know in nlp uh you can actually get a lot of data um and that's what works like bird have shown where you you know kind of crawl the web and you get all all sources of text data there's a lot of that there's a lot of that out there yeah and now you show how and it can be anything you know any any text whatsoever whatever content and you feed that to the beast called transformers and it builds a representation so that's fantastic and a first great milestone and the second one is that they have all these fantastic tasks diverse tasks like question answering um text generation uh or text fill like they their tasks are endless and they are very different from each other and they have shown that even in those very diverse tests having a global representation coming from you know learning through self-supervised learning with a lot of data is helps all these tasks significantly now envision um in computer vision things are not quite like that folks that follow computer vision probably have a sense for where this is going exactly correct so you know we are kind of in a weird awkward phase where we kind of want us to sort of replicate that line of work from nlp we're very much inspired and and looking up to them but we don't have exactly first of all we don't have a big data set our biggest data set is something like imagenet and i'm talking about public data set here uh something like imagenet which is one million images and imaging has been a fantastic data to serve the community for 10 years but the problem with imagenet is that first of all it's been around for now quite a while it's a frozen dataset so nothing changes so as we develop new ideas on a dataset that is constant and kind of frozen time we tend to sometimes our idea is overfitting on that data set and these um a lot of a lot of the gains that we see in images don't actually transfer and that makes sense you know we've been we've been dealing with this data for a while now um i would actually say that maybe we've trained more models on pixels on imagenet but you know i'm not going to get there um and then the other question of course is what are our downstream tasks that we're showing that performance in and this is another sort of sore point for for us computer vision people where we're also seeing a little bit of uh of a you know kind of sad state where we know we have our object detection and maybe our our segmentation but they're all essentially classification tasks we don't have the richness in output and in test that nlp has all of our like our pre-training is classification our downstream task or classification so it's really hard to tell if we're actually making progress or not so this is something i think that it is for the community to think about uh i guess a couple of reactions to that one is maybe the the second one um informs the first but you know there's classification but then um is the idea that all of the the complex problems that we think of like uh you know bounding boxes and all these other things that kind of just boil down to classification it so at the end of the day it's all classification even if it you know looks uh more complex correct so there isn't that's that's absolutely true and um i mean our most complex task i would say maybe is objection because it involves sort of maybe predicting intermediate boxes and then on top of that you want to predict classify the object type or maybe predict other 2d properties by classifying the pixels within those boxes but even that is cast as a class and what about things like uh like vqa visual question answering is that more do you think of that more of an nlp task than a vision task or the only thing i have to say there is that um and this was also a point made by you know clip a gray paper that came out last year is that even there the data sets are very small and we really can't tell how effective these methods are because we really don't have a good sense of these tasks um and in you know i mentioned clip which is a prime example that showed that all the gains for transformers came when they downloaded that huge data set of theirs like i think it was 250 million if image text pairs we don't exactly know the source where the data set is from but um we know it's from the web and this is how and they were only able to show that amazing property of capturing jointly images and text when training on such a huge data set um so this is sort of the what we aspire to to get to um having training on such large images uh and of course that comes with a lot of other questions about you know are we doing our due diligence and making sure that this is a good you know data set in terms of ethics in terms of like content like are we making sure that it's uh unbiased data and so forth uh which brings of course new challenges but i feel that for we need to move to that scale or otherwise we're going to kind of be stuck in our little imagenet sort of standard regime which is not a good state to be in yeah yeah i think the the data set side of that was my second question and that was we you know we hear about and see new image data on the one hand we hear about and see new image data sets you know all the time they tend to be specialized and fairly small you know on the other side just like nlp had access to the web uh vision has access to the web there are tons of uh vision or there are tons of images on the web you know do you think that the in order to get beyond kind of image net and classification you know we need to define like unsupervised problems or semi-supervised problems uh beyond just kind of simple label you know supervised learning types of problems is that the big um you know one of the big barriers yeah i think that's definitely one big barrier um you know the the problem with vision and actually with all datasets is that um collecting a large-scale dataset by crawling the web and just releasing it is almost impossible and maybe for a good reason as in you know you have to make sure that you have consent from the creators of the images to actually release them you also want to make sure that uh you know we are now entering the stage with in ai where it's no longer opportunistic we need to be very cognizant of the data so we're using the models we're putting out there how they can have like maybe they can have potential harm and so these are all aspects of our work and we're responsible for it it's not like someone else is responsible so there is actually releasing and collecting and releasing a large data set is is a huge responsibility um that comes in a lot of work and that might be a roadblock to seeing larger data that's coming out i don't have a solution to that problem but i feel that you know if there is motivation to go there i think we'll make that happen and i think that we also need and this is maybe my own personal opinion we need to rethink the problem that we're solving these pretext tasks that you know are commonly referred to as sort of the the task that you try to solve either in supervised learning uh for example in bert it was filling in the like you know masking words and filling them in we've seen recently uh work from actually you know my lab and coming in colleagues where they do the same exact the same thing but they're trying to they mask out patches of the image and they try to fill them in um these are all this is a great task but it is still still sort of you know constrained to be on just in predicting pixels we need to go we need to enrich in are these pretext tasks if we want to maybe solve for richer downstream tasks moving beyond classification i don't think that having a simple pre-text test will solve complicated tasks down the line like you know maybe 3d reasoning or 3d understanding which is a field that i've been working on i do think that there's limitation in transferring that so we need to account for that and so but i think it's a good first step i think that we just need to broaden our horizons a little bit and what problems we're solving how we're solving and how we're thinking about building representations either through self-supervised learning or through supervised learning with labels but move out of this you know image to single object label let's say