Vision Mamba BEATS Transformers!!!

Key Takeaways

now we have Vision Mamba if you're not familiar with mamba mamba is supposedly an alternative neural network architecture to Transformers and we have seen different versions of Mamba in couple of weeks back and now we have a vision Mamba which means Mamba would work for vision data not just text data and this is quite exciting for a very great future so in this paper we're going to learn about Vision Mamba and how does it Faire against the existing Transformer based models for vision task the paper is called Vision Mamba efficient visual representation learning with bidirectional State space models this paper is coming from a bunch of Institutions from China I guess um I mean like mostly it looks like that so what is this paper is trying to do here is so this paper is trying to use this new architecture called Mamba for a vision task now if you're not familiar with vision task or if you're not quite familiar with what we mean by Vision a computer vision is a deep neural network subdomain where you use images to do certain task for example you have got an image uh let's say an x-ray image and from that you need to classify whether the patient has got a pneumonia or not a pneumonia and that is a classification task you have an image and you want to detect an object in the L you want to see whether there is the presence of a car that is a detection task then you do segmentation where you pick different images so these are different kinds of Downstream tasks that happen with image data set and this is you know largely part of computer vision even though we have been mostly dealing with large language models that are predominantly texed recently computer vision is very critical it's critical in a lot of different places because humans create a lot of videos not just the videos that we create on smartphones but we have got like cctvs uh we have got like this surveillance cameras we have got now we have got drones and all the other kind of stuff so people have a lot of videos and they want to process it they want to figure out stuff a lot of retail agents use Vision data set to figure out what is the product that somebody's interested they want to create something like a bounce rate like whether you picked the product kept it back so vision is extremely important and scale is a very important aspect scale and the latency is also an important aspect in Vision unlike text so now having said that what they're trying to do here is they're trying to see okay Mamba has been doing really good for the long sequence modeling of text so if we can do something which is building efficient generic Vision backbones purely upon ssms is an appealing Direction the problem here is the representing visual data itself is a challenge for ssms due to the position sensitivity of Vis visual data so if you see text text is a sequence but unlike Vision which has like more you know complications and the requirement of a global context of the visual understanding so even if you split an image into let's say grid of 8 by 8 you still have to have that Global visual understanding the global context about what the entire picture is it's not like you can see eight individual pictures and then you can say okay this is maybe a picture of Mona Lisa so that Global context is required also the position sensitivity of visual data so in this paper what they have done is they have shown that the Reliance of visual representation on self attention which is the Transformer based model model is not necessary and propose a new generic Vision backbone with a bidirectional Mamba blocks which they are calling as whim uh I think at this point we have so many whims so this is one of the whims of it which marks the image sequences with positional emings and compresses the visual representation with biral State space models the most interesting thing is on image net classification which is a very popular data dat set Coco object deduction another data set or benchmarks ad 20K sematic segmentation Vim achieves higher performance compared to wellestablished Vision Transformer models like deit deit in this case is a paper or a model from Mai I think it stands for data efficient image Transformer while also demonstrating a significantly improved computation and memory efficiency if you have seen my previous mamba videos you know one thing that people are obsessed with a Transformer alternative architecture primarily because they want something that is more memory efficient that does not increase a lot when you increase the context window or input so in this case it could be like the resolution so for a higher resolution like 1248 by 1248 th2 1,248 by 1,248 Vim shows that it is 2.8 times faster than deit and saves 86% of GPU memory like the transformer architecture would have hit o out of memory in this case while Vim could actually perform the task for the batch inference to extract features on the images of a resolution 1248 by 1248 I think this is huge this shows an exciting future I've I've been impressed by previous Mamba papers but showing this kind of uh scale this kind of efficiency with vision is completely I would say insane and exciting if you purely want to look at the benchmarks they've got Downstream task classification semantic segmentation I'm not sure in segmentation some segmentation detection and if you see deit and wiim so they've got like this two different models couple of benchmarks and if you see here Vim the vision Mamba scores above every single um in every single instance the vision Mamba scores above the existing Transformer equivalent that you have picked and at this point if you are familiar with computer vision models especially the Transformer based models you might be asking a question yourself but you know why don't they use vid the vision image Transformer I think it's from Google rather than using deit for comparison down in the table they have got WID as well so we'll quickly see that but the speed comparison meanwhile here is that you can see for 1248 there is a huge difference it's also very important to note that at 738 pixels by 738 pixels you don't see a difference between the Mamba based architecture the vision of Mamba the whim and deit so you don't see any difference here the difference starts coming up once you increase the resolution size which is very important and critical for example you want to do satellite imagery you've got a high resolution picture very high and you want to go deeper to see something or imagine you have got an industry where you have got like you are generating pcbs for smartphones you want to detect whether the PCB is faulty or good these are like really high resolution task and resolution matters a lot there and you can see that Mamba Vision Mamba whm does quite well in these kind of tasks and it is also faster and it doesn't eat your GPU it saves 84% GPU here and that is like 86% yeah 86% GPU here and that is quite interesting and amazing so what is this paper doing this paper is doing four things for us one it introduces Vision mamba or whim which incorporates bidirectional SSM we'll quickly see the architecture for data dependent Global visual context modeling and positional embeddings for the location of our vision visual understanding without the need of attention with without Transformers the proposed whim has the same modeling power as wit while it has only a sub quadrant quadratic time computation and linear memory complexity so the memory is not like quadratically increasing or the time computation is not quadratically increasing it is a sub quadratic time computation specifically VM is 2.8 times faster which we have already seen so we conduct extensive experiments on image net classification blah blah blah and it says that this result demonstrates that viim achieve Superior performance and finally benefiting from the efficient Hardware design of Mamba whim is much more efficient than the self attention based deit for high resolution computer vision tasks like Vision Video segmentation aial image analysis medical image segmentation computational pathology so this has huge impact in wherever you can use high resolution image so to quickly take a look at the architecture so if this is the input image the input image is split into patches so these are the patches you can see and then the patch patches are projected uh so then project them into patch tokens you can see the patch tokens here and that is what is happening here now if you want to do classification you have to somehow say that this image let's say it's a cancer image or this is image let's say a malignant tumor image or this is a benign tumor image so that information is required and that is what goes inside the class token here and that goes inside the Vim encoder the vision Mamba encoder and that is where it goes and the by uh directional part that they were talking about is this so uh if you know B type of models b eer t b type of models or by directional models which could be also used to predict text in the opposite direction or like most of the times people use it for fill in the blanks in the middle while most of the other models that we see these days are like unidirectional models which will go only one directional so this in this case image case this is a b directional model and different from from Mamba for Tech sequence modeling encoder processes the token both forward and backward Direction so this is the high level architecture um it's it's really good in fact if you compare it with WID you can see that the image net top 1% accuracy for WID is 78% and 76% for two different variants and here you can see the Vim woma one is 73% the second one is it is hitting 80% so with the similar amount of quality of the model or even for example the number of parameters is less here you are hitting almost similar or better accuracy while having lesser computation power and also you are able to scale it much more than you know I don't know how many of you know but lot of image processing used to happen at 128x 128 256x 256 even when stable diffusion started it was all 512 x 512 but what Vision Mamba shows us is that when you increase the resolution Transformers may not hold hold up really well but Vision Transformer or sorry V Vision Mamba whm model here would probably give you more efficiency that means less latency in this case and also the memory would be much lesser so it's I think at this point like it's like a no-brainer for anybody to use Vision Mamba for tasks like this especially when the image resolution is high but you can go ahead and then read the rest of the paper and if you're more technical than me of course you will understand more things on the paper now if you're questioning about whether they released the code and the models as promised on the paper it is not yet out so on Jan 18th uh they have released the paper and they're going to release the code and models very soon so if you want to follow you can start the repository and in one of the last videos one person asked me is this a sponsored video definitely this is not a sponsored video I'm just telling people to start the repository because it always gives a pleasure to the developers and uh this is Vision bomba very interesting promising uh Direction uh starting in January itself like I'm very excited to see what is going to happen at the end of the year so anyways we have got an efficient neural network architecture that can do really well for vision tasks as well see you in another video Happy prompting