Glenn Jocher: Image Augmentation in YOLO v5 and Beyond

Key Takeaways

[Music] i'd love if you could touch a bit upon the augmentations that you're taking advantage of and how those are improving model performance i started off essentially trying to throw everything at the augmentation pipeline i had rotations i had shear i had gain scaling and then of course color space scaling so we had hsb scaling different ratios and lastly i use this mosaic data loader which would take four images randomly place them and then apply all these augmentations at the same time so augmentation is a tricky thing i actually discovered that some of the augmentations that i'd applied even in small amounts were hurting the results on the coco dataset and a lot of the time you don't get the satisfaction of really understanding why you get a certain result in ml so it takes a lot of experimenting a lot of trial and error so i've actually toned back the augmentation techniques now and in the latest repository in yolo v5 we have a high degree of scaling augmentation and we use a mosaic data loader and we do color space augmentation so the augmentations are done a little bit differently in v5 than in the v3 repo that i had and v3 literally upscale the entire image and pass that in as a batch during training so one of the downsides of this is that it uses up a lot of memory if you want to scale up your images very large so in v5 rather than scaling the actual image we keep the image the same size so for example the default training size is 640 pixels so we input 640 images for every single batch and the augmentation the scaling is done within a feed transform that'll increase the size of the image or decrease it but it always crops it at the same 640. um it'll randomly translate it also i forgot that but this is actually it's a bit of an improvement because it means that every image in the batch can be scaled differently from every other image in the batch which increases variation and it also means that you don't need extra memory to train very large images so when you do the very large image you're leaving some training performance on the table when you do smaller images so if you have a batch that decides to fit into memory that's this big and then one of your batches is smaller like this you have a whole lot of gpu memory that's not being used for that smaller batch so this exploits the gpu memory much better one interesting nugget in that is it sounds like you ran a seater's experiments to identify which augmentations would generalize effectively to any domain what work do you think is left to be done in ml in augmentation optimization strategies because we see in our own tests that augmentation is dependent upon the domain so it's really tough to like always use this checklist of augmentations right mosaic generally increases performance i don't know that we find a founding place where it decreases performance if you're building mobile apps rotation sometimes is really useful because the orientation of the phone to the device to the object is going to be quite varied i'm curious to hear if augmentation is something that you believe might be contextual to the problem but you're including it in the training pipeline how do you think the augmentation portion of training will evolve well this is an excellent question and i don't think that there's any easy answers like you said there's some domains like satellite imagery where rotation is incredibly important uh and then there's others for example like a lot of the cocoa dataset for example where like a lot of the images are the way you usually take them with a cell phone and they're just not rotated that much in the short term the medium term and probably the long term i think there's always going to have to be a human hand guiding the augmentation strategy at least a certain extent of course there's fully automated methods like auto augments yeah which efficient debt also uses for some of their large models but again this is the bridge too far for i think most organizations to spend the money in the gpu time these are the two answers essentially there's a brute force method if you can manage it then more power to you but if you can't then uh i don't know there's there's not quite any easy answers today other than relying on your own experience and the experience of those around you yeah that makes a lot of sense [Music]