Vaishaal Shankar - DataComp

Key Takeaways

[Music] hi everyone we are today we are excited to host Vishal for our computer vision seminar uh he'll be talking about data comp uh and currently he's a research he's a research scientist at Apple's ml research goou so I'll hand over to v um thank you uh yeah U yeah my name is ball and today I'm going to be talking about uh so a project we've been working on for the last year or so on trying to come up with a systematic way to construct data sets and uh and hopefully uh this is uh this is an engaging discussion and people can interrupt me if anything uh if they have any questions but uh let's get started um so so in machine learning computer vision over the last several decades data sets have sort of paved the way for better models um I'm not uh I'm not saying there's like a strict causal relationship here but like uh from mnist in the 90s when Mist was in around when Mist was invented was around the same time Yan laon uh started playing with convet and we got uh and we got very high performing confidence on mnist around the same time I think Lunette around the 90s and if you fast forward to 2010 Alex Kriski came out with Safar 10 and around the same time he came out with Cuda conet which uh was one of the earlier popular GPU training libraries for uh for deep learning and and I don't really h i i shouldn't really have to justify imag net which came out in 2012 and it was significantly bigger it was a million examples thousand classes uh and it led to it led to a flurry of research it led to resnets transfer learning self-supervision various different uh various different optimizers a lot of research was tested and evaluated on image net and and now when we go to the modern era in the 2020s we now have we went from a million images to hundreds of millions to billions of images and one of the one of the examples of this is open ai's web image text so open web image text is the data set that was used to train their original clip and Dolly models and so the the main difference here is that these image these models are not these data sets are not labeled with human labelers but they just have images and then the natural text that they come up come with so and the these large data sets are can be used for models such as clip which allows you to do zero shot prediction on a variety of tasks image generation so things like mid Journey stable diffusion all derive from these kind of data sets and so every time we get a new class of data sets we end up getting a new class of capabilities and now the the the Curious Thing is even though this is how this is the 10,000 foot view of the field the way we do machine learning research on a day-to-day is still the following uh way we have a fixed training training data set so when image net came out researchers would fix image net and then they would iterate on the model training procedure so they would iterate on the optimizer the architecture the augmentation etc etc and you and this works really well so like from Alex net to efficient net to noisy student uh vits they were all architecture changes and they were able to make uh drastic improvements in accur I think it was like around a 35% Improvement in top one accuracy just by just by iterating on the uh iterating on the around 30% Improvement just by iterating on the model training and then we have the fixed evaluation we evaluate an image net or Downstream tasks so any questions about this setup so this is sort of the standard ml research setup okay so one question that we ask here here is why not shift things around why not compete on the training data that is we iterate on the actual training data sets uh but we fix the model training so uh over the last 10 years we've we have a set of tools that we sort of trust we know like atom W Works uh Transformers uh with postn Norm uh uh p uh work so and we have a set of evaluation uh evaluation benchmarks so why not just keep those fixed and change the training data right okay so and the data set that we're going to use as a Target here is going to be the openai web image Text data set the uh and the reason we're going to use this is because uh clip is very well studied we sort of understand how to evaluate this in a in a quite rigorous way and web text isn't really public and U any questions about this point okay so uh let's let's do a short overview of a clip um this is the model we're going to train so we're going to keep this fixed but let's let's let's go over what clip is so clip is a data set of 400 million images collected from the web as I said before and what instead of instead of minimizing uh instead of doing standard supervised training the way what clip does is you take every image and every image has some text associated with it so in HTML Pages images have an ALT tag so this is this alt text is the text associated with every image and the idea is that you train a model to maximize the dot product between an image uh an images representation and the text representation for that alt text so so you maximize every image text uh uh you maximize a do product in every image text pair and you minimize images and texts that aren't related and then you train this on billions of images for uh on many gpus for many days and then once you do this you have a model that you can use to sort of do prediction on any sort of tasks that is we take our image we run it through the encoder and uh we take the set of the set of classes we would like to classify so say let's say a plane a car a dog and we just take the string a photo of a plane a photo of a car photo of a dog and we run it through this text encoder and we take the dot products and then we take the the image text pair with the highest dot product and that would be a prediction so this lets us do prediction on a variety of tasks without without training on that task so this allows us to have like a model that we can train once and evaluate on a wide wide array of tasks any questions about this setup and uh one thing that's really near and dear to my heart is that uh when you train models this way not only do you get strong models with high accuracy you also get models with very high uh robustness by robustness what I mean is that if you take a standard imag net model um resonet trained on imet it'll get some accuracy on image net but when you change the uh the the test distribution from image net to net V2 or image net to image net R the accuracy will drop a lot and what we found was that these clip models uh even if they have the same accuracy and image net are much more robust so sort of this extra training data that's that's used even though it doesn't boost the accuracy as much on imag net it boosts the robustness so instead of going from 76 to 64 you'll go from 76 to 70 instead of going from 76 to 37 an image net R you'll go from 76 to 89 so U these the this sort of this sort of weekly supervised training leads to high robustness so this is this is one very desirable property of these clip models and so uh this is something we would like to induce and um if you're interested in like uh a deep dive of why we why we're so sure that data determine data is what the thing that makes clip models robust we have this uh pre-print called we have we have this uh paper published in icml 2021 called Data determines distributional robustness in in clip and uh and in that paper what we do is we go through a series of hypotheses whether it's the fact that clip model is of language whether it's the fact that uh it's the training distri whether it's the training distribution training set size loss function test time promp prompting or model architecture and we systematically eluminate all the possibilities and we show that it's uh in fact the training distri distribution that gives clip all of these properties so it's not specifically the way it's the contrast of way it's trained it's just the fact that they have a really nice training distribution so this leads us to what what is the clip training distribution they didn't actually release it they released the model weights um and they released a bunch of valuation check uh benchmarks but they didn't actually uh release the the raw data set and if you look at the if you look at the the paper the original clip paper you see that there's only a paragraph about the data set and they say there's we got 400 million images from the internet and they give a very rough way of how they query the images so this makes reproducing these kind of models quite difficult uh but thankfully the open source Community uh is quite agile and quite creative and they were able to create a data set that is of somewhat similar quality by using a clip model so what they do is they take billions of image text paays from common craw and they simply filter them for English uh captions and they use an existing clip model and not what a clip model does is it takes an image image embedding and a text embedding and uh lets you let you get lets you have get compute a similarity score between the image and text in betting and they remove all pairs of image text that have a low similarity so in this case they used 0. 28 and they uh don't ask me where they got this threshold they got this threshold from in inspecting a notebook and so and they use a very weak clip model the when the when glos created only the open a b32 model was released um and e and the uh the nice part is that you get a pretty decent Mo uh data set but it is it is in fact worse than the original uh wi open AI data set so it's around two 3% worse on image net and 9% worse on object net and the other thing that's the other thing that's kind of problematic is that we they sort of did this on a Sho string budget so it's really hard to reprod we don't actually know the other design decisions this was sort of a one shot thing but it is cool that like the model that was used to do this filtering was an open ey b32 so that got that gets around 60% 62% on image net and you can use that you can use that model to filter common crawl and get something that's around 73% in theion so it's not purely distillation any questions about Lon any this is U okay so when we saw this uh our question was can we do better and can we sort of understand uh can we get the community sort of uh work together in building these data sets so that's where we introduce data comp so the idea behind data comp is that it's a multi-step it's a multi-step uh uh workflow where we allow the we allow we have a competition where we allow the participants to iterate on the exact data sets that are used to train these clip models so the way this works is it in four steps you we in order to allow for a wide array of participants we have a m a bunch of compute scales from extra small to uh extra large uh we have two ways you can participate whether you're going to filter or introduce more your data and then we we have our fixed training and evaluation Pine so the uh so let's talk about the compute scales so one one thing that we do recognize is that training these clip models is quite expensive training a clip model from scratch is around uh 50,000 GPU hours on an A1 100s uh for 38,000 and what we find is that that's uh if if that is the only way you can iterate that's going to limit the total amount of players in this uh in this game so what we do is we introduce a series of uh uh scales where you have a small scale which is you only try to filter 12.8 million images we have a medium scale where we have 128 million you try to filter 128 million images a large scale where you filter 1.28 billion images and an extra large scale that you've try to filter 12.8 billion images and so the small scale is just like fine-tuning a model in misset so this any academic lab should be able to do this and in fact the medium scale is just training is equivalent to the comput of training from scratch on emiss this is also something that an academic lab should be able to do large and extra large get slightly more expensive but the hope is that we can make progress on all of these scales and all you're trying to do is you're trying to come up with a data set that is as uh as good as possible for the sample budget and um this is uh and the way we set the sample budget is uh uh quite important so what we do is the sample budget is how many images you are allowed to see during training it's not the total amount of it's not your total data set size so by sample budget uh being 12.8 million for small what that means is that's the S you can either have a 1.28 million uh data set that you can do 10 epochs of or a 12m million data set you do one Epoch of and then similarly for the larger pools any questions about this so the idea is that each scale is a fixed train amount of training from people so the the main the the main uh one of the main contributions here is that we give we construct these large schools already and and what we do is we take common crawl which has around over a 100 billion image text pairs we run uh we remove uh pornographic material so we do not not say for work filtering we remove any images that are in any of our valuation sets we blur we detect and blur all the faces or we compute bounding boxes and during the download process the the faces get blurred um um oh uh question uh are there any insights into whether 10 EPO of 1.2 million or one Epoch of 12.8 uh million so that uh uh yeah so this is this is a good question so we had a question on whether uh whether 10ox of 1.2 million images is better than one Puck of 12.8 million and that's exactly the the point of the competition we don't know right because like the the the pools that we create are completely unfiltered right like we don't do any sort of quality filtering so the actual quality of the the data set could be really low so one Epoch of a low quality data set versus 10 epochs of a high quality data set could make a big difference so and the idea is that we as the uh as the competition creators we don't actually dictate what is qu high quality so the idea is that these uh these uh these images are just sort of a random sample of images in the internet and what the the participants do is figure out a good subset is um any questions so then what we do is we provide these we we provide these four scales and each uh each uh uh each pool is contained within the other one so the small is contained with the medium medium is contained with the large and large contained with the extra large so the extra large pool is we call it common pool 12.8 B so it's the largest image text Public Image Text data set available U so there's 12.8 billion images but these are uh the the these are decidedly lower quality than let's say even lion uh uh lion 2B U and we have some experiments to back this up but the idea is that now we can use this as a common testing ground to sort of filter uh filter and figure out what matters for our Downstream applications all right so um uh we also uh uh we also have another track where you can do the filtering but you can also so you can also bring your own data so if you think you have some data source that is very important that is not represented in common crw you can you're allowed to bring it and participate but there's a separate track so you're not competing with the same people that are doing just filter so there's sort of two parallel tracks one one where you do filtering of common craw another one where you can do filtering and add uh add uh your own data sources but you still have to abide by the total compute budget and then after this point it's relatively straightforward you we just you use open clip to train the Train the models uh we have we have tested that the the uh that given a good data set or training pipeline can produce state-of-the-art accuracies uh state-of-the-art models so you don't part you can't modify anything you can't modify batch size hyperparameters nothing so it's we really want to make sure that the the thing that leads to any sort of accuracy gains is the data set so and then in order to evaluate we have 38 different evaluation tasks U including image net and its variance U all of uh the visual task adaptation Benchmark and an open a valve Suite so everything that's in the original open AI paper and then various other distribution shift benchmarks such as Wilds then we have fairness benchmarks and retrieval benchmarks so the idea is that we have like this sort of very comprehensive Suite of evaluation tasks that give us some uh sense of how good the these models are one thing that we know is that we for Simplicity we don't actually do we we don't train on any of these training sets for these evaluation tasks we're just doing evaluation so there is going to be some bias on uh the uh the there's going to be bias because most of these tasks are sort of the answers are all in English so if you're trying to maximize this you're probably going to throw away uh data sets that have multilingual characteristics and this is something that we're working to remedy okay but uh yeah so the whole point is that it's zero shot zero effort and and we've in our paper we've we've compared to actually doing the linear probe just to make sure that these are perfect well correlated and we found that doing a linear probe evaluation is very similar to zero shot evaluation none of the none of the answers change at least for our benchmarks okay and then when you submit all you do is you submit a list of uids and your eval metrics so which subset that you which subset that you submitting so you're not submitting like a model you're just submitting us and then that's very easy for us to verify and if you're doing VD you have to submit the URLs that you're adding cool um and there's a public leader leader board on our website okay cool um so let's look at some of the uh submissions or like the baselines that we prod we provided in our paper uh to give a sense of how much signal there is in this task so one one uh so the uh the first Baseline is you just take our pool and you you run it you there's 12.8 billion images and you do one Epoch training on the entire pool and so this is these are the accuracies for the small medium large so you take the this the 12.8 million pool 128 million pool 1.28 million Po and you train one Epoch and these are the accuracies you get and you see at an extra large scale it's not that bad it's um still quite a bit worse like 3% worse than the uh open AI model so wit 400m is the open AI model but it's uh it's not terrible but uh it's 12.8 billion images so it's a lot of images and you're not U you're not getting uh you're still doing worse and 3% at this scale is still quite a bit of accuracy okay so uh one thing you could do is just do sort of basic filtering by basic filtering I mean just remove all the remove all small images any images where the caption is not in English and remove any images where the caption chooseable when you do this you do get a substantial accuracy boost on small medium and large we didn't run this Baseline on extra large uh and then uh and then we uh then we have some more interesting filtering methods so what we do is we we uh we do image and text Spas filtering that is we sort of just take the uh we take images that are close with content that are close to imet the imag net training images so note that we're not actually taking imet training images because we do duped our cool but we're just going to take images that are close to imag net training distribution and when we do this with we get a uh we get an improvement and we do the same thing for the pest so we get an improvement um but it's still we uh we were still not like very happy with the amount of improvement we got um and then so the uh one thing we did is we reproduce lion so when we reproduce lion exactly which is you do English filtering and then you do vitb 32 with clip score 0.28 we actually get a model that is better than lion um so it's like around 2% better than line which is somewhat surprising um and I think part of this is because we had a much cleaner strategy of doing the pipeline or and our the size of our pool our filtering pool was different than the way lion did it so that accounts for the around the 2% accuracy Improvement so even just reproducing existing methods with our clean uh uh pool gives us a 2% Improvement but again this isn't that interesting like maybe that's just one of the problems is just be a bug in the lion creation or it's uh uh more more likely it's because our outer pool our common pool was higher quality um so then what we do is we take Li take the lion style uh data set and we remove English filtering so we remove English filtering we remove basic filtering so there's no image size filtering no uh text based filtering no English filtering and the accuracy goes up again so the accuracy goes up another round percent so now all we're doing is just taking uh taking the clip the clip score of the open AI model from our data set and just taking the top 30% so no other filtering is going on now um what we can do is these are all all no all of these things are just set of indices right so when we once we do all of this we get a set of indices and we can intersect them and if we intersect the clip score which is hey the image and text should be somewhat uh uh should be somewhat related as at least based on the original openi model uh if we intersect that with hey my images should be somewhat related to the imag net distribution again we're not saying the image net images need to be exactly the imag net images it's just they can't be too far from the imet distribution so the way we do this is we take the image net uh the image net uh training set we cluster it and then we only accept an image from our pool if it's some distance in embedding space away from those cluster centers and we had around 100,000 cluster centers so so this now gives us a substantial Improvement at least at the large the medium large and extra large uh scales so uh what we find is that basically combining these two features image and text must be related and images must be close to imag net distribution um gives us a 3% uh or 2.8% Improvement on the Explorer scale which is quite substant uh substantial um uh uh to put in perspective in order to get that 3% Improvement you can do it with the lion data set but you would need to use a model that is 10x bigger so we can get that with the same small model by simply filtering our data set in this one way okay any questions about this um so one thing you could ask is okay u v you sort of kind of cheated here uh you looked at the imet distribution and you sort of bias your data set towards imet so is that isn't that just going to make all the other Val metrics go down the whole point of this is to build a general Vision representation and uh we're Happ note that that didn't happen so when we uh obviously the gains are less but even on the average across all 38 of our benchmarks doing this uh l14 clip uh and in image in image based intersection image net base intersection we get a 1.3% improvement with respect to at least the last thing um the l14 clip score and it's still it's still substantially better than lion um or the open an mod so around 6% better so in fact the the gaps are bigger U when we look at the average across all data sets than uh than just imet so and this is partially because we think imet we're what we're saying is we're not saying that we want images like imet we say imet is a good proxy for a good distribution of natural images and if you're really really far from that really really far from that it's probably bad and indeed the images that are uh excise by these filtering methods are things like images of there's millions of images of Sudoku puzzles in these data sets and those are not really useful and so those get excised okay so this is a 1.3 billion uh uh uh image 1.3 so this is this is our strongest performing Baseline so this is sort of the the the entry to beat to say and uh one thing we we we also tested was what about bring your own data the additional data sets and there are some uh there are Public Image Text data sets that are uh human verifi uh that have some higher quality image uh text Pairs and we combine them and when we combine them we see a slight Improvement on the average performance so uh it goes from 66.3 to 67 so they do help a little bit um but it doesn't really help imet so depending on what you you care you uh you care about you can you can uh count this as a win but uh the nice most of the gains are coming from the the filtering uh the filtering mechanism and one thing that uh one thing that we want to uh comment on is that the uh the the Zer these numbers these results are still valid even if you replace evaluation from a zero shot evaluation to a linear probe based evaluation and this is important because Zer shot evaluation has some caveats so this is something that we did just to make sure our results are valid even in a full fine tun in a in a more uh realistic case where you would linear probe your representations because we want to make sure that we're getting better representations better models in general from these data sets Okay so we call this l14 clip intersect image-based uh uh Baseline data comp 1B so this is our first like data set that comes out of this competition so it's a sort of state-of-the-art data set in some sense and it's just a combination of these two approaches so it's a combination of clip filtering which uh uh of just clip filtering and no language filtering and image clustering so there no other rule-based filtering there's no heris such as the images must be 256 pixels etc etc we just let the clip clip model do everything no and no calling to some external language detection we just let the clip model do everything and then we use the clustering to sort of uh remove images that the clip model would let through but doesn't really help because the images are are far from the manifold that we care about all right and uh and then so we did some other ablations to make sure that these results hold across other architectures so even if we changed from the original vit we changed to a confex we still see a roughly 3% Improvement we did this on the medium scale U and uh this is also true if we make the model smaller so this is important because what we did is we made the the models we made the best model behave by improving the data set we we improve the performance of the best model so much that it behaves like a larger model on the old data set and you can see that uh a uh and this is also true here in that like a the a a vitb a vitb 32 which is around twice as efficient as a vitb6 train on data comp uh is better than a uh vit B16 trained on trained on the original op data set right so we're uh not only are we improving the best model we're improving all models across the entire efficiency Market um and these like this is just this is sort of a framework for people to participate in and we're just getting started here so there's uh already a couple entries that seem quite promising so uh there's one entry that uh that shows that you can uh REM use the text removing the text bias in these clip models to improve the accuracies uh I think this was this is hit archive recently and there's another there's another uh work that's coming soon on training a custom data filtering Network U instead of using this open AI Network training some training a custom uh model just to filter the data and uh that also leads to some improvements and the whole point is that uh we as the data comp creators don't really don't want to be the only ones that are creating new data sets the just like the entire Community sort of collaborated and creating these better and better models of the last decade we want to see if we can do the same thing for data sets and uh yeah so that's like uh so the in summary data sets matter and uh uh data comp is one way you can you you can sort of push on that idea and it allows you to simultaneously improve on many architectures and many tasks and the current sub submissions show pretty strong potential and things I'm interested in to study in the future is whether uh what capabilities do we get from these better data sets so if we mine data sets for specific type if we mine for specific type of data sets like egocentric data or web or like uh outside uh uh satellite imagery what kind of better capabilities can we uh loock and we have a website called datac comp. which uh shows where you can download these data sets um and also participate and uh would love to hear what people uh would love to see what people can do um yeah and this is our website thank you thank you so much R um if you guys have any questions please free to type it in the Q&A thanks for inviting me um this is fun thank you um thank you so much for coming and giving this awesome talk thank you all right thank you bye