Contrastive Learning for Tabular Data - SCARF

Key Takeaways

this video will provide a quick overview of the scarf algorithm bringing self-supervised contrastive learning to tabular data using random feature corruption so this is using frameworks like bootstrap your own layton where you have these two positive pairs one formed with the data augmentation view of the original example and bringing that kind of idea to tabular data so this is tabular data say it's information about a customer and a business things like name age maybe buying preferences and these different features that describe customer data these kind of type of data sets and we'll look through this catalog this openml cc18 is a data set of uh as a collection of tabular datasets if you're interested in just picking one of these to do academic research or getting a better sense of what these data sets look like compared to the usual deep learning data sets of vision language audio and so on even though the type of data are more common in the machine learning real world anyways but so this is a strategy to bring this kind of advancement of self-supervised pre-training contrastive learning into tabular data this quote communicates what i think is the key id of the algorithm which is how we're going to mask out and corrupt the data augmentation to form the positive pair to align the representations of something like a momentum encoding average like the bootstrap your own latent framework so describe we sample we sample some refraction of the features uniformly at random and replace each of those features by a random draw and then particularly this line from that features empirical marginal distribution so each of these features say it's the age of the customers or how many toilet paper rolls they bought if this is like a shopping basket analysis kind of problem you would have that empirical district marginal distribution marginal distribution of each of the random variables uh say you have x1 x2 x3 so and then they each have their own separate uh distributions as well as the joint distribution and so on so you're going to sample how you're going to mask that and corrupt the example from the marginal distribution of each of the features so if it's age and so on it would have a different distribution of how you sample a new age value and different densities like just say it's a different gaussian distribution with different mean variance parameters or whether it's some poisson distribution total other distribution but so that's how you're going to be sampling which features you're going to be using corrupting in order to form the data augmentation view which is the second positive pairing as you align these v v prime representations in the contrastive learning framework this is the openml cc18 collection of these tabular data sets the authors are going to test this scarf pre-training algorithm with we have all these different domains of tabular data we see the number of instances the number of features in each of the tabular records and the number of class labels to classify them in for supervised evaluation and then supervised fine tuning so each of these domains with each of the features has its own empirical marginal distribution which is how you sample the corruption for forming the data augmentation view for doing this self-supervised pre-training representation learning that facilitates with supervised learning for mapping these data sets to the this number of classes in the in this collection of data so here's some high level takeaways from the study they find the scarf pre-training improves predictive performance it improves performance in the presence of label noise and controlled experiments improves performance when labeled data is limited also in controlled experiments and then the other corruption strategies like not sampling from this empirical marginal distribution are less effective and more sensitive to feature scaling scarf isn't sensitive to batch size and is fairly insensitive to corruption rate and temperature tweaks and then the tweaks of the corruption don't work any better and then the alternatives to the info noise contrast of estimation contrastive loss don't really work any better than the tested algorithm thanks for watching this quick overview of the scarf algorithm bringing latest advances in contrast of self-supervised learning to tabular data please stay tuned for the rest of the ai weekly update series on henry ai labs [Music]