Audiovisual Self-Supervised Learning

Key Takeaways

this video will quickly explain some ideas presented in a facebook ai blog post on audio visual self-supervised learning including a paper that is a best paper candidate award for the cvpr 2021 conference on contrastive learning with visual and audio data so we start off with this idea of contrasting audio and video together we have these uh audio files that pair with the video data and things like ucf 101 action recognition where say you have a video of a person playing tennis and you have the sound of when you hit the tennis ball or baseball or these other kind of actions that are paired with the audio so they motivate this idea of contrasting audio and video together you wouldn't expect a tiny dog to suddenly roar like a lion these kinds of ideas of pairing audio information with visual information in this video data stream so the idea that they're going to propose are these techniques to separate audio and visual models that do this late fusion technique for a shared representation space so say you have a one-dimensional convolution model that process the audio data and then you have something like the time s former or maybe just an image frame encoder that processes the video data say they each go through separately 20 layers of deep neural network processing and then once you have that final output representation vector those two representations are going to be contrasted with the contrast of learning objective and they explore and detail two different algorithms for structuring this contrastive learning between the audio and visual features so the first of which is avid where you use these usual contrastive learning frameworks with the positive pairs sampled from the same video clip and this achieves state of the art on the ucf 101 hmdb 51 datasets and it's the best paper candidate at the cvpr 2021 conference this is the video provided in the blog post to showcase this idea that the visual attention is looking for where sound may be produced so they also motivate this with this idea of if there's a fire truck passing by in the video with the alarm going you would focus the visual attention on the fire truck as it passes in this kind of idea this is kind of a funny example because i'm not sure if the person dancing really makes any uh noise but this is the kind of idea of that you probe these models by looking if the visual attention is focusing on things that should make noise like if you slam symbols together or something like that is the visual model looking at where is going to cause the noise for this kind of representation alignment between visual features and then the corresponding audio so in addition to avid the blog post presents the extension to the research which is the rxid model so this is an extension to the contrasted learning framework so similar to supervised contrastive learning in uh which has been tested in computer vision natural language processing where you extend the positive alignment to all images or text that share the same class label so you're using more positives now so usually you have two positive pairs and then a large batch of negatives is the usual setup for contrastive learning they're extending it to have more positive so additional related videos and audio so say it's more tennis videos or more dog barking videos these kinds of ideas you extend it so it has more positives for aligning the video and audio and this may be useful because as they further discuss this idea of faulty positives you could have these videos where the audio really there really isn't much of a signal and as they motivate that you could have say someone doing yoga where there's really no audio that would help with to understand that uh that video audio signal for the contrastive learning objective the blog post also describes a strategy to mine these similar negatives and positive pairs online to structure this loss without having to do some expensive uh offline computation where you say pause the training and now form this index by computing all the representations and then do the nearest neighbor lookup rather they have the strategy of training an instance discriminator to facilitate the retrieval of the nearby neighbors so putting this all together these are results the extension from the rxid over avid and the improvements on ucf and the five shot learning performance we only have five demonstrations of each class category like playing tennis kicking a soccer ball hitting a baseball these kind of ucf 101 action recognition tasks and how much it's benefiting from this additional audio training so thanks for watching this quick overview of the audiovisual self-surprise learning blog post from facebook ai thanks for watching and please stay tuned for the rest of the ai weekly update series [Music]