OpenAI CLIP model explained

Key Takeaways

hi everyone in this video we are going to cover the clip model proposed by open AI contrastive language image pre-training the main idea of clip is based on natural language supervision so they use self-supervised tree training based on the natural language as the learning signal they pre-train their model on a very large data set of image text pairs that they collected from the public domain clip model has several use cases one of which is zero shot transfer so in this video we will see more details on how clip is trained and evaluated first let's discuss the motivation for the natural language Supervision in the context of image representation learning so far most computer vision models are trained in a supervised manner but obtaining the labels and annotations for supervised training is usually expensive and therefore there are limited amounts of data available for supervisor learning on the other hand there is unlimited amount of raw text that can be used for natural language supervision so to summarize the advantages of natural language supervision first is that it's easier to scale compared to the supervised training and second with natural language supervision we automatically connect image representations and natural language representations so this enables several potential application ations for multimodal image and Text data for pre-training the clip model they collected a data set of 400 million pairs of images and text and they called this data set web image text or W it for building such data set they gathered 500,000 queries based on all the words in the English Wikipedia that were repeated at least 100 times as a result of this they got uh image text pairs like this example shown below also in order to balance the data set they included up to 20,000 pairs per query natural language supervision for image representation learning is not a new idea clip is actually inspired by these two models Vex and convert that both share the same motivation since obtaining labeled data in a specialized domains such as medical data is difficult and costly so instead they want to pre-train using natural language supervision with unlimited textual data so first let's see these models briefly the vertex model is composed of a visual backbone and a textual head as shown in this figure the visual backbone extracts features from the input image and the textual head receives the features from the visual backbone and predicts the caption word by word after pre training the vtex model this model can be transferred to Downstream tasks however the major drawback of this approach is that it is difficult to predict the exact caption since there are various possible captions that can go with an input image so as you will see convert and clip models they both opted for a different approach the convert model is a specifically proposed for medical images where getting the supervised labels for images requires domain experts and therefore it would be very costly two example images and their description is shown on the left and a sentence that is randomly sampled from the corresponding medical document of each image is shown below them so convert tries to learn image representations using natural language supervision the first difference between convert and Vex is that here the visual and the textual networks work in parallel in contrast to Vex for training this model they use contrastive loss that tries to increase the cosine similarity of representations of images and texts from the correct pairs now moving back to the clip model we saw two different ways of natural language supervision Vex tries to predict the exact caption while convert uses a contrastive loss that tries to maximize the similarity of corresponding represent ations as we said earlier predicting the exact caption is difficult therefore clip follows the approach of convert instead so given a batch of image text pairs clip's objective is to find the correct pairings between images and texts in the batch in other words which text matches with which image with higher probability assume we have a batch of image text pairs so that means we we have images i1 to i n and text T1 to TN on the left it shows the correct pairings in the batch so each image is correctly paired with the corresponding text but there are also n^2 minus n incorrect pairs as shown in the right so the goal here is to make the representations for the correct pairs to be similar and the representations for the incorrect pairs to be dissimilar here is the diagram and the pseudo code for training clip the visual and textual encoders work in parallel similar to the convert model feeding a batch of paired images and texts the visual encoder outputs representations i1 to I in and the textual encoder gives T1 to TN we normalize these representations with L2 and perform dot product with between the normalized image and text representations and as a result we get a matrix of similarities as shown here note that the two labels is on the diagonal of this Matrix so we can construct an array of labels for correct pairings simply with this np. AR range function then we compute the cross entropy loss on this Matrix once for the dimension zero to get the loss for the visual ENC and then on Dimension One of this Matrix for the loss of textual encoder and finally we take average of these two losses now let's look at the architectural details of clip for the image encoder clip explored two families of models the reset based models and viit based models they experimented with five different reset models as shown here and they applied some improvements by replacing the global average pooling with the Transformer style attention pooling for the vi models they experimented with three models two viit B models and a viit l in addition they also added an extra version of The viit L that is pre-trained at higher resolution 336 pixels and in fact this model has achieved best performance for the text encoder they use Transformer model with architectural details as in gpt2 for tokenization they use lowercase bpe and a vocabulary of size 49,000 for efficiency reasons the maxed sequence length is capped to 76 tokens and as output of the text encoder they extracted the features associated with the token EOS for end of sequence for the final text representations now moving to the experiments to Showcase some of the capabilities of clip there are three sets of experiments conducted by the authors zero shot transfer where they investigated the capability of clip for classification on completely unseen data sets representation learnings with transfer learning and finally evaluating the robustness of clip to Natural distribution shift for the zero shot transfer they set up the experiment as follows so imagine we are given an unseen data set which includes a list of images and their class names and we want to use clip for classification in a zero shot setting so we first extract the set of all possible class names and reuse the image text pairing capability of clip to map each image to its correct text which is in fact its class name while the initial experiments using only class names for pairing showed promising but the authors also used prompt engineering techniques to further improve the results so instead of Simply using the class names for the text we can wrap them into a more descriptive text phrase like these examples that can be tuned for each data set for example on a data set of fruits we can build text phes like this a photo of Apple which is a type of fruit or or a photo of orange which is a type of fruit and try to match each image with one of these possible text faces another trick is to use ensembling with different text prompts like building multiple prompts to describe the attributes of each object like its size or color and finally select the one with the highest average score here is the results of the zero shot transfer applied to several data sets and compared to a fully supervised model serving as the Baseline that was built by adding a logistic regression on top of the resonet features as you can see the green bars here indicate that clip is outperforming the supervised Baseline model in several data sets and in some cases like the top three data sets by more than 20% however we should also note that for some specialized domain data sets such as the satellite imagery data sets or fine grained Medical Data the performance of clip is lower than the Baseline the next experiment is representation learning for which they decided to fit a simple linear classifier on top of Clips features this figure compares the performance of different clip models shown with the star with other existing models based on the computational cost on the x-axis and average score on on the y- axis and as you can see the best clip model vitl which is pre-trained at higher resolution 336 pixel outperforms all existing models and the last experiment is on the robustness of clip features under natural distribution shift this figure shows some example images from the class of bananas in imet as well as five other data sets such as imet V2 object net and image net sketch the performance of zero shot clip is compared with the res net model that is trained on image net and as you can see the zero shot clip model outperforms res net by up to 75% this result showcases that zero shot transfer models such as clip are more robust to such natural distribution shifts so that's all for the video on clip for contrastive language image pre-training thanks again for watching and stay tuned for more videos