How AI Taught Itself to See [DINOv3]

Key Takeaways

When we look at images, we can easily recognize objects, semantics, and understand the 3D geometry of the world. But how can a computer do this? In this video, we will explore how AI learn to see without label data. For a computer, an image is just a grid of numbers. To make sense of these numbers, AI needs to extract meaningful features from an image. Good features help us identify which regions of the image are similar to the red cross reviewing semantic structures even in a complex clutter scene. Let's look at a simple example to see why features are important. Here we have data points from two groups blue and orange. In this case, building a linear classifier to separate the two groups is very hard. No matter how we try, we can't find a good way to separate them. Now let's transform the data into radial coordinates. By changing our perspective on the data, separating the crew groups with a strain line becomes much easier. This example highlights the importance of having good feature representations. But how do we design an effective feature representation? One way is to learn features by training a neuron network to solve tasks like classifications. On top of the extracted features, we place an output projection head, typically a linear layer or a multi-layer perceptron that predicts class scores. We then use a softmax layer to turn the scores into probabilities for each class. The true label is a one half vector. We train the model to make its predicted probabilities match this label by minimizing cross entropy loss. This loss measures the difference between the predicted probability distributions from the softmax layer and the actual one hot encoded label. After training the model on a large label data set, we can use its learn weights as a great starting point for new tasks. This process is called transfer learning. It allows us to fine-tune the model for a wide range of applications such as fine grain classification, object detection, or semantic segmentations. However, class labels are often too simple. They only tell us the main object in the image like bear or cat, but they do not describe all the rich fine grand details inside the image. Natural language like captions or descriptions can give us much more information. For example, a caption might say, "A friendly bear waves hollow." Or, "A cat recines on the payment." This tells us about objects, their actions, and scene. To use this extra information, we add a text encoder. The text encoder turns the caption or description into a feature vector, just like the image encoder does for the image. Intuitively, we want the feature vectors for the image and the feature vector for its matching caption to be as similar as possible. This means the model learns that the image and its captions are related. At the same time, we want the image and text features for pairs that do not match to be far apart. How do we do that? Let's visualize the image and text features as 2D vectors. We normalize them so that they have unit lengths. We measure similarity by looking at the angle between the two vectors. Since the vectors are normalized, the dot product directly reflects how similar they are. The closer the dot product is to one, the smaller the angle and the more similar the features. This is what we want to see after training. Along the diagonal, the features from matching image text pairs are well aligned. off the diagonal the features from non-matching pairs become disissimilar. This is known as contrastive language image preing or clip. By training on 400 million image text pairs, the image encoder learns to capture and generalize a wide range of visual concepts. But this type of training is expensive because it requires human annotation for every image. Can we train our image encoder without any labels using only images? If we can achieve this, we can leverage the enormous amount of unlabelled images already available to train our models at scale. This is the goal of self-supervised learning. Just like supervised learning, we extract image features, add an output head, and use a loss function. The challenge here is to create a supervision signal for training but without relying on any human provided labels. Over the years, researchers have developed creative strategies for self-supervised learning. For example, we can train the model to colorize an image, predict the rotation angle of an image, impend the center patch from the surrounding contacts, or more generally, predict missing local patches from randomly mask input images. However, turning out this manually designed task does not reinforce learning representations that are semantically reach or transferable. Let's revisit the contrastive learning idea. We extract image features from two different augmented views of input images. The objective is to bring the embedding vectors from matching pairs closer while pushing embeddings of different images far apart. More specifically, we compute the pair-wise similarity of all pairs using vector product. Here SIJ denotes the similarity score between I embedding vector and the J's embedding vector. We highlight positive pairs with yellow boxes and negative pairs with blue boxes. During training, the model learns to increase the similarity for positive pairs. Intuitively, we train the model to extract feature representations that are stable under different augmented views. This is the core idea of sim clear. While contrastive learning has been successful, making it work is challenging. The main issue is that it needs a large number of negative samples to avoid learning trivial representations. As a result, training requires very large batch size and is computationally expensive. To tackle this challenge, let's look at how it could work with a single image. Given an input image, we create two augmented views like different crops. We call the image encoders as students and teacher models. We extract features, feed them into a projection head to obtain logics, and then apply a softmax function to produce probability distributions. With this setup, we try the student network to match the teacher's predictions by minimizing the cross entropy between them. A key detail is that we stop the gradient from flowing into the teacher network during training. In this setup, only the student network's parameters are updated to match the teacher's predictions while teachers provide a stable target. This approach is known as knowledge distillation. Typically, we use it to distill knowledge from a powerful large teacher model to a smaller student model, making inference more efficient. In our case, both students and teacher models have the same architectures and model size. But there are some problems. First, if we don't have labels from people, how do we know what a class is? Fortunately, we can define the output dimension to be a large number like 65,000. This enables the model to discover and represent a wide range of visual concepts without requiring us to define them in advance. Second, in knowledge distillation, the teacher model is usually already trend. But without label data, how do we get this teacher model? One effective method is to update the teacher model's weights as a moving average of the students weights over time. This allows the teacher model to change gradually, providing a stable and consistent target for the student to learn from. Specifically, after each training step, we update the teacher parameters using the following formula. The value lambda controls the update rate for the teacher model. A larger value means slower update. In practice, we set the initial value of lambda as 0.996 and gradually increase it over the course of training. This is great. However, this approach alone is not enough to prevent the model from collapsing to trivial solutions. For example, the model could end up producing outputs where a single dimension dominates the distributions resulting in uninformative or collapsed representations. To address this, we use a trick called centering. Let's illustrate how it works with an example. Suppose we have a batch of four samples and our model predicts probabilities over three classes for each sample. After applying softmax, most of the probability mask is concentrated in the first dimension. This is bad. Next, we calculate the average logic the center across the batch and subtract the center from each of the teachers logics. The centering tricks encourages the model to spread its predictions more evenly rather than focusing on a single output. Centering helps prevent collapse by ensuring the average output of the teacher network remain balanced. across all output dimensions. Instead of computing the center vector for each batch independently, we maintain a moving average of the center over time. That sounds great, but using the centering trick may lead to collapsing to uniform distribution. To address this, we sharpen the distributions by lowering the temperature parameters in the softmax function. Imagine we have logics 2.0, 0 1.5 and 0.1 for three classes. Using temperature 1, we have the probability distribution P. As we lower the temperature T, the probability distributions produced by the softmax become more peak, centering more mass on the largest logics. If we do the opposite and increase the temperature, the output probabilities become less sharp and the probability values are more evenly spread across all classes. In practice, we set a students temperature to be a fixed value of 0.1. For the teachers, we begin with a lower temperature and gradually raise it to a higher value using a linear warm-up schedule throughout training. Due to the temperature difference, the teacher's probability distributions is sharper than the students. This means the teacher's predictions are more confident with higher probability mass on the most likely class. This provides a strong signal for the student encouraging it to match the teacher's competent predictions. Instead of using only two views, we create multiple crops of the input image. We made two large crops that cover larger regions of the image called global views. We also make several smaller crops that focus on local regions. The teacher model only sees global views that contains more information about the image where the students use both global and local views as the input. By using this strategy, the model learns to associate detail information from small local patches with the broader context provided by the global views. As a result, the model becomes better at recognizing objects and patterns even when only small parts of images are visible. This method is known as self dissolation with sound label or dino. Here we visualize the local attention maps. Compared to model trend with supervised learnings, the attention maps from Dino are cleaner and focus more on the object. Here are more examples. In the second version, the researchers enhance the method by implementing a better centering techniques for the logics, introducing a regularization terms to encourage more diverse feature learning and expanding the output dimensions to 128,000s. They also scaled the training data set to 142 million images and increase the training batch to 3,000. One interesting improvement is adding patch level losses. But how does that work? Both the student and teacher models use a vision transformer as their image encoder. In other words, before feeding the image into our model, we first split the image into smaller patches. We prepare a special learnable class token to the sequence of image patches. The class tokens features encodes a summary of the whole image capturing its global context. This is the feature we have been working with so far during the training of Dino. Now we randomly mask out some input patches for the student but not for the teacher. We then introduce a patch level loss. This encouraged the students model's predictions for the mass tokens to align with the teacher's model's predictions for the corresponding visible tokens. It helps the student model to learn fine grain patch level features. Both image level and patch level losses use MLP projection hat with the same design, but each has its own separate parameters. This aberation study demonstrate how each technique incrementally contributes to improving overall performance. A few weeks ago, the same thing released Dinov3. Aside from further scaling up model size, data, and compute, Dino V3 has an interesting trick to improve the quality of dense visual features. As trending goes on, we can see how similar the red patch is to others. Over time, the features become less focused and the similarity maps get noisier. This makes the model worse at dense prediction tasks. Dinob3 used a previous version of the teacher model to recognize the correlations between all patch pairs. More specifically, we compute a cosine similarity between all pairs of patch token embeddings. This is called a gram matrix. We add a loss that encourages students models gram matrix to closely match that of the teacher model. This trick is called gram anchoring. These regularizations preserve the spatial relationship between local patches while still enabling the students features to be refined during training. In practice, we use a teacher model checkpoint at 200,000 steps. The teacher models process higher resolution images to extract more detail features. Afterward, the resulting gram matrix is resized to match the dimensions of the student models matrix. When we compare the self similarity maps before and after applying gram anchoring, we observe the spatial structures become much cleaner and more defined. Compared with the dense features from Dinov2, we see that the feature maps from Dinov3 are sharper, contain less noise, and show better semantic coherence. Self-supervised pre-training methods like Dino make it much easier to adapt models to new tasks. Often requiring only a small adapter and a limited amount of label data. General purpose visual representations open the door to wide range of new possibilities and applications. This is an exciting time for computer vision. Thanks for watching and I'll see you next