DAViD: Data-efficient and Accurate Vision Models from Synthetic Data

Key Takeaways

In this work, we present highly accurate, cost-effective, and robust human- ccentric vision models trained solely on highfidelity synthetic data. Our approach provides accurate, robust human-centric dense prediction models trained on small yet highfidelity synthetic data with low training and inference costs. Particularly we show that we can achieve highly accurate results on human ccentric dense prediction tasks of soft foreground segmentation depth estimation and surface normal prediction. In the literature there are different approaches to train models that generalize to diverse scenarios. Big transfer leverages a massive data set of 300 million images with noisy labels for pre-training, providing a strong backbone for various classification tasks. For generalizable depth estimation, depth anything v2 leverages the giant version of Dino V2. First training on 600,000 synthetic images with perfect depth annotations, then generating pseudo ground truth for 62 million diverse images to train student models. Depth Pro in contrast first trains a custom DPT model on a mix of real and synthetic data sets to enhance fine details. It continues training on highresolution synthetic data with perfect annotations. In another work, Sapiens focuses on human centric vision tasks and pre-trains its backbone on 300 million real images of people using self-supervised learning. The pre-trained model then initializes weights for various downstream tasks such as depth and normal prediction fine-tuned with synthetic data. While all these approaches show promising accuracy and generalization, they typically require very large models and expensive training. In this work, we take a simpler approach of training on a diverse highfidelity synthetic data set that generalizes to real humanentric scenarios. We show that training on a small yet highfidelity data set is enough to achieve very competitive accuracy. Our models require only a fraction of the cost of training and inference when compared with large foundational models of similar accuracy. To train our models, we exclusively use synthetic data generated through an existing pipeline that creates realistic human-centric data sets along with precise ground truth annotations. In this work, we extend the use of this highquality synthetic data to dense prediction tasks where both realism and annotation quality are more critical and for which annotations on real data are often impossible. We train all three models for depth estimation, surface normal estimation, and soft foreground segmentation on a single training data set of 300,000 images. The images cover faces, upper body, and full body scenarios equally. We designed this data set such that it is diverse in terms of poses, environments, lighting, and appearances and not tailored to any specific evaluation set. Our models are built on top of dense prediction transformer or DPT. We first resize the input image and pass it to a VIT encoder for a fixed time encoder inference. Similar to a unit, we use a convolutional decoder with four blocks in which each block is connected to the encoder through skip connections. To enable inference on any input image resolution, we use a simple convolutional model called resizer. It injects image features into decoder blocks, allowing the decoder to reason about image features at input resolution. We use the same architecture for soft foreground segmentation, depth estimation, and surface normal prediction. Since we use the same architecture design and same training data set across all three tasks, it is feasible to extend the model into a multitask one by adding decoder heads. This is computationally advantageous over running three separate models when all three output modalities are needed. Our human centric dense prediction models deliver high quality results that reliably capture a wide range of human characteristics, handle diverse lighting conditions, and preserve fine details like hair strands and subtle facial features. This demonstrates our model's robustness and ability to handle complex real world scenarios with high accuracy. Here we compare our multitask dense prediction results with the existing soda models. Sapiens 2B. We show that not only our model produces highly accurate predictions with fine details, it runs orders of magnitude faster than competing methods. Our relative depth estimation model generates a 2.5D representation from a single image, enabling 3D point cloud creation and novel viewpoint rendering while preserving complex depth relationships and object proportions. Our soft foreground segmentation model enables precise subject extraction, preserving fine details for highquality background replacement in applications like video conferencing. Another application of our model is relighting from surface normals. By predicting accurate surface normal maps, we can rerender images under novel lighting conditions, creating shading and ambient effects that lead to a visually plausible relighting result.