Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models | CVPR 2023

Key Takeaways

in this video I present our cvpr 2023 highlight work Odyssey open vocabulary panoptic segmentation with text image diffusion models this is the joint work of researchers from UCSD and Nvidia it will be presented in person during the Tuesday Morning session at poster number 281. going Beyond generative tasks Odyssey leverages the Frozen internal representation of larger scale text to image diffusion models for open vocabulary panoptic segmentation tasks it segments and recognizes any category Beyond The Limited training ones here are some qualitative example results from Odyssey it can perform panoptic segmentation of the input image and recognize unlimited number of categories in the real world the model is quite robust to the input images domain change and category distribution shift for example the Canadian flag countertop and faucet these categories are not in our training data set but the model is still able to Output detailed segmentation masks and recognize them at inference time in the past year text image generation using diffusion models trained on internet scale data has revolutionized the field of image synthesis it offers unprecedented image quality generalizability compositionability spatial relationship understanding and semantic control via the input text in this work we propose to leverage the diffusion unit of a text to image diffusion model for open vocabulary panoptic segmentation the proposed model is termed Odyssey let's first Define the task of open vocabulary panoptic segmentation given the input image in standard closed vocabulary panoptic segmentation the model would classify every vehicle as a truck however in the open vocabulary setting the model can classify them as a pickup truck or a trailer truck so it may understand data pickup truck needs a trailer truck to move another truck formally an open vocabulary panoptic segmentation we aim to segment and categorize any object even ones not seen during training there are countless categories in the real world and how to segment and classify them without explicitly labeling everything is quite challenging since text image diffusion model can generate images with many diverse Concepts we wonder whether it could also be used for recognition tasks of many Concepts as a proof of concept we visualize the results of clustering and diffusion model's internal features while not perfect the discovered groups are indeed semantically distinct and localized in contrast when we examine the internal features of the clip model which has been proposed for use in recent open vocabulary recognition works the Clusters are much less semantically and spatially distinct motivated by this finding we propose to exploit internet scale text image diffusion models for open vocabulary panoptic segmentation of any Concept in the wild here is the training pipeline of Odyssey given an input image we first add a small amount of noise to the image and then input it into the Frozen text image diffusion unit to extract its internal visual representation since the text image diffusion unit is conditioned on a text embedding as well we generate a text embedding for the input image via our proposed implicit captioner which consists of a Frozen image encoder and an MLP the text image diffusion unit is conditioned on the generated implicit text embedding to extract the visual representation in the experiments we show that the proposed implicit captioner outperforms using no caption at all or off-the-shelf pre-trained captioning networks then we input the extracted features into a mask generator The Mask generator is a meta architecture and can be instantiated as any model that can generate panoptic masks and mask embeddings here we use mask former to predict class agnostic binary panoptic masks and mask embeddings mask prediction is supervised by a binary Mass classification loss to predict the label of each mask we need to learn to classify each mask embedding here we explore two settings the first uses the category labels of each mask and encodes them into text embeddings with a frozen text encoder we apply a cross-century loss on the dot product of the text and visual embeddings the second setting uses weaker supervision where only the global image caption is available we extract and encode all the nouns and nouns phrase in the caption and use the grounding loss to learn the mask embedding it's worth noting that most of the components of Odyssey are frozen during training we only fine tune the MLP and The Mask generator with 28.1 million parameters that is only 1.8 percent of the full model to perform inference with Odyssey we input the image into the text to image diffusion unit and the implicit captioner to extract its internal representation The Mask generator predicts class agnostic binary masks and their mask embeddings based on the diffusion features we further perform Mass cooling on the image encoder features to extract another set of mask embeddings given the testing category labels we encode them with the text encoder to extract their text embeddings we perform a DOT product between the mask embeddings and the text embeddings then we take the geometric mean of two predictions to get the final open vocabulary panoptic segmentation results here we show some experimental results we train The Odyssey model with the Coco datasets mask and label or caption supervision and evaluate on the ad20k data set Odyssey surpasses prior Works Mass clip and open set by a large margin and achieves state-of-the-art performance on both open vocabulary panoptic and semantic segmentation tasks we also compare Odyssey with other state-of-the-art visual representations the diffusion features of Odyssey show significant advantage over the other representations in the ablation experiments we explore which tone step is optimal for Odyssey the larger the T value is the larger the noise Distortion added to the input images the smallest noise at T equals zero has the best results on all metrics concatenating three times steps yields similar accuracy but is three times slower here we compare the proposed implicit captioner with empty caption and off-the-shelf captioning bass lines for empty caption Baseline we input empty string into the text encoder and use the output text embedding for all the images we use heuristic captioner and blip as off-the-shelf captioning baselines since our implicit captioning module derives its caption from the clip model trained on internet scale data it is able to generalize best among all variants compared we provide qualitative results of Odyssey on the ego 4D Coco an ad20k datasets ego 4D contains many egocentric videos covering a wide range of scenes and objects although it has a large domain Gap to our training data set Odyssey can still segment and categorize each object correctly for example in the left figure faucet and in the right figure grocery bag are not present in the training data set and yet are recognized correctly by Odyssey similarly power shovel and lawnmower are also novel categories here is the visualization on the Coco and ad20k data sets here conveyor belt aquarium Chandelier and pool table are all novel categories to summarize firstly Odyssey takes the first step towards leveraging the Frozen representation of larger scale text to image diffusion models for Downstream visual recognition tasks secondly Odyssey demonstrates the great potential of text to image generative models that open vocabulary segmentation tasks our code is available publicly thank you for watching please visit us at poster number 281 in the Tuesday Morning session to learn more