Pixel-level Scene Understanding in One Token: Visual States Need What-is-Where Composition

1. 1Learn visual state representations from streaming video observations using self-supervised learning methods
2. 2Jointly encode semantic identity and spatial location of objects in the visual state
3. 3Capture what-is-where composition to enable effective decision making
4. 4Apply this approach to robotic agents operating in dynamic environments

Computer vision engineers and roboticists can benefit from this research as it improves sequential decision making in dynamic environments

💡 Visual states must capture both what an object is and where it is located to enable effective decision making

💡 What-is-where composition is key to effective visual state representations in robotic agents