Grounding Spatio-temporal Language with Transformers | JRC Workshop 2021

Key Takeaways

hi and i'm very happy to be here at this workshop and welcome to the presentation of grounding spatial time for language with transformers so uh this is joint work with christian carch clement va katya hoffmann from microsoft and playbook let's begin so first we will introduce the work by saying that it is a part of grounded language learning and grounded language learning is a subfield of ai and it is about learning language in a way that is connected with the physical world so traditionally natural language processing is concerned with linguistic input only but in grounded language learning it is important to try to connect this language with the physical world so in recent years one possible option to consider this this field or to study grounded language learning has been language condition deeper enforcement learning where an agent or reinforcement learning agent is provided not not only with the state of the world but also with for instance a language instruction and has to perform what the instruction specifies and the rewards it receives depends on the instruction so this is a special case of gold conditioned reinforcement learning which has also received some interest in the recent year so previous work in this field has shown the importance of not only learning a policy for the agent so a function that learns to translate the states and the language into actions but also of trying to model the reward function to learn to say if a given sentence is true or false in a given of a given state of the of the world so the settings um so for instance here you can see on the left side one of the classical works in this field which is called baby ai and you can see here the environment which consists of a grid world with several objects scattered around with their colors and the shapes and you can see here the agent which is represented as a red arrow so the agent is provided with this synthetic language instruction which is put the blue key next to the green ball and has to execute this instruction so has to behave not only based on its state and on its observations but also on the language it receives and these settings usually only consider instantaneous language and descriptions and the architectures the deep learning architecture used in this agent reflect this and in this work we are doing now we want to be able to extend these settings uh to consider sentences uh with the temporal aspect such as the animal that was next to the rock jumped so here the the temporal aspects comes because jumping is an action that is performed over time and to decide if something jumped if an animal jumped you have to look at its evolution over time and notice also that we refer to the animal to a particular object with its spatial relationship to another object in the past and this relationship may no longer be true in the present so we cast this ground language grounding problem in our case not as a reinforcement learning problem but us more narrowly as learning a reward function r of theta so as a deep neural network and you can see this reward function also as a truth function that has to learn to tell if a sentence a given sentence w l is true or false given an observation s of i and t here the wl the l index index is the different words of the sentence and each observation is a an object vector which contains information about the shapes the sizes etc of of objects in the environment the i index here indexes the different objects that exist in the environment and the t here indexes the different time steps so we have access to the the whole traces of the objects over time as an input observation for our reward function and then we evaluate the ability of our models to generalize to unseen sentences uh this is a kind of systematic or linguistic generalization and to do this we monitor the f1 score on sets of test sentences on our set of observations so rapidly to describe our environment and the spatial temporal language we use in our setting so you can here see here a visual depiction let's say of our of our setting which contains a collection of objects which contains an agent which is represented by its hand over here and the agent can interact with the different objects by grasping them moving them around et cetera the all the objects can also interact with each other for instance if an animal passes over a source of food or supply it will grow in size and advance also we monitor the temporal evolution of the states of the objects over a whole episode and at the end of an episode language descriptions of everything that happened during one episode is generated at the end so the language we use is very is very synthetic and it's generated according to a grammar that allows us to um generate only true sentences for the the states that we have the temporal traces of the states we have in our environment so to describe the language the synthetic language in a bit more detail we have a set of basic primitives for our language which is composed of different objects uh for example cat but we have 30 something objects 32 objects and also some categories such as living thing furniture etc we have also some attributes for our objects blue green red which corresponds to their color and we have also sets of predicates which importantly um allow us to talk about the actions of the agents over the objects in the environment so these are the basic language and they do not uh entail any spatial or temporal concepts in the language then we have also the spatial aspect in our language so we have decided to restrict ourselves to referring to objects as their relations specified by the relations in space to other objects so we have one-to-one relations uh we can refer to an object to a thing for example uh by the fact that is that it is left of another thing and similarly we can refer to an object by its relationship in space to all the other object by instance saying that it is the leftmost object and finally and this is important uh the tempora the temporal aspect in our language is represented by three things so firstly the temporal predicates that we consider such as shaking and growing can only be decided by looking at the temporal evolution of the object over time and the the the second aspect of the temporality we consider now language is putting a predicate to the past so this is realized in the language by adding the was token before the verb to indicate that the given action had been has been taking place in the past and we also use past spatial reference to indicate that a given spatial reference was true in the past and is no longer true in the present so here for instance uh there is an example that has past spatial reference because the chameleon here was bottom of the television it moved on the top and right now in the present the agent is grasping it so grasping in the present was button on television because the chameleon was bottom of television at first so here there is another illustration illustrating uh temporal predicates and also the past tense for this potential predicate so first the agent grasps the tv and shakes it so this happens over a time interval and this is the temporal fatigue aspect after a few time steps the agent releases the tv and moves away so the action of shaking is considered in the past and is no longer true in the presence hence one of the destruction descriptions that is generated at the end of the episode is was shake red television so from this set of episodes we create some samples of observation sentence tuples as of by twl and we process them with some architecture based on transformers which have enjoyed really good performance in natural language and also other tasks in the recent years we train them with supervised learning to predict if a given sentence is true or false with the given temporal trace of objects that we consider and we implement different intuitive inductive biases in this architecture to study the effects of different kinds of aggregation on this task so the first model i'm going to present shortly is the unstructured transformer where all the different time steps for all the different objects in this dimension here are simply flattened in a single single list in a single set of objects the language description is concatenated and the query and learned query token is also concatenated at the end several rounds of self-attention as is classical transformers is performed on this set of objects and a final reward token is used to make the the prediction at the end so you can see this as a kind of reduction operation over this this whole dimension which has been flattened here then we also consider the spatial transformer which performs a kind of hierarchical attention so it is a sequence of two different transformers the first one is applied over the different time steps here so at each time step the language is concatenated to the different objects here and also the puree token is repeated at each time step here so this produces after the rounds of self-retention this produces a final vector which is of the same length as the number of time steps in our episode and this is queried again so reduced again with another transformer with different parameters to produce the final prediction token r here which is used for predicting if the label is true or false and finally we use also something we term a temporal transformer which is the transposed architecture as you can guess from the picture here where the words of the language are concatenated to the different objects within the time dimension this is first reduced along the time dimension with several rounds of self-attention so the objects have the opportunity to attend to their temporal revolution this produces a vector which has the same length as the number of objects then this is queried and reduced again to produce the final the reward token r which is used for prediction we also measure the influence of letting the words interact directly with the observations or aggregating them as we do for the spatial temporal dimensions for our different uh architectures before uh and so this is to study the effect of letting the word tokens interact directly with the objects versus summarizing them in a in a single token that uh incorporates the whole language information here so we divide our test sets which are composed of new unseen sentences according to the kinds of meanings that they consider so first we have the base sentences which have no spatial or temporal um meanings or concepts in them the spatial ones which include the spatial relations uh the temporal ones which includes uh the past the past predicates and also time interval or temporal predicates such as shake and grow and the spatial temporal ones which is spatial and temporal and so this is still ongoing work but we have some preliminary results here which i'll go over kind of quickly we can see simply that the unstructured and temporal transformers are the best uh within a small margin of error so this is only three random seeds for each of our architecture here is the f1 score over there the whole our four splits and they performs uh they have very good performance and similar performance on these three splits so we can see that on the spatial temporal um splits the word aggregation which are those two models here seem to perform a bit less and very interestingly the spatial transformers which are which are the first two ones here perform very badly and close to baseline here uh on this uh on this task so this is quite interesting so to conclude we have presented the first step towards grounded language learning of spatial temporal concepts we have presented a family of transformer architecture on this task to study the effect of different inductive biases on grounding artificial temporal concepts and we have identified that aggregating along the time dimension seems to have no measurable impacts compared to having an unstructured transformer but the spatial transformer performs quite quite badly and this is suggesting that there is an importance of maintaining object identity in this task to be able to ground the language correction so thanks for listening and