Smart Navigation - How AI Robots Understand and Explore Environments

Key Takeaways

facebook researchers developed a language-guided navigation task where the agent follows language navigation directions given by a user in order to realistically move in the environment they even made a public release version of the code available for everyone on github let's see how they achieve that and some amazing results [Music] this is what's ai and i share artificial intelligence news every week if you are new to the channel and want to stay up to date please consider subscribing to not miss any further news here's another paper presented in the eccv 2020 it is a language guided navigation in 3d environments from the facebook ai research team as the name says they developed a language-guided navigation task for 3d environments where the agent follows language navigation directions given by a user in order to realistically move in the environment in short the agent is given a first-person vision which they called egocentric and a human-generated instruction such as leave the bedroom and enter the kitchen walk forward and take a left at the couch stop in front of the window then using this input the agent must take a series of simple control actions like move forward for 0.25 meters turn left for 15 degrees to navigate to the goal using such simple actions via ln ce lifts assumptions of the original vln task and aims to bring simulated agents closer to reality just to give a comparison current state-of-the-art approaches move between panoramas and cover an average 2.25 matters including avoiding obstacles for a single action they developed two different models in order to achieve such tasks the first one a is a simple sequence to sequence baseline the second one b is a more powerful cross model attention model which we can both see in this picture the first model takes a visual representation of the observation containing depth and rgb features and instructions for each time step then using this information and the instructions given by the user it predicts a series of actions to take denoted as 80 in this image the rgb frames and depth are respectively encoded using two resnet 50 architectures one pre-trained on imagenet and the other one trained to perform point goal navigation then it uses an lstm to encode the instructions from the user astm is the short for long short term memory which is a recurrent neural networks architecture widely used in natural language processing applications due to its memory capabilities allowing it to use previous words information as well these actions a are then fed into the second model the goal of the second model is to compensate for the lack of visual reasoning in the first model which is super important for this kind of navigation application for example you need a good spatial visual reasoning in order to understand an instruction such as to the left of the table your agent needs to know that it first needs to know where's the table and then go to the left of that table which is done using attention attention is based on the common sensical intuition that we attend to a certain part when processing a large amount of information more specifically it is done using two recurrent networks as you can see in this image one network is tracking observations using the same rgb and depth input as the first model while the other network's role is to make decisions based on the user's fed instructions and visual features this time the user's instructions are encoded using a bi-directional astm then they compute a list of simple instructions which is used to extract both visual and depth features following that the second recurrent network uses a concatenation of all the features discussed including an action encoding as inputs and predicts a final action now let's just watch some amazing examples where the agent is following the instructions written below i invite you to check out the public release version of the code under github which i linked in the description of course this was just a simple overview of this new paper i strongly recommend to read the paper linked in the description for more information please leave a like if you went this far in the video and since there are over 90 of you guys watching that are not subscribed yet consider subscribing to the channel to not miss any further news clearly explained if you would like to start or improve with machine learning i've linked all the best online courses in a reporter in the description thank you for watching [Music] foreign