Building Robotics Applications Using NVIDIA Isaac SDK

Key Takeaways

hi everyone welcome to nvidia's gtc 2020 my name is asusa torabi i'm the product manager for isaac robotics platform in this session we will discuss about isaac sdk the toolkit for developing ai-powered robots and how you can use it in your robotics projects let me take a moment to share the plan for the next 30-plus minutes we will first start with why isaac sdk what's the problem we are trying to solve and how i will briefly describe main components of isaac sdk i have then divided this session into three parts first i will talk about the tools that sdk provides to help develop and accelerate robotics applications then we will take a deep dive into one of the fastest growing areas where ai robotics is playing an important role intra-logistics and how isaac sdk can help you in this application area then we will talk about how to develop and test ai-powered robot brain using ivexin and finally wrap-up with main takeaways and q a before we get started let me set some learning objectives though this webinar can benefit several types of individuals we have tailored it to address the needs of a roboticist looking to bring gpu accelerated application and ai to her robotics project or building ai-based robotics for intra logistics applications or an nl engineer looking to use simulation to train and test his robot's perception my goal is to impart on you a good understanding of what the izak sdk platform is and the tools it offers for robotics development application models and algorithms for intra-logistic use case available in isaac sdk and how izaksim can accelerate robotics development with ml workflows and continuous testing so with that let's get us started let us take a step back and look at why isaac sdk was created why isaac sdk we believe everything that moves will be autonomous robots will need to sense reason and act in tomorrow's factories and workplaces using ai and they must be highly flexible adaptable and safe to collaborate with and work alongside humans incorporating this level of ai into robots will require a software platform to develop performant apps training virtual world and deploy in real world real world alone is cost and time prohibitive nvidia's isaac sdk enables and accelerates this complex software development process isaac stk is a software framework built on nvidia's gpu accelerated ai computing platforms from cloud to edge isaac packs gpu accelerated dnns for perception and ml workflows for training and transfer learning modular algorithms for robot navigation and manipulation applications photorealistic simulations for training dnns and deployment testing of algorithms with hardware in the loop this is isaac sdk software stack the entire stack is built on nvidia's edge computing and cloud data center computing hardware accelerated by cuda for compute qdnn for ml tensor rt4 inference rtx for ray tracing and physics for physics simulations isaac engine provides the plumbing for robot algorithms and applications on top of engines are number of robot capabilities we call gems it is expanding universe capabilities including what our ecosystem provides such as sensors robots and more we include hundreds of reference applications that provide a springboard to get us started and finally a tightly integrated isaac theme for training perception networks reinforcement learning as well as for continuous testings of the application stack isaac engine features a modular computational graph based architecture with cuda accelerated messaging between nodes in the graph each application is defined with such a graph isaac engine also includes a number of tools such as record and replay to save all messages inside application graph in a log file and replay all recorded messages from a log see api to get access and send messages to isaac sdk applications python api for all algorithms enabling python programmers to include isoc in their apps and use tools like jupiter notebooks a visualization tool for debugging and visualizing application outputs isaac gem is a collection of ready-to-use modules including hardware drivers for a number of sensors and robot platforms fast and accurate deep neural networks that you can adapt to your problem and data to accelerate robotics application development this include perception dns such as 3d pose estimation image segmentation stereo depth estimation and more the robotics algorithms include localization path planning multilighter support motion planning for navigation as well as manipulation use cases sdk also include modules to include audio and text to speech for human robot interaction isaac include hundreds of sample applications in real and simulation environments some of them include reference hardware design and software applications such as card air robot for autonomous indoor delivery kaya robot for educational purposes and to get us started with isaac sdk based on jets and nano leonardo is a robot arm reference design based on jetson agx savvier for manipulation in simulation environment isaac provide applications for automated mobile robot for intra logistics for example for dolly docking and delivery for manipulation it provides perception and motion planning algorithms for robot arm pick and place boxes isaac sima applies nvidia's omniverse platform to robotics with a state-of-the-art rtx graphics and gpu-accelerated physics simulation isaac sdk also provides support for unity3d as a simulation back-end isaac's theme has various assets for factory and warehouse scenarios navigation and manipulation applications it has domain randomization to generate variation in data for supervised training and reinforcement learning it can also be used for hardware in loop testing for robotics application in simulation shown on bottom row at left we also use simulation data for training or perception algorithms such as object 3d pose estimation and free space segmentation on the ground isa ecosystem consists of several sensor partners that have brought up cameras and other sensors targeted for robotics applications for example leaps and frammos have certified drivers for geeky stereo cameras or sick has a lidar for safety and navigation ecosystem also consists of robot platforms such as universal robots frank imeka quadrupeds from unitry and mobile robots from bmw and segway as we will see soon isaac sdk embraces the ros community with the goal to bring best of gpu acceleration to ros developers with that as the background let's first look at the tools that isaac sdks packs for robotics development isaac sdk includes a number of tools in today's session i will focus on python api behavior tree rust bridge cross compiling for jetson and visualization debug tool python programmers can use isaac gems algorithms in their application using isaac python apis this also make it very easy to deploy test using python tools like jupyter notebook isaac sdk also includes several sample applications coded using python apis the next video shows with the few lines of python codes in jupyter notebook you can deploy the navigation app on a real robot that uses the jetson platform and visualize through isaac visualization tool behavior trees are one of the primary mechanisms to control the flow of task in complicated isaac robotics applications it has multiple codelets to define a variety of behaviors such as sequential tasks repeating tasks and parallel tasks behavior tree is enabled through python api for easy programming for example this tree shows dolly docking application in a warehouse the root of the tree shows sequential behavior with multiple tasks in order from left to right robot approach the dolly detect the dolly it drives under the dolly lifts the dolly navigate to drop off location drop the dolly drive out from under dolly go back to robot original home position ross offers a large number of software libraries and tools including framework algorithms sensors and robot platforms to help build robot applications it is extremely popular with roboticists and researchers everywhere now with raw spread you can bring the isaac sdks ai and gpu accelerated algorithms and simulation to ros application stacks as well as bringing ros packages into an isaac application stack rosbridge is a useful interface tool for ros developer to make collaboration easy and provide developers with a rich set of ai and gpu-accelerated algorithms simulations and more this will allows developer to leverage both the platforms to make great robotics applications isaac bridge could be used to convert ros messages to isaac and vice versa this picture shows rust bridge in action eros message from an rgbd camera input is converted into a isec message to be used by an isaac gpu accelerated algorithm on the left hand side you see ros reviews showing the camera's image and on the right hand side you see isaac's side giving the super pixel visualization in this example we use rosbridge converter to convert ros messages into isoc image and depth proto messages to be consumed by super pixel gem isaac sdk offers a visualization and debugging tool named site to remotely inspect robot perception planning and execution isaac site api enables creating variable plot visualize data in 2d 3d rendering this tool also has a web front and named website to look at the data from robot via site api using this tool you can also inspect inner working of algorithms and components it enables a variety of control and visualization and debugging capabilities with graph visualization virtual gamepad widgets mapping visualization and localization monitor the image at right shows isaac site running navigation application you can see a robot navigating in sim global map local map and computation graph of application isaac sdk supports cross-compilation for various jets and hardware platforms with advanced build system enables clean module dependencies hermetic builds and cross-compilation for various hardware platforms just by one comment you can run an application on a real jetson board at right on top you see the lacago robot in the real world on bottom you see how we connect lidar sensor and robot driver to isaac navigation stack which has been run on a jetson board the next video shows the deployment of isaac sdk navigation stack on leicago robots you can also see the map live on site visualization leicago navigates automatically to a target and then stops now armed with the knowledge about what isaac sdk is and what tool it offers for roboticists let's look into one particular application robotics for intra logistics we will look at what isogems algorithms and capabilities does isage sdk pack for intra-logistics robots i will talk about perception localization and planning models available in isak sdk for intro logistics applications in the next slide first we start with the video that shows all that's possible with isaac sdk in this video you see a concept of factory of the future that's enabled with isaac sdk you see robots operating alongside human workers real time autonomously and harmoniously robots demonstrate perception and moderate skills this entire factory is presented in ifaccim where each robot clone of aerial there are three autonomous mobile robots that perceive dynamic objects such as robots and human workers and navigate safely they also perceive 3d pose of certain objects like dolly there are also more than four robots armed that are picking placing objects on a mobile robot in a synchronized manner you will also observe many different perspectives including camera views on amr or robot arms an awesome map view that shows how the planner optimizes the path to destination and avoid obstacles in its way now let's take a look at the video factory is a dynamic environment with robots and machines sharing the space perception required for both collision avoidance as well as for detecting objects and its pose for manipulation and navigation applications isaac sdk provide multi-lidar collision avoidance in the navigation stack so robots avoid obstacles while moving it also has dnn based object detection and object 3d pose estimation for molecular camera the next video shows in simulation how dual lidar planner with obstacle avoidance works there is a forklift that is blocking the amrv perceived by the backlighter so planner change its path to avoid the obstacle now let's take a look at the video isaac sdk provides gpu accelerated global localization that increases robustness against ambiguity compared to single lidar localization it estimates the pose of the robot without prior information it works both in simulation and the real world isaac sdk provides a navigation stack which has path planning to optimize robots paths to target an lqr planner that computes an optimal smooth trajectory for robot planner is available as gem separately as well as part of the navigation stack the next video shows with the map how navigation planner works and amr navigate to a predefined waypoint it also shows how planner optimize and switches to the most optimal path to get to the destination for manipulation isag includes an lqr planner that produces a simple smooth plan to move a list of joints in the robotic arm from a starting estate to a target a state the 2020.2 release of sdk will also include rmp planner which provide fast and smooth collision avoidance naturally integrated into the joint space and cartesian control interface cartesian interface is the standard type of end effect or api provided by robot manufacturers rather than just providing a straight line movement in end effector space or a straight line in configuration of space we enable similar interfaces except with fast built-in collision avoidance the motion planning algorithms work both in simulation and real world at right on top you see robot arm pick and place boxes in simulation and on bottom robot arm that stacks colorful cubes in simulations using lqr planner the next video shows kinova robot arm is stacking colored cubes on top of each other using perception models and lqr motion planning you also see visualizations for cube 3d pose estimation in the final and third section i will go over how isak sdk accelerates development and testing of robots in simulation with supported workflows one of the main goals of isaac sdk is to enable training dnns and algorithms in simulation as it can provide an infinite stream of inexpensive labeled data this will reduce the time and expense to generate training data in a shorter amount of time isaac's theme has domain randomization sensor models photorealistic quality that all contribute to closing the gap to transfer trained models with simulation data to the real world training robot's perceptions and behavior in ice axime drastically narrows sim to real gap and delivers orders of magnitude cost savings the picture at right shows an example of randomized training data for dolly tree depose estimation the next video shows the isaac seam domain randomization it shows variety of assets under different lighting condition position sizes colors in the warehouse also shows factory box assets in different colors and lights and positions isexhim could be also used to test isak application in a photorealistic and physically accurate seam environments before deploying in the real world for example the picture at right shows testing navigation application in a virtual factory environment isaac sim environment includes all related dynamic systems for a real factory for hardware in the loop testing ultimately hardware in the loop testing will speed up deployment in the real world so now i will summarize what i presented in this session the main takeaway from this session isaac sdk is a ai powered modular robotics platform it contained gpu powered low latency models and applications it has many features for navigation and manipulation especially for intra-logistics applications finally it has a fast accurate application development and testing hardware in the loop solutions in simulations thanks everyone for attending this session alrighty everybody thank you for joining us for this session i'm going to hand it over to our speaker at tucson to help us facilitate and go through the q and a two see you there yes yes all right fantastic well fantastic presentation thank you so much for that it was really exciting we've got a lot of questions here in the inbox uh you guys want to shall we just start from uh from the first one into tucson um so i think um how we should start yeah let's go for the safer end so i think she answered most of them that's right so yeah yeah go ahead yeah let's let's i think we just saw a couple of questions calling and go ahead and just uh let's do them live so uh glitch tree once you ask the first question for us yeah i think there were a lot of questions about uh ross and ross too i think at this point in time we are supporting ross bridge ross ii is planned for the future as it matures um yeah so that's one question and there were questions yeah yeah sorry if you're all on the line we have a lot of questions that just came in so we're going through them so just be patient yeah yeah feel free to jump into so there was one question from um about i would like to know what's the difference between d stream and isaac sdk and if they can be used together so deep stream is our camera uh streaming uh uh sdk so it takes the camera inputs and and outputs uh essentially the uh inference uh the ml uh models are done on it and so we do use deep stream uh for uh anything that is camera streaming within isaac sdk so both are integrated we have a few more questions that came on in there were the questions about uh domain randomizations and uh generating photorealistic data for training yes isaac themes provide this um so you can search for icelanding omniverse kit um you will see you can have access to it and that could be used for the generating domain randomized data for training and deep neural networks excellent thank you tucson i actually was a good question i saw a few of those coming for that too so there was a person yeah go ahead i'm gonna go ahead so there's a question about do you have a course on isaac with ross and so uh we are actually planning a detailed webinar or a hands-on webinar uh in a month from now which will where we will talk about how to use ross as well as with isaac and there are a couple of other questions in the same um and we are also planning a a blog that you should see soon which will tell you exactly how to use rosbridge and there are several use cases we can imagine there are developers who like to use navigation stack entirely built on raws and tap into specific capabilities in isaac that is a possibility or the other way around you could build a complete navigation stack using one of our reference designs and use um capabilities in ros so both are completely interoperable so there's no concerns there okay there is a question about if we want to have a custom neural network uh yes you can implement your own uh there is e-web there is a way and you can write your own codeless and add your own neural network in the framework very good thank you so sri questions about our gems open source i think that one was also asked many times about the binaries or if it's open source correct so gems by default are not open open source we keep them um at this point the it's kind of compiled binaries but some of the algorithms as a tusa mentioned we we are able to for example pose estimation dnn you can replace your own neural network for object detection uh so we make it modular but it's all binary at this point in time you can retrain some of the neural networks but still in binary at this point now if there is a business need we can have a conversation see whether we you need any more specific if you want to discuss that it's a matter of business discussion but out of the box it's close source but uh isaac engine and the rest of the stack is completely open source you've seen developers take it and customize it to suit their needs and things like i the visualization tools and and so on are completely customizable so there are questions about if you can implement like a new applications like models to do a two wheel balancing robot so iso iso sdk is a robotic platform you can they are some um pre like the models you have access to it there are some apps and you have access to it but you can also implement any new apps you want to in this platform excellent all right great uh do we have a few more questions that came in here yeah yeah so sweet there's the questions about can we talk about price fans i'm not sure yeah at this point in time the way uh isaac sdk is just the value-added software on our uh hardware platforms so isaac hd is free to download and you can use it and uh it best works with isaac nvidia's hardware platforms whether it's on the edge or in the cloud now uh there's no pricing for isaac sdk so you have to purchase our hardware platforms to be able to run it efficiently so that's where that's how you would uh you would access it so if you need to run it on your desktop you would have to have a gpu um and if you have to run it on a robot you would have to buy our jits and platforms yes and that was also a question about what is the minimum gpu requirement it depends on your application it is going to be used yes correct so there's no minimum gpu requirement we expect gpu to be there and obviously it's a one architecture so whether you're running a jetson nano or you're running it on a super pod which is a highest in supercomputer it runs isaac sdk is compatible with every one of them okay so what are the resources there for starting with it sdk so i would say probably isaac sdk documentation is one of the resources so uh we will also work on many uh developer blogs that are coming some of them are out there some others are coming soon and also we have a developer for room for the questions that i would suggest uh we will gonna work on it and we will answer to all of the questions regarding our society so is there any other resources we should streets those are the main resources thanks to sir we have developer forum and uh which is uh we moderated very frequently and uh blogs are coming lots of good blogs and there was a question about realistic camera models for common cameras like connect real sense etc so at this point in time we are providing photorealistic camera models uh for rgb and depth and also lidar we haven't specifically narrowed down to types of cameras uh like weather real sense those levels of capabilities will add in future so whether you use a kinect real sense or a time of flight you should be able to model them in isaac sim so that's part of the isaac sim again one other question was about are there any parts of source code isaac sdk available to the community and partners and having the ability to modify lab gems in particular would be greatly appreciated by the team we sincerely appreciate that thanks for the feedback isaac as we get today as i mentioned the isaac engine portion is open source some of the gems are actually open source we have taken it and gpu accelerated them whether it's stereo depth for example or april tags detection these are all open source code we just taken that and accelerated neural networks are completely modular so you can train it with your own data so we would love to understand what specific gems are of interest feel free to reach out to us on our community forum and tell us which gems you would like to see open source it's a matter of understanding what the needs are and based on that we can work with you there is a question about availability of rnp planner yes we will have this in our next release so yeah that was the question if you will have it in the next video and then looks like there was a question here about licensing for isaac shree can you comment on that what is the licensing i just i i said yeah sure i'll i'll repeat that isaac today is uh licensed under um an end user licensing agreement that we have you know so you would download it under that and use it and um there is no uh you just just have to register as part of our developer forum and download it excellent thank you and uh the question is can it be yeah sorry uh can it be readily used in production it's intended to be a production ready software it's a high quality software we've been built it built from ground up to be used in production now uh we do come out with uh releases at least about two or three times a year so with every release we are focusing on the latest release for um our development activities so if uh if anybody needs um a support for uh an older release that's the conversation we should have as to how to enable a long-term support excellent thanks shree the questions about supporting uh which um each operating system are cyclistic support currently it supports um linux ubuntu 18 18.04 so yes and also there was a questions about interface for the simulink and matlab so i do not we do not have this support currently so is there any comprehensive iso training course available there is no course but we are working on the webinar in the next month so uh there was a couple of other questions let's just kind of go through it could be used the sdks work on autonomous cars isaac sdk was built for all kinds of robots and um in theory it could be used on autonomous cars too but autonomous cars nvidia has a platform called drive and drive works is a sdk for that autonomous cars require a more compliance with regard to safety by different regulatory bodies so we would recommend you to use drive platform for that from nvidia there is a questions about uh difference between testing in simulation environment and the real world so our goal is basically to adopt uh and train our model in a photorealistic simulation data and also they close the gap between the uh like transferring the simulations to the real data um so uh to the difference is that the testing in simulations of course it's in a simulation environment but um you should be also able to test those models for the real world so yeah thanks satoshi so i'll just continue with some other questions are there plans to support autonomous aerial vehicles within isaac or is it already possible um we do not have a reference design for an aerial vehicle with an isaac sdk as the tucson presented we have mainly for hindu robots quadrupeds and and holonomic wheels and industrial robotic arm and so on however we have seen a few of our developer community customers build autonomous areas the aerial vehicles just with isaac so and they have been able to successfully deploy um drones and such using isaac sdk now oftentimes people use a specific jam for accelerated vision or something but they have been able to use it next one we're sure you're getting a lot of questions right now it's a good discussion we have absolutely thank you thanks for thanks for a bunch of questions and frankly speaking if we are unable to finish all the questions today we'll uh we'll make an effort to capture all the answers and put it on our developer forum and uh make sure it's all answered yeah that's important to note that yeah if your question isn't answered we do make an effort to ensure that they are eventually answered in the developer forums or included in some of our messaging lines yeah there are a lot of questions about isaac sim here uh although this we are really focusing on isaac sdk but i'll try my best to answer some of them one question was how much effort would it take to create your own robot model uh i i'm not too sure whether you're talking of a real robot or a virtual digital twin of the robot and sin a real robot uh really we have a bunch of good reference designs that you can work off of uh carter is our indoor robot in a reference design uh kaya is our holonomic uh three wheel base uh reference design uh but then and then there are also supported platforms like leicago's leicago from unitary and so on but if we have it created in the virtual world isaac's um we've known people to create it very quickly there are very useful tools to both uh create robot and add physics to it and include sensor support for rgb and lidar and so on that should be doable very rapidly there is a question what's the difference between data stream and eye cycle sdk and if we can use it together so different stream is a video streaming and video analysis framework and our sophisticated for robotics and yes we can work them we can use them together we can use some capabilities from data stream in your iphone sdk application well it looks like we're all just inundated with the inbox there's so many great questions coming in wow there's a question about excellent designs tree uh i don't know if that's an easy answer ackerman steering yes yeah all the reference designs available using non-holonomic ackermann steering robots as a matter of fact we are bringing in ackerman steering uh to robots this just a matter of addition um the good thing is uh the the the robot controls it's very easily portable uh we have we were for example just an example we took carter's navigation algorithm and ported it on to um quadruped and just ported very easily and one of our couple of our customers and partners have also been very able to do it now coming to ackermann um you might have seen in our keynote yesterday there was a parking autonomous parking of autonomous cars that used uh pieces of isaac sdk so it's the easy to port it to aquaman we will bring the support in an official release in a future release maybe in the next one or the one after but you don't have to wait for it you should be able to port isaac sdk to any kind of robot platform and our developer community many people have successfully done it yeah just kind of repeat the question um is isaac sdk support for autonomous vehicles yes it is meant to support all kinds of autonomous machines autonomous robots however we've been kind of very specifically recently focused on inter logistics applications which means autonomous vehicles inside a factory or a warehouse kind of environment but it could be used in other use cases so there's a question again a lot of questions comparing ross and isaac um i i just want to repeat isaac and ross are very complimentary the whole idea of isaac is to bring ai gpu accelerated algorithms to robotics developers whether you use raws or your own custom stack right so um so isaac has the guys that navigation stack uses a lot of gpu accelerated algorithms for example gpu accelerated global localization is a very important piece which could be used in a ros navigation stack too so that's where i think a tucson presented the whole idea of rosbridge and so both of both of them are totally complementary we just wanted to build the isaac based navigation stack to showcase what is possible with um nvidia's platforms so that as a ros developer you could use what you need it's totally up to you you could use a complete cross navigation stack with specific isaac algorithms or you could choose to build the whole stack in isaac and use ros algorithms both are options for you so keep keep let's keep taking questions for yourself so i think one of the question is can i use isaac navigation stack without a lidar today today as of now our navigation stack is built on lidar but we are bringing more capabilities around visual automatory visual inertial odometry visual slam etc so you could continue using your navigation stack whether it's your custom or based on ros or any other platform and you can use some of these capabilities from isic uh with i with lidar also we are using as global localization as a gpu accelerated capability but then um as we also have a stereo visual inertial odometry built in and we apply the future release you'll be seeing visual strand capabilities or you could use uh lidar so there's a question what's the interface between the justin and a robot in the real world jetson and robot in real world so uh you can deploy you can write your application and you can easily deploy it on adjustment on the real world like for example navigation so we start the questions related to this cross compilation yeah i think that yeah sheree and natus just want to let you guys know so it looks like we're getting close to the end of our sessions i think let's finish the response for this one and then we have time for one more question sorry yeah there's question about will there be more hardware sensor supports coming in the answer is yes um we are planning to bring in a lot more sensors into our isaac ecosystem but uh we want to be very focused on what we bring on uh we that's where if your goal is to use um if you're prototyping and building a research platform you would highly encourage you to use ros community so you can bring a ros if you have a sensor that's got ros driver bring it on and you can use it with specific isaac algorithms so we've got some examples coming in one of our blogs very soon but um if you're looking for a commercial deployment specific capabilities we are we're working with some specific partners to build industrial grade cameras and industrial cases great lighthouse and other types of sensors so that you that will help you accelerate deployment of robots in a in production environment but we are very focused on what we want to bring that's sensor and robot platforms so we have a question about uh lqr motion planning for the collision detection uh in the next release we will have some support for the static uh collision detection of team it looks like that's all the time we have for to do all the live q a i know it went really quickly uh with the presentation but it's been an excellent morning uh so we'll have to wrap up q a for now uh but just for the folks on the line this presentation will be available through the same link in the session catalog within one hour so if you want to come back and check out the content i'll be available on demand uh so just log into the session catalog thank you for joining us today for the session and we'll see in the next one but uh thank you everybody and uh have a great day thank you thank you thank you so much bye you