JETSON AI LAB – E07 | NVIDIA Interns

Key Takeaways

hey everyone welcome back to jetson ai labs we're really excited for today's episode uh we've got a special show for you with a bunch of our summer interns from the jetson team at nvidia wave hello everybody hey there and um this summer intern program has been interesting uh since you know basically all virtual and we've got a lot of interesting experiences from you know their time at nvidia that they'd like to share with you along with some really interesting projects that they've been working on um so why don't we go around quick and you guys can just introduce yourselves really briefly uh before we get into the news uh tomas do you want to go first yup sure uh yeah so hi everybody hi youtube uh my name is thomas i'm uh i'm originally from poland right now i'm located in san jose california and i graduated with masters from san jose state university in computer engineering and uh right now i uh i'm an intern for five more weeks and the jetson team working working as an embedded engineering intern uh yeah and later i will have some cool stuff to show you looking forward to it alexa you want to go next sure uh hi everyone my name is alexa um i currently study electrical engineering at the cooper union going into my last year of undergrad uh and this summer i worked as a hardware intern on the gpu products team i'm working on my jetson nano baseboard which is an open source educational baseboard for the jets and nano so i'll talk about that a bit more later hi my name is mikias desta i am doing my masters in computer science at georgia tech and i have worked on a couple of projects here at nvidia and i will share that with you later hey everyone i'm avi i am an incoming phd student at the university of illinois or vanessa campaign in the electrical and computer engineering department um and so i have one week left my internship here at the jetson a team at nvidia and then i'll continue there for my phd so excited to share my project with you as well hey everyone i'm rishabh so thank you dusty for giving us the chance to come on jets in the eye labs first of all because we we were interns last year i think technically not summer interns anymore so you fall in turns last year and i graduated with my masters in robotics from wpi this summer so really excited to discuss my projects with you hey everyone thank you so much for joining my name is asavari and i was actually an intern last fall so um i joined full-time pretty recently um earlier this month and i'm just excited to talk about my internship projects and nice to meet you all awesome thank you guys and as always we have dana and jim from jetson hacks here how have you guys been doing well doing great that's great okay so the plan for today's show is essentially uh we're all gonna go around and share uh what they've been working on and you guys can ask some questions in the chat which we uh encourage you to do but before we get started with that we just have a couple quick news items that we'd like to share that have happened over the past month um probably the first one of which let me pull up my screen here quickly uh jetpack 4.6 came out earlier in august here and has a bunch of new features including support for the triton inference server and a bunch of cool new things with how you can boot um directly from nvme as opposed to sd card on platforms like agx savior and xavier nx and jim i think you had mentioned that you created a video recently on that topic is that correct that is correct there's a article up on the website and a youtube video about native boot from external storage so if you can keep your enthusiasm down about that what it allows you to do is using scripts on your host you can flash your jetson i think they're both xaviers so agx xavier and xavier nx to boot from an external mvme drive yeah that's awesome that's been like a long running request from the community understandably and we had to make a bunch of changes to the bootloader and kernel in order to enable that so essentially it used to be like you could have nvme but then you could put your root file system on there but things like the kernel or bootloader still lived on the sd card so now essentially everything can go in nvme and can be really fast i personally always use uh nvme on the xavier devices and can get up to like 3.2 gigabytes a second read on those so those are really fast um and then i should also mention we have a couple webinars out this one on a jetpack 4.6 deep dive that my colleague suhas gave yesterday you guys can watch that on demand and in there is a demo of how he set up the nvme and some other features including over the error update using a b redundancy so you can essentially have two images on the device simultaneously for secure ota updates and then coming up next week we have another webinar on vpi and the new python bindings for that vpi if you're not familiar is the vision programming interface library that it has back ends for cpu cuda and also the computer vision accelerator engine on xavier and has a bunch of features in there for things like stair disparity optical flow a bunch of pre and post processing computer vision algorithms that you can run so i really encourage you guys to check this out because it's basically the highest performance computer vision library out there for jetson today um and dana anything new in your neck of the woods well along with jetpack 4.6 we have new released container tags for the getting started with ai on jets and nano course and it's now in four languages so we have an additional two languages from before so it's in english simplified chinese traditional chinese which is new and japanese which is also new so awesome yeah along the lines of the containers all the containers for 4.6 are now up and there there is actually something new with the containers that we've done is we are starting to transition over to having all of the cuda libraries and cootie and stuff installed directly into the containers so there are new l for t cuda and alpha t tensor rt um containers out there now on ngc that have all of that stuff inside the container which is how you would normally do it on like a pc or server platform it's just that previously on jetson we had mounted those all from the host so in future releases we're going to be migrating more and more towards that we call it the fat container approach as opposed to like the slim containers which we had been doing on jetson previously and the big advantage of that is that with the compute stack and cuda and all those libraries bundled inside the container in the future you will be able to run containers that are built against different versions of jetpack on the same jetpack so you know you'll no longer need to rebuild container for every new jetpack version you have a container that's built for 4.6 and 4.5 running alongside of each other with that are independent and running independent versions of cuda which should make it a lot easier to deploy these types of applications um okay i think that's basically it for the news i encourage you guys to check out those webinars and if you haven't already the latest updates from jetpack 4.6 you can just typically apt-get install those without having to reflash your device if you so desire okay so before we get too far into the show here i do want to save as much time as possible for the endurance to present all their great projects um so with that we're basically gonna go in order here as the brady bunch layout that we got going on so uh tomas is gonna go first and share with us what he's been working on at his time here yeah thank you um yes so hopefully you can see my uh screen can you see my screen already um there we go yeah so i have a little slide deck for you um so the core project i was working on was uh that sort of uh it's sort of jet tracer uh style robot in that it is it is a rc car with um jetson on top of it um but uh but yeah uh here here the the the the goal was that the robot would serve as a uh as a reference uh multi-camera application for uh jetson and that people can hopefully uh take it and uh and sort of build on on top of it so here you can see um here you can see a custom jetson nx board with uh gmsl two interfaces gmsl 2 is is sort of similar to me pcsi but it's it's sort of uh it's a different interface that's that's used more in automotive and uh and industrial robotics and yeah as i said the goal was to show a multi-camera reference application and the challenging part was to offload all the um as much of of of the compute load from the um from the cpu and gpu to to different accelerators that you can find on uh on on xavier so um so so hopefully at the end of it as much as much as possible uh of of the cuda and cpu resources is left to the user um so you can do pretty cool stuff uh with it and and it's it's it will be available on uh github by the end of my internship but but basically using as little as a little more than one cpu core you can basically capture three 1080p 30fps streams push it through uh through two two deep neural networks that are object detector networks and uh map every every every image to a numpy array and code encode a video stream in with with h.264 codec and do all the file io for it display everything and uh and by by the end of it you're still left with with most of uh your your cpu power uh for your application and i think the best part is that you can you can still run another full real-time network on top of it because uh you're left with essentially 100 percent of your uh cuda resources um did you just use g streamer and deep stream for the camera acquisitioner how'd you do it efficiently that's that's that's the idea uh g streamer nvidia released over the years uh a lot of components for 4g streamer and and as of today this is the best way to to access the hardware hardware acceleration for for uh for for many of the computer vision stack but but the problem was that it was not necessarily the most friendly um uh friendly interface so using my project you'll be able to to essentially instantiate a pipeline and it will create uh create a gstreamer pipeline dynamically under the hood and what you're doing is essentially just just getting a numpy array um from that pipeline if if that makes sense um so and uh yeah and typically typically if you if you did that um if if you did that using opencv calls uh it it might be a little less efficient uh and uh yeah also you you get uh all the results of of the object detectors uh and and uh file uh file io out of the box uh here is uh here's sort of like a highlight of the internship when you do an internship in nvidia uh you you get to you get to meet with the ceo which i don't know i thought it was kind of cool um a really cool cool guy by the way like if you if you get to uh meet him in person like it it's it's it it's it's the coolest billionaire i know um and yeah and the last thing uh it's not my internship project but it's it's a project i was working on uh recently it's it's called jet drone so uh i think that that i thought was really missing in the uh jetson ecosystem was was um was a jetson-based drone and um and not not just not just the drones that were out there and that i was previously working on were sort of you know like a off-the-shelf uh diy drone with just jetsons trapped on on top of it um which which firstly it it it's it it wasn't super it didn't look super super well uh but but also uh it's um you didn't end up with um you uh the the whole thing ended up being uh being much much heavier and and and you you basically ended up having with all those wires around it so i thought it would be really cool to create uh create a drone that that's that is sort of jetson native and uh and yeah that's that's what jedron is is about so right now it's just the hardware uh but yeah the whole chassis is uh 3d printable so except for those carbon fiber tubes and i will post a link later to that the whole chassis is 3d printable and you get stereo cameras uh right here um and later there there will be uh code released for uh for actually uh calibrating those and and performing uh and calculating uh stereo disparity and uh yeah here here you you're probably familiar with that you get uh xd60 connector for your battery and this thing is is a full uh full jetson nano developer kit which is connected to those two uh cameras and uh and uh yeah i will release it i will release it probably uh later [Music] later this month and uh yeah hopefully i uh that that will be um that will be home to some cool applications in the future um i do have a video uh to show you if we uh so yeah that's basically the uh the maiden flight of uh of that drone right now it's not not super not super it isn't doing uh anything super impressive but uh yeah the next step will be um we'll be showing all the piloting that the drone with uh with with jetson on board and then hopefully doing some cool computer vision stuff uh on top of that yeah also if you if you have any questions about that please feel free to drop comments uh in the chat awesome that that's a really nice looking chassis so did you like design that whole thing yourself tomas yeah uh yeah so it's it's designed uh i designed it in infusion 360. so um so yeah i will drop um i'll drop a link where you where you actually will be able i don't know if i'm am i still screen sharing uh here we go this is this is the 3d model uh so i can i can share it i can share it with you uh later and you can export all those components as stl files and just just print them and uh yeah those are the lenses for for the cameras and uh if we if we turn that thing uh here uh then you can this is a pretty familiar view for uh for a lot of you guys i think this is this is just a jetson nano developer kit um and uh yeah so the the link will be uh somewhere in the uh comments all right i think i'm already over my time so uh i will pass it to pause the mic to the next person no worries thanks tomas those are a really cool project and uh great work uh okay next we're gonna have alexa talk about the custom pcb that she made for jetson nano yeah absolutely thanks um so i was an intern both last summer and this summer and i worked on the same project which is called my jetsanana baseboard so the goal of this project is to enable makers to create their own hardware designs um and like basically just design their custom pcb for the jetson nano um and so if you could show the uh github please um so here's the it's on github already uh y'all can star it work it like start using this right away um so i was responsible for schematic layout verification all that good stuff so i actually do have um the first a0 prototype here that was validated and the issues were fixed with it um so in the spirit of open source i've shared all the documentation all the files that i used um also the issues that i found while doing validation are here so you can check out um things that were incorrect uh if you want um and yeah so i think this is a really cool project um in that it has the potential to enable more makers to get involved in hardware which is something that is a little bit less developed than open source software open source hardware is i think not as common um and so i think we're lowering the barrier to entry in terms of hardware which is really cool um i can talk more about the technical specifications if anybody is curious but um yeah happy to answer questions i think we're also interested if um because i think you're alexa you're the first person on the show that actually is doing like a hardware baseboard design and has designed a carrier board for the jetson so it'd be very interesting if you just kind of like walk us through the process of what you had to do to design fabricate and validate um this board yeah absolutely so i actually have um in the documentation i have uh set up this file which basically goes through the the process and like um how how you might approach designing the board i think the most important thing for me was um reading all the documentation that nvidia puts out so there is like a product design guide for the jetsano there is like all these specifications um and so for me it was my first time doing a hardware design so i had to learn about like schematics and uh layout and all that stuff i definitely learned a lot about um for mostly high-speed signal routing which is like a challenging thing um but initially on the ideation process uh we knew they wanted to build a board that was a little bit easier to design than the nano so there are some features that the nano has that this board doesn't have like camera connectors ethernet and displayport and there's some additional features that are more user interactive that this board has like there's a servo header there's some leds for toggling gtios there's an oled display that sort of thing and those features that are more user interactive um fit with the goal of teaching people how to use linux for tegra and those kinds of features so you mentioned like this was actually your first time um doing a baseball design which i find to be very impressive considering the high speed signals and things like that so did you do any other like test designs or other like really basic things to familiarize yourself with that or just basically dive straight in i kind of um well so the the schematics in the layout for the jetson nano are already posted externally it's already open source uh for the baseboard at least um so i reviewed those schematics i tried to learn from what nvidia's designers had already done and understand why the circuits are like the way they are um so that i could replicate aspects that i thought would be good for my board and that sort of thing um i also made sure that like one of the great things about having internship at nvidia is just the sheer amount of like knowledge that there is on my team i have a lot of really smart people doing a lot of really awesome things so i just decided to take advantage of asking them questions and figuring out uh what they knew and how they could help me um in doing this i did design a few like smaller boards that were like turning on an led but i didn't get those manufactured it would just like get used to using key cad and that sort of thing okay is that what you what software suite you use for this keycard yeah okay i showed you chose keycard because it's uh the free like open source um like pcb designer that's the most common and has the most features uh like internally we use cadence allegro but the licenses are like very expensive for individual makers so it doesn't really make our sense yeah so like there's a lot of software people on the channel and in the community and while there are a lot of hardware designers they typically uh tend to more on like the professional side of things and um i can't recall if you you have an electrical engineering background or not but for somebody who's coming from software um but like wanted to get into pcb design what like tips or guidance would you have for them and like how do you even get started in such a thing right so i my background is in electrical engineering that's like what my degree is in um and i guess my advice to people starting out would be like choose a small project same thing with like a software thing like you wouldn't immediately like go architecting a whole database or something like that you would like do a hello world program and like figure out if it worked um and so i would suggest maybe building like a smaller board that would enable you to do something simple like turn on turn on and off an led or like um i don't know like control a motor something like that it's like very simple um and then once you have that down pat and you understand like how everything works then you can go to like more complex signals and like start reading more about the harder stuff uh there's a saying that's like it takes less time to build a four-inch telephone mirror that it does to build uh sorry a four-inch that an eight inch then just an eight inch so the idea is like by failing and by doing something small you can like do a bigger thing uh later somebody's asking about the total cost um in the chat so for us we paid um dollars about uh on jlc pcb for 10 boards so it works out to like 40 a board so that's just like um an estimate you can there are definitely other manufacturers um that will build this for you for varying prices depending on what their capabilities are and did they put all the components on it for you or was it just the raw pcbs and you had to like actually assemble it the challenge with this was that they glc pcb specifically doesn't put some of the parts on the board that they don't have themselves in their library um so we did have to solder on some extra parts um so that is like a consideration like for a maker like if you don't want to solder the sodium connector on by yourself at home you should probably find someone who's willing to solder it on for you okay this yeah the sodium connector can be a little tricky uh i suppose but uh sure awesome so alexa one last question for you like what future plans do you have what are you planning to do next now that your internships ended um well i'm going back to school for my last year so i'm i'm excited to you know spend uh spend some time learning and going more in depth in this um yeah and i'll i i hope to continue working with folks at nvidia in the future i mean i really enjoyed my internship experience so uh it was great awesome yeah it was great having you and it was great to have a actual like hardware project that that put out there so um you know everybody from the team and the community really appreciates that yeah absolutely thank you all right so uh mikius we're gonna go into you next and uh you can talk about what you've been working on and what you plan to do next in the future sure thanks dusty uh so yeah as i said uh i worked on a couple of projects in nvidia i'll talk about two of them now the first project that i worked on is that it's a hand pose estimation what a hand pose estimation pretty much means is that it it's pretty much predicting the joints on your hand from an image uh it's becoming popular this days because of uh because of the ar and vr like the emergence of ar and vr it has applications and um and human computer interaction sign language recognition and also like remote surgery and those kinds of things so when i started my project what i did first is that i worked on i first researched on um deep learning architectures that can um that that can do hardware acceleration with tensor rt that can use uh tensor rt and that are fast on jetson devices um and like there are a lot of papers that talks about headphone estimation but most of them can be categorized to two approach the first approach is the bottom up approach where you first detect the joints and then you learn the relations or the associations between the joints and the other approaches the top down where you first detect the hand and then learn about the where where the joints are so i did the bottom up approach there's a project that has been done on nvidia by john welsh on trt pose that's just a human pose estimation project i followed that and it uses resnet18 as its backbone and uses like attention mechanisms um so i followed that and we collected our own dataset inside our nvidia which is do you want to share your screen oh okay i thought you were seeing my screen sorry oh i see okay i thought it was sure okay so you guys missed a little bit so okay yeah so this is uh the project that i worked on the handful estimation so i was saying that we collected you know somewhere around 2.6 000 inmates or 2.7 images we annotated them um like we were annotating 21 key points on a hand so as you can see here we have 21 key points we annotated and trained them we used some we did like some data augmentation um to make the model work well so as you can see it it's working well and stable here once i finished doing that i did some applications that you know shows how to use this first application that i've done is that the hand gesture recognition what i did is that i pretty much made this jupyter notebook where you can train your own hand gesture recognition or classifier it doesn't need that many image it takes the output of the pose estimation and you can create your own hand gesture recognition pretty much you know you can say this is fist this is like you know stop uh with few data uh points so uh that was working well as well so you can go and take a look at it i just used a simple svm support vector machine for the classification um and then i used that application to do two two more demo applications the two demo applications are uh the first one is that this cursor control where elates users control their computer screens using just their hand it was pretty interesting because you know as a use case what i did is that i just bought tuna sandwich online just using my hand i didn't do any clicks or i didn't use the keyboard it pretty much is handy for the covet time especially so and like on the right side you can see that it's not just only clicking uh you can also do zooming zoom out using your hand pose your your you know your hands or fingers um the other application that i worked on is that a simple a mini pen type of application where you can draw like erase clear on your camera screen uh here on this video are you guys able to see this video okay cool so you can see that you know i'm just drawing on my screen on my live screen you can do rectangles you can erase it you can you know um you can do rectangles you can erase clear so yeah simple mini mini pin type of application that's really cool cool yes yeah i've actually i've tried this project and it's a lot of fun to play around with and um i thought it was really interesting how you use the svm on top to actually do the gesture classification um what do you think that was the most time-consuming or challenging part of getting this to work uh this was the most challenging was the data collection annotation and also you know uh i've learned a lot from it like you know this is just a real data so like the first time i just tried it it wasn't working right like we had to take different angles for the data collection from different people it was not generalizing at all at first we had to use some data augmentation like you know we were masking the hands and throwing the hands into a different background so that it could learn with different backgrounds and also the lighting conditions were also really mattering you know because we didn't train it with many lighting conditions first but then once we did the masking with different backgrounds it was working well how did you how did you annotate it did you have some kind of widgets that you kind of built in to kind of collect the data and you know here's the tip of the finger that kind of thing yeah so we made an annotation tool like uh john welsh like made the annotation tool for me and then we were able to go and annotate the data so you just go click on the hand um and then you assign it later and then once you assign it a letter it was able to like you know we have 21 liters that corresponds to each joint on the hand and we save that to a json file and then later we train it how long did it take to train it uh it didn't take much uh probably like you know three hours to six hours it depends on like you know how many iterations we were doing but yeah six hours i would say is the average and you trained it on the nano no or no or elsewhere okay an x86 machine like all right new v100 gpu machines yeah but on jetsan it's mostly the inference part so this was my first project my second project on deep stream uh d warper plugin project and the what a deep stream is for some of our audience that doesn't know it is a multi-platform scalable framework that you can build and deploy ai powered video analytics apps and services so it has many um because can you change the tab in your browser sure there you go so this is it yeah thank you desti um so it has many plugins and one of the plugin is the dwarp plugin what the dwarper plugin does is pretty much it the warps inmates to a different it transforms them to a different type of projection so you could have different type of projections for example a panini push broom fisheye 360 type of cameras you can have image videos from those kinds of cameras and then you'll be able to de-warp them to different projections to perspective or push broom or pan any like this the advantage of using this tv application this uh plugin is that some of the machine learning algorithms or like object detection or people detection kind of struggle on detecting on a fisheye camera or like in different uh projections so you can project them to a different uh image or image a different projection like for example perspective and you'll be able to detect it better so as you can see from this example while we were just using the fisheye camera we were only uh able to detect two people but after after we do warped it we were able to detect more people and also we were able to see um the image better than like the fisheye 360 camera so i what i first worked on in this project is just updating the uh dwarp plugin like it only had three types of projections i added like 15 more projections and then i just made this sample up where it shows you know how to use the warper plugin and also on top of that how can we detect people or you can do your own type of detection you can use object object detection or any type of detections from the deep stream it has this config file where you can just go and play around it and if you have questions let me know um yeah so i it needs some kind of 3d geometric knowledge i didn't have that knowledge but you know there are a lot of smart people in nvidia i was just like you know going and slacking them about it and they're pretty helpful on that um so yeah this is the projects that i worked on uh one thing i just want to mention is that uh you know i worked at ibm on research oriented project before and like somehow how i differentiate what i worked on here is that i here you know we we focus on optimization performance and i'm glad that i worked on with real time from you know data from the camera stream everything was real time when i was working in ibm it was more of a research type of project so we use uh synthetic data synthetic datasets also we weren't as much concerned about performance and optimization and it was pretty cool to uh have to learn this techniques about tensor rt and i'm grateful for that awesome thank thanks so much mickey it's really appreciated um okay so next uh we're gonna have a bee and share what he's been working on um yeah so hi everyone so this uh internship i was working on scene text recognition on the jetson uh xavier agx board uh and essentially i can go over what scene text recognition means what it's uh referring to is if you have a picture or an image of a scene and you see words in that image the program is going to extract those words and display them for you so if you've ever heard of ocr or optical character recognition that's commonly used to take scanned images and extract the text you know so you can edit them and that's similar to what's going on here with scene text recognition but the only difference is it's text that occurs in more natural scenes so for example reading signs like street signs or reading labels on shelves or um examples like those so those are instances where scene text recognition could be useful uh so uh how my internship started really was a lot of background research and understanding you know what goes into this recognition what goes into being able to extract text from images and uh what i found out was there are two main steps in the process the first is the detection and then there's the recognition so for the detection step uh a deep neural network will extract where the text is located in an image and give it a bounding box so in that it's similar to most no object uh detection algorithm object detection neural networks that find bounding boxes or different different parts of an image so right now the detection essentially outputs bounding boxes for the text in the image and then the second stage of the whole pipeline is the recognition part and in the recognition part the a separate neural network takes those cropped images from those bounding boxes and outputs the text from those and that's more of a simple like classification type neural network that works on doing that so putting both of these together you can get a really good scene text recognition system working and that's kind of what my project was oriented around here um and then another major part of this project that i was working on is as michaelis just mentioned we often focus here on nvidia on optimization and speed up so i was using tensor rt to get these existing algorithms to work really well and fast on the jetson agx xavier board and i used the torch to trt repo for that and a lot of tensor rt api code as well in order to get that running so i also have a little demo here that i can run and share for you as a video give me one second okay do you see the screen up and let me play it so as you can see here right now the detection is working on this image on this video and it's creating these bounding boxes and then the recognition here is taking those bounding boxes and opening text you know um you can see it's not perfect every time especially while it takes time for the camera to focus but eventually it does get the correct words and you can see throughout this video i'm just holding up different objects and having it read what's on the object and especially for these crowded ones and i think there's some spanish words in here it gets a little confused but over our all it's doing pretty well and one thing that's not shown here is that the code can also uh it tells you the confidence score for each of the words it's identifying so we can always filter out you know if it if it's not confident that this this is the correct word i can we can filter that out as well i think this one it performs very well because it's pretty clear in plain text and you can see it gets it almost exactly right nvidia and jetsonite xavier developer kits quick start guide um that's awesome yeah this one's interesting i thought i would give a fail case so when you put it vertically it doesn't get it doesn't show you the text so um that's something i think can be extended with you know this work i'm doing i think the whole idea of the code i'm working on is hopefully after next week by the end of my internship it can be open sourced as well and people can swap in and out the detection and recognition algorithm so you know right now i might not work on vertical text but if you train um a detection and recognition algorithm to do that i think it would be able to at some point so how did this actually work does it detect individual characters or is it just like one model that you know spits out all of these words or yeah so there are some scene text recognition that tries to create one model to spit out these words but i think it works much better when you segment it into two separate models uh you have the detection and the recognition so the detection model takes a frame and it's giving these bounding boxes it outputs like the pixel locations of those bounding boxes and that's what i'm displaying here in the image that's one neural network and then the recognition neural network takes the image takes like the cropped image like just what's inside of this bounding box and then outputs the characters so it's actually two two separate steps that's doing this and i think that that works the best and it's more modular as well so you can swap out either either detection or recognition you know and test performances with various models that makes a lot of sense yeah and so one thing that um came across my path when i was working on asr or speech recognition was that at the end of the asr pipeline it typically has a lot of misspellings in it but then you use like a language model that essentially classifies the outputs and gives you the closest english words so i was wondering if like in your research that you seen anybody using these language models to get you know correct misspellings or improve the accuracy yeah actually in the research i've seen uh i don't think they have used language models to improve the accuracy but i do think that's an extension especially when um we're looking at the practical applications now so okay let me stop sharing for a second just speak so i think in in like the research community when when you're when they're running these scene text recognition models they have set benchmarks and data sets that you run the models through and get output text and uh they can you know determine the accuracy and compare state-of-the-art models and i think in that case they're just trying to focus on the recognition algorithm but when it becomes when it comes to practical applications i think you're right that it'd be a good idea to maybe have some language model at the end and um being able to convert because as you saw in the demo video you know not all the words are perfect there are some misspellings so having that extra step i think would really help in that application awesome yeah so you had mentioned that you are an incoming um phd student at university of illinois right do you have any ideas about what you might want to study or do for your thesis i do so i did undergraduate here at the university of illinois as well that's why i am right now in my apartment and i think uh what i was i'm one of the projects i'm continuing on is in the human centered economy lab here and it's uh the project is has to deal with human robot interaction and a lot of the projects in the lab are um about autonomous systems and robotics and i'm finishing up a user study dealing with uh like humans being able to speak and communicate with robots and seeing how humans perceive that interaction so that's an interesting project that i'm working on in that lab right now and then another lab i'm looking at works more with hardware acceleration and machine learning algorithms and um heterogeneous computing so a lot of like fpga hardware and trying to create an experiment with you know architectures that can help machine learning and ai run faster so those are the two research areas i'm looking into right now as i start my phd awesome well we wish you the the best of luck um with your program there and um yeah look forward to seeing and reading any research that you come out with in the future thank you okay so next we're going to talk with rashab and nasa and these two are interesting because they were interns last year and they are now full-time employees at nvidia so you know welcome aboard again it's great to have you guys back and you know formerly part of the team full time and uh it's been great working with the the both of you since you've come back uh to the team um so first uh rashad is going to talk about what he worked on um during his internship and uh then we'll take it from there yep thanks so much dusty here glad to be back so i'm going to talk a little about the projects i worked on last fall during my internship and they were sort of split into two parts so just give me a second so i might share my screen can you guys see my screen can you guys see my screen um yeah sorry wrong screen there we go okay yep yep no problem so yeah uh hi everyone so i worked on maybe two projects uh during my internship at nvidia like the first one was uh basically developing roster packages for deep learning related applications on nvidia jetson and it was the motivation behind that was mainly to enable like roboticists who getting started with ai to sort of use the jetson series of products and use roster with it in their own entire pipeline and the second one was pretty interesting as well it was on validation of the cemeterial bridge uh using a simulation platform called isam which nvidia developed internally so uh yeah so the first project which i just told you about is like the roster uh based project for airline applications on jetson what the main objective over here was to sort of develop packages for classification and detection of objects uh with the roster framework so i'm not sure everyone may be familiar with what the ros framework usually is so if you for example working with a robot and you know like you've got multiple sensors you've got a lot of processing which is going with the robot could sort of maybe localizing itself in an environment using camera information lidar information and wheel encoder ticks so there's a lot of things going on so usually debugging sort of very big applications in which there's a lot of sensor input and output gets complex so the raw system proposes it sort of you distribute all these problems into separate code bases which you call nodes and you allow these nodes to sort of communicate with each other over topics and images topics like images so the main objective which i sort of undertook over here was given like an image stream coming in from maybe a camera or from a file can you have like an object detection and classification note built up and draws which can let you do these functionalities here so i worked on four packages the first two was a simple classification and detection and the next two were accelerated classification and detection using torch to trt and that used that use like the tensor rt runtime environment and engine in the back end which really gave us a good speed up in terms of throughput and reduce the latency so you sort of get able to get real-time performance and it was pretty cool and yep so the next part of my project was mainly on the side of validating the sim to real bridge so it was really interesting because the main idea behind this was can you sort of gather data in a synthetic environment in a simulated environment and then can you train a model and check if you can actually deploy this model in the real world and sort of how well it performs so over here i use the platform called isaac sim which is a nvidia's own platform for a simulation and i was blown away by the photorealism and the amazing quality and attention to fine detail in this platform so towards my screen at the center sort of you can see uh the simulation environment which has been set up by me in the omniverse kit and that there's a description of all the physics all the lighting texture and all the other aspects in your simulation too which is towards the right so i'm going to sort of just show a quick video of how an important component called domain randomization and synthetic data generator actually work you can actually acquire data from a camera perspective on the chatbot and you can then save that data and train a model on it so this is actually the perspective of the chatbot which you're looking at over here and okay so yeah so the perspective of the jet board is what allows you to sort of get a lot of synthetic data i think there was a problem with the video but in the blog as well i can show you what the synthetic data acquisition looked like so you just sort of import your jetbot into this isaac sim environment with the other objects which you want to add to the scene you then correct your physics as well which you want the objects to exhibit and then you can just sort of add domain randomization and collect data and then you can train your own model so the next step actually is to sort of deploy your model in the real world so what we did here we used the train model on synthetic data and we wanted to actually deploy it on a chatbot in the physical world so yeah and these are like a few samples which we actually got from isam so towards the left we have like so the the left images and the images at the center are sort of images and simulations so you can actually see the quality of what objects you can get like it's pretty clear what features you're looking at and it's very photorealistic and towards the right is actual real world data which we acquired with the chat board so some of you might have noticed like there's a pinkish tinge in some of the images in the real world and you can actually add that as uh compensation and domain randomization even while getting synthetic data so that was pretty cool and so this is a video of that a trained model on synthetic data actually being deployed in the real world and this is what it looked like so you can see like the jetboil sort of recognizes the objects and accordingly avoids the collision with them and it was pretty cool because it's only seen data in the synthetic world and the environment to which it was trained on is really different from where it has been deployed so that sort of shows you that how well it's learned the features so rashaab do you what aspects of the simulation do you feel are important for the sim to real transition to work sure yeah that's that's a really good question dusty so i think one of the fours uh the most important aspects which we initially figured out was we didn't even use the component called like the domain randomization we sort over trying to fix objects in the scene itself and just see if we could build the model so what happens in that case is there's no look they're not a lot of variation and features for which the model can sort of generalize well in the real world so that played a critical role like changes in lighting texture and object scales so those were very important and so this was i think this this made a lot of sense in general that it's it's pretty common sense that you would want at least a good similarity between the real world and your simulation but a few other features which we found out which were important like so when you look at the objects in terms of the chatbots perspective you want to make sure that your camera calibration is done correctly in simulation as well as in reality so in case you don't do that then you do get some inconsistencies in what the chatbot perceives the environment to be so i think the photorealism which isaacson provided along with synthetic data and domain randomization in the environments were very important for sort of the centurial bridge to actually work uh in case you use like in a simulation environment like gazebo which is popular in robotics it's not even close to what photorealistic and attention to quality isasm provides so you sort of can't expect to get results like that in any case so yeah i've used gazebo in the past and you're exactly right graphics are subpar although that one does run on the jetson yep so yeah if you have an rtx card the isaac sim is all the way yeah and these are the sort of public levels which are already available for the workout completed so this is like the roster note for object detection and it's pretty much real-time as you can see in the image right here and also we had blogs uh which we published along with all the work so it's there for the uh isaac sim work which we performed as well so here you can sort of it's a step-by-step guide literally to actually import the chatbot into an environment build and simulate objects and profile provide randomization too so so yeah i think that was it from for me in terms of my project the most interesting parts during my internship were obviously getting to interact with many teams because i think the people at nvidia are really smart and when we came in from sort of academics we were learning about deep learning machine learning robotics it had been a lot of focus on training on understanding what the concepts are in terms of theory but we got the chance to actually understand the problems related to deploying neural networks which was we could really gain more appreciation for that like how do you construct the entire pipeline and to end and work on it and uh my two cents for anyone who sort of wanted wants to get started on robotics and ai in general is things do tend to be pretty complicated i think you you might get over when looking at raw scent like deep learning there's so much going on so uh pick up a simple online course which will give you sort of the basics and try to work on a simple project to as soon as possible because doing a project and sort of seeing it working is very important for you to sit sort of motivated throughout the way and you can build on it and you'll surely make progress that way so yeah thanks a lot guys let me know in case you have any questions awesome thanks thanks for sharing it's been great working with you since you uh started back here thanks again okay last but not least we have asa up next to uh talk about what she's been working on thanks for being patient and uh joining us here asa hi thanks dusty um so before i jump into what i worked on during my internship just wanted to spend a minute talking about the transition for me from being an intern last fall to joining full-time this fall um so after i finished my it was a long internship about four months it was a semester long thing and that ended last december i had a semester of school left so i went back to georgia tech finished my master's and um graduated this may and so the transition for me because it was part of the same team i was joining the same team i knew the people i knew the kind of work they're doing and the kind of work i would be interested in doing so it was pretty smooth in that sense and um because it was an online internship uh during covert last year i had been sent everything i needed so i had