Building Augmented Reality Experiences with Unity3D with Abhishek Singh, Resident Researcher at NYU

Key Takeaways

you how did you get e e e e e e com I'll say e e e e spe e e [Music] for that's for [Music] [Music] hi I'm abishek and I've recreated the iconic first level of Super Mario Brothers as a firsters life-size augmented reality game that I'm now going to play as Mario [Music] [Music] that just waiting for the live thank you all so much for coming we're really excited about today's talk uh cs50 is you may know has been experimenting in the world of virtual reality or VR over the past year and we've had our eye on augmented reality or ar which you might know in some form from like Pokemon go if like a year ago you were playing that quite a bit um our friend abishek here though from NYU who recently finished up there has been working as a resident researcher truly took things to the next level uh he went viral recently as you may have seen with his implementation of Super Mario Brothers in augmented reality and while perhaps a bit socially awkward spent quite a bit of time walking down a path uh in Central Park creating this amazing world that he's kindly offered to share with us today so abishek Singh thank you hi everyone um I'm abishek and let me just kick these things off I mean David's already done an INT introduction but let me just um go over it again uh so I graduated last year from um NYU IDP and ITP is the interactive telecommunications program um at NYU it's kind of like a mix of a design and Tech and art kind of program and that's where I kind of uh started doing a bunch of crazy odd slightly off kind of projects uh one of them was uh this shoe and this shoe is something that you can wear and you can uh control and customize completely from your phone another thing that it did was um an interactive installation in Time Square uh so this went up on a 40t billboard uh in Time Square you could text in what outraged you on the internet that day um I also built this 24ot Dragon uh that you could sit on and ride and that would I by by riding it you were controlling a dragon a virtual Recreation of this dragon in a in a virtual world as well and um following that I also made this little desktop assistant um and this is a desktop assistant that responds to your voice but he responds entirely through gifts um um and this actually was the project that went while uh prior to this current project uh and yet here I am today uh talking to you guys about AR so how exactly did that happen well David already revealed that but um I'm sure all of you are familiar with this like this might have been created prior to most of us being born but at the same time we all recognize it and we all kind of grew up and played it at least once right um and this is Super Mario Brothers and what I did was I recreated this except I recreated did it um in AR as a life-size uh firsters game so I took the iconic first level of Super Mario Brothers and rebuilt it uh for the hollow lens and that's what I'm here to kind of talk to you about today um in addition to my experiences of building this game I also want to just talk to you about augmented reality in general and also some of the tools at least a cursory look at some of the tools that are available to us to start building um stuff in AR so um AR is essentially taking virtual objects and placing them into the real world but I say not just placing them into the real world but also placing them with them having knowledge of what the world around them is knowledge of surfaces and maybe being able to even interact with them and to be honest AR is quite pervasive already right um we all know about Snapchat filters and we use them like on a regular basis and this is AR to some extent it's augmenting a reality and Snapchat is pushing this further now so they've taken that and they pushed it to what they're calling World lens in which their stickers or these 3D objects are actually aware of where they're placed positionally in the world and apple recently made a huge announcement um and they released AR kit like opening up um hopefully to what will lead much larger adoption amongst consumers um and all this comes under under a subcategory or a subset of AR which is called marculus AR so marculus AR is essentially what is based on a technology called slam and slam is simultaneous location and mapping so what it's doing in this is using the devices camera but it's using the devices camera to essentially figure out unique spots surfaces and planes within the environment once it figures that out in real time it can then move around pan around the objects and keep them fixed fixed and anchored to their actual location compare marus to the other thing which is marker based AR and it's not QR codes but it's kind of QR codes on steroids and QR codes on steroids is something which is being used by this company vioria and it uses similar except it's less computationally intensive and it's less of a tech technological challenge what it does is it takes a unique image that you can upload so you can upload any unique image and it uses figures out points within that image and using the distances and the positions between that points it can find out the angle of um the angle of the image and accordingly figure out how it needs to orient your hologram or your virtual object on top of that image so that marker based image now um David mentioned this uh earlier Pokémon go I haven't mentioned it yet and the reason I haven't mentioned it yet is because when Pokémon Go released that is a video this is what they showed right this looks like AR like the objects are there they're in the scene they're interacting with it but when the app actually launched this is what you got right and um this to me is not true AR the reason is not true AR is because the Holograms or the virtual objects have no idea and they have of of the environment around them they have no idea of surfaces they don't have no idea of objects and they can cannot interact with it in any way it's simply just a screen with a sticker overlaid onto it right uh but all of these still come under the category of what is mobile a what Apple uh Facebook and now and obviously Snapchat for some time have been pushing if you compare and mobile AR is essentially your mobile headset or your mobile handset is a window into the world so you point your handset at either a surface or at a particular marker and you see a hologram in its place on the other hand um as compared to mobile you have these headsets and Microsoft is obviously pushing a lot in this and is probably got the most sophisticated headset out there right now which is called the hollow lens and it they call it holographic Computing it's a headset that you wear on your head um and you can see objects overlaid in the environment uh it has since it's a completely new form of computing it has completely different interactions we all used to mobile interactions swipe tap all those things but for the hollow lens it uses these four forms it uses gaze which is essentially where you're looking uh it uses gestures and these are specific gestures that I'll just cover uh shortly it uses voice uh similar to Siri um and it also has a clicker that comes along with it which can double up as a mouse um and interact with objects that you're looking at so the two gestures that the Holland supports um is the air tab the air tab is you make a tap and you just kind of tap in front of it and this is similar to what a mouse click would do and the other one is the bloom and the bloom gesture is just like the home button on your iPhone what wherever you are it will take you back to the home screen but what the holliness does really well is spatial mapping so it has a bunch of cameras four cameras to be precise and a depth sensor which is similar to what the Microsoft had in the connect which you can then look around in your environment to map it out um and so it knows where surfaces exist where objects exist so that you can actually place your holograms on those surfaces or have them interact with those surfaces in real time the only issue uh I mean one of the several issues and this is obviously a very early um I would say prototype or development kit of a headset is that it has an extremely narrow field of view so it's not as immersive as if you have tried virtual reality it's not as immersive as that but it's definitely a big step towards having these headsets and for me headsets is definitely something which is not probably going to happen in the next couple of years but maybe five or seven years from now just because it keeps your hands completely free so mobile is good for uh stepping stone into AR you headset so uh after all this now that you have a sense of what the AR kind of space looks like what the tools are what the different terminology is uh let me Begin by how I got to making Mario and it actually all started with this one simple Cube up here so um now you must be wondering how I started with this Cube and I'll get to that so uh it when I got an access um to hollow lens and that's the great thing about universities is you get access to technology and equipment that otherwise you possibly couldn't afford um I I already knew that what I wanted to create was a large expansive Outdoor Experience the hollow lens and the Microsoft uh FAQs they specifically say it's meant for indoor use and it is not U meant to be used outdoor and meant to be used in kind of small quarters but I always saw the opportunity of AR to be something that you can use Outdoors something that you can move through something which may have a fitness element to it something that um that would actually take technology out from within the rooms um and into the real world so I decided that I'm going to create some sort of Street Runner game right I would create a game that would exist in the streets of New York and I would run through it uh completing it and probably possibly record it just for my own fun while I was learning Hollow lens development and everything starts with like either a bouncing ball or you have like a cube that you place and there the basics the absolute Basics I took this wi Cube and I kind of moved it around in the space I moved it around in 3D space in the hollow lens except during one of those tests I placed it slightly above my head and for some unknown reason I stepped under it and I jumped and as soon as I jumped um all the memories from my childhood came rushing back and I was like this is Super Mario and this is this white cube is the brick that I'm supposed to uh that I'm supposed to like hit with my head and have a mushroom come out of uh so so starting with that I was like Super Mario solves everything that I was going for I wanted to create a large experience I wanted it to be outdoor and it added the additional thing that it now had a purpose like what my game was lacking I did not know what the purpose of my game was but Super Mario had a purpose you the player is moving through it to reach the end to complete the level and in addition to that it has this huge Legacy behind it right everybody it's immediately relatable and I knew that it had viral potential so um the first step was a ton of research and a ton of research meant what watching the video on Loop over and over again till I practically memorized every single interaction and every single element on that it also meant going back and playing the game something which I hadn't done in in several years to actually get um a sense of all the game mechanics after I had done that um I was the next step was recreating in 3D and the first step to recreate in 3D was to get an handle of the assets I was I thought this would be pretty simple Super Mario's been there for 30 years people have definitely created 3D asset that they could easily download for free online um and include them into my game but that wasn't the case it wasn't the aesthetic that I matched and it was priced at a ridiculous $5 it's now at a discount um so uh it made more sense to instead spend one week building it myself um well at least it made it it made more sense at the time in retrospect probably $5 would have saved me a lot of time um uh so um I I decided um that I'm going to um design all the elements myself so I picked Autodesk Fusion 360 which is like a cad uh program and I I started designing these elements the only problem was that I have only been used to seeing the elements in this way I don't know how they look in 3D I don't even know what the underside of this brick looks like so I had to reimagine and rethink all these elements that how they would grow in 3D so turns out I decided that this is how the the Brick would look instead and this is how the underside of the brick would look and then going back I found 2D cutouts of all the elements and then recreated them in 3D so this is the Goomba which for the longest time I used to call bad mushroom um and uh this is the Koopa Trooper which I obviously always call bad turtle um so once I had all the assets um built out the next step was bringing it into the game G engine of my choice and the game engine of my choice is Unity not only because of my familiarity with it but also because it's the one that supports holodance development and has an SDK which allows you to build and deploy to the hollowland so just a quick segue into Unity for those who are not um used to it or not familiar with it Unity is essentially a game development engine which has Dem democratized the creation of 2D and 3D games for developers everywhere uh it's pretty simple when you think of it you create an object and you attach a script to it and that script allows you to give that object a behavior add interactions or add animations of any sort and you can script it in multiple languages cop JavaScript Unity the reason I've listed them in this order is because this is the popularity of the community everybody kind of prefers cop and you'll also find many more links and many more um help tutorials on that so once I brought it into uh Unity The Next Step was painstakingly recreating the entire stage right entire stage but not only the entire stage but each and every single game mechanic and now when I think of it and when I look back I'm like truly amazed of how these programmers were able to do this in like 1985 when there was no Google there was no stack Overflow and you had to basically figure all these things out yourself um so um building the level took some time and at the end of it I was able to recreate this entire level so what you're seeing here is the unity interface and you can see the level within it um but it's not in 2D as you see I I kind of scroll around all these elements were then recreated in uh in 3D and I I built this to scale so I built this to human scale so the final length of the entire level was about 110 M long which means I needed a place which was at least 110 M long to be able to play it myself um there obviously several challenges along the way um and what I like to call several hacks or programmatic illusions that I had to undertake to actually bring Mario to life because Mario after all is a video game right so he can defy the laws of physics which are not necessarily possible for me yet um so so one of the things was he can jump twice his height I I can't do that and at the same time uh he can collide with bricks and I can't collide with Holograms uh so how do I kind of work my way around that uh so do to do that just to kind of get an understand understanding is that the hollow lens knows its position in space so the hollow lens is placed on my head it kind of knows where it is located in space so I attached a collider to the hollow lens starting from my head and moving down to what my height is and I attached the collider to every brick 3D brick in the scene except the bricks had to be much higher uh to give the same effect so I just offset the collider from the bricks so brought the collider down the Collider ERS are invisible they're not visible to anybody but by jumping just a few inches off the ground I could still collide with them and have the same interaction right so now you can see bricks are much higher than I could possibly reach but I'm still able to jump and interact with them and get the coin out of it the next thing was spontaneously growing when you eat a mushroom so yes I can't spontaneously grow either uh so how how did I achieve that instead of me growing I decided idea that everything around me would shrink so um that would give at least the illusion that I had now grown um the other thing that you see um in in Mario is there's several visual cues V visual cues interspersed throughout the game as soon as he gets a power up his costume changes um and he has this kind of blinking effect so I couldn't figure out how to do the costume change uh but I did figure out how to do the blinking effect okay as soon as you hit the uh the power up you see kind of this overlay which mimics the blinking effect so you know you've got the thing but um stripping and immediately changing in the middle of Central Park was little bit off limits um and and finally the other thing is that how do I interact with something that doesn't really exist right the Holograms have no physical uh I mean they have nothing there is no material they just completely uh creations of light um and he's supposed to jump around uh jump on these Holograms jump on these pipes so I found a really simple fix to this I was working in 3D so I decided to just walk around them um um and then moving on like these were just what I like to call programmatic Illusions but in addition to that there were also a bunch of hacks and a bunch of hacks um with the to tweak the hollow lens to work over such a large expanse the hollow lens is meant for indor use it's meant to be used within small quarters it's not meant to be used Outdoors on a flat surface which has very little differentiation on it like a Road 5 m from here looks exactly the same as Road 5 m behind so it's very difficult for the hollowland to figure out what uh it saw or what it's going to see um and at the same time it was outdoors and it's over 110 M expanse uh so to achieve that um I had to do a number of hacks but the first thing before reaching any of that I realized was that what if someone else wants to play this game what if someone else who is not as tall as me or who is much taller than me wants to play this game everything would be lower for them or higher for them and they would not be able to complete it so the first step what I did was I integrated this calibrator of sorts as soon as you start the game you first enter your height in centimeters and once you enter your height in centimeters it calibrates the level to your particular height so if you really short the bricks will be closer to you and if you really tall the bricks will be further Above So irrespective of whatever your physical build is you will be able to complete the entire level um after that I went about like scouting a location and in scouting location I I did make a mistake I spent too much time developing indoors in my computer rather than going and testing outdoors in the actual location that I was going to shoot so I picked Central Park and I picked Central Park this area is called the mall in Central Park and it's the only straight stretch of land um in Central Park and probably in New York which is so empty that I could actually go and uh play this without being run over by a car or arrested by someone uh so um so I I picked this place and as soon as I went to the place and the first thing I I realized was that the Holograms were much more transparent than when I was filming them indoors and that's the reason why Microsoft also recommends that you film uh that that you use the Holograms and use the hollow lens indoors the reason is that indoors you can control the lighting the lighting is more consistent it's not as harsh as sunlight uh so the Holograms which are based essentially on playing with light are much more opaque to get around that um I again found a programmatic hack instead of um capturing the video directly through the through the hollow lens I did capture the video directly through the Hol lens but instead of capturing through the hollow lens Hardware itself I captured it from within my app running on the hollowland so I wrote a custom script that would allow me to increase the opacity of the uh hologram as they were being captured so while this is the exact View and the reason why I have added these translucent um translucent squares on the edges is that this is the actual field of view of the hollow lens what you're seeing and and the rest of it is what is captured by the hollow lens cameras but it's not what is visible from within the headset U so I figured out how to get the Holograms to be completely opaque which was extremely um necessary for me at least just just to show the potential of AR and just to get the point across because it was much more U compelling seeing it this way than seeing a translucent washed out hologram instead but every action has some kind of reaction and the result of this was that the Hologram uh the holand sees everything that is black as transparent so so that's how um everything that gets is black gets overlaid with the real world so I had created these pits right the pits that you remember in the stage that if you fall into you kind of die I created these pits and I created the exterior to be completely black so that they would be completely transparent to the hollow lens but as a result of writing my own Uh custom script to capture this the Holograms were now being rendered in my final capture as completely black so even though in the hollow lens while I was walking around everything looked fine in the final video capture it was looking like one gigantic black Cube embedded into the floor uh so to find a way around this I tried two things um both were writing custom shaders so Shader is essentially like a program which gives you on a pixel or vertex level control so you can Define how every pixel is rendered U on onto the screen so the first thing that I did was I wrote a I I wrote a a custom Shader that essentially what it did was it made the black transparent to the camera but opaque to everything behind it so as a result of that I could not see anything unless I stepped right over it but the problem with this was that it was extremely uh computationally intensive and as a result of that the final capture was extremely jittery and there was too much drift and lag while I was moving through the stage so instead of that I used something called a stencil buffer and a stencil buffer is some uh is what you can consider to be a mask so everything that you see through that mask is visible and if you don't look through that mask it's not visible um and the resulting effect is something like this so you can walk up to the pit and you can look down and it looks completely overlaid onto the floor and the stencil mask or the stencil buffer is overlaid right on top of it so it's like a window into this object but it's only when I look through this window that I can see anything that's um inside it all that was fine but as a result of not testing Outdoors enough uh the the other issue that arose was that there was a lot of drift and when I mean by a lot of of drift was the entire level as I moved further into it was shifting tremendously to the left or the right or up or down and you can kind of think of that as a pivot right if you if you look at my elbow and if you think of my elbow as a pivot as I move my hand everything that's further away moves moves more than everything that's closer to the pivot and the result of uh and the issue was that I had one single pivot I had one single pivot where the stage was initialized so any movement any recalculation that the that the hollow lens did to figure out where it was positioned in space shifted everything further into the stage by a factor of five uh so to get around that I had to do the biggest code rewrite ever practically go back to scratch and split the stage that had now coded as one gigantic stage into small sections into small chunks so I took the entire thing and split it into 23 sections right and each section had a spatial anchor and a spatial anchor is something that the Hollin can use to Anchor a hologram in space so each of these cubes that you see on the screen over here um was was used as visual cues by me to signify the start and end of a section and there was an anchor which is obviously invisible completely based in the center of each section and by placing the first anchor so I would just place the first anchor on the floor and then place every subsequent anchor completely perfectly aligned to it following that so at the end of it it each even though the entire stage was working as one cohesive hole it was actually made up of 23 different parts each part being anchored completely uh independent of each other and that helped get rid of the drift completely and also helped it stay level on the floor once all these kind of issues were sorted out and there were a bunch of other things which if I get into we'll probably be sitting here all day um um I decided that it was time it was time to finally film this and put it out there but before doing that I obviously had to dress up as Mario because the entire entire point of this was that it's a firsters game in which you step into the shoes of Mario so you are Mario so I had to be Mario so um again it was kind of a do-it-yourself kind of outfit I purchased the uh overalls from Amazon I had a full sleeve uh overalls and the gloves were from Amazon I built the buttons the yellow buttons out of wood uh just kind of tap them to myself and I uh luckily had a full sleeve red t-shirt um so after that I went ahead to Central Park went early in the morning because um one it's less crowded right I would get fewer well weirded out looks um and at the same uh at the same time the light was uh less harsh so as the sunlight gets uh harsher and harsher uh it's worse for the Holograms to perfect itself so uh let's just have a look because it was not just about the Holograms looking perfectly it was also about the audio working perfectly a lot of the things like I cannot physically feel a hologram so I had to rely a lot on audio cues to know that I had performed the correct task or the correct event had kind of occurred so Mario obviously has each and every um asset file of their audio available online so I just use those and Trigger them at perfectly the perfectly timed events and you can see all of these you're used to in the Mario game but it was not just about what we are used to see in Mario right the other thing is like what we're not used to seeing in Mario and one of the things that I always wondered was what was inside the castle at the end but now that I was building it myself I was like okay now we can actually see what's inside this castle at the end um and since it's like a 3D Castle you actually get to walk into it and look up uh so at the end of the stage once you uh clear the level unfortunately I couldn't slide down that flag um you simply step into the castle and the castle is a life size it completely occludes everything around it and you can see that not only does it uh set of the flag but you can look out to the window Windows you can look out through the roof and you can look out through the door um so I kind of had this headline in my head when I was like oh this has kind of viral potential um the end effect looks really good and probably someone is going to post this saying like a crazy dude in Mario costume seen running through Central Park Early in the Morning um but um but I kind of underestimated New York I did get a headline but it was no one thought it was weird um but but I beg to differ because I was able to then zoom in on someone and I did get a bunch of looks uh so U there were definitely a lot of confused looks but then again it's New York so people have definitely seen things weirder than U me jumping around early in the morning in uh Central Park um and it led to a lot of interesting conversations a lot uh led to a lot of people just kind of taking pictures fig out what exactly was happening cuz most people had not seen a hollow lens before and most people were not familiar with augmented reality or virtual reality so there was a lot of um I guess interest by seeing what I was doing there early in the morning um once I shot this and I shot this on a Tuesday I posted it out on Wednesday by starting to post it on Reddit um and that's when it kind of went viral so it didn't go viral immediately Wednesday I posted on Reddit it got deleted for some uh reason well I didn't delete it but apparently I was violating some rule so uh it got and I I thought it's kind of over I said I'll try again next week but somehow it got picked up by a couple of uh sites um and I tweeted out to Hollens I tweeted out to Holland saying hey check out what I built um never heard anything back from them um and went went to sleep um I woke up next morning and the video had completely skyrocketed um the Tweet was now at like 5,000 uh retweets and everybody across the internet was somehow talking about this thing right uh I woke up to 30 emails all asking for permission to use this and by the end of it it was covered in almost all major Publications uh across the internet and from it was not only about recreating Super Mario in AR right for me also it was about just generating this interest and getting conversation started and having people not only talk about what is possible now but what will possibly be possible five or 6 years from now and and that's what it kind of led to it led to a lot of interesting conversations when people seeing is this the future of gaming is can AR now be used Outdoors because that's something that has not been tried before can AR be used for Fitness activities will these headset 5 10 will will these large expansive outdoor experiences become the norm and many people are now questioning do now do I now have to deal with weird guys jumping around the place uh and not looking down at their phone uh they already got a taste of that with Pokémon go but except now you'll be with something probably mounted on your head um and another thing that I always get asked a lot is what is my opinion on AR versus uh VR um I have worked with VR and I have worked with ar personally my opinion is that AR has a lot more immediate potential and AR is possibly going to be a stepping stone maybe to VR and one of the main reasons that I feel AR um is going to have much Mass adoptions not only because a lot of companies are supporting it currently but more importantly uh is the user experience that it provides VR demands full immersion right it asks to cut you out of what you're doing you have to wear this headset you're completely occluded from the world around you whereas AR just allows overlays it onto the real world so it it de demands less effort from the user which is why I feel it's a better stepping stone and a better stepping stone rather than VR and um that's basically about it so if you have any questions I would love to kind of take them now y uh so I have two questions so the first one how long did it take you to build it and the second one are you planning on building other games I've I've got both those questions very often uh so it took me about a month to build it um in including uh building out the three the 3D assets and all those things it was not it was a Sleepless month but let me put it that way um and um in addition um to that if I'm going to build more games I'm not too sure um I I was thinking I might do maybe like another level of Super Mario one of the underground levels in the Subway or something like that um but at the same time there's been also like a lot of interest um of like recreating like Doom or one of the older U older classic games but but I I do have a feeling that there'll be a lot of developers who will now start working on it themselves yeah yep yeah did you action AR in action have you heard I have not oh good pleas okay um I want to invite you um and actually have it's something for the second question um AR in action has people using AR in every field the inal one um the MIT media lab January we had 800 people participate and they came from all over the world from Tokyo doing things like you know dinosaurs coming out of TVs construction awesome and it had in June also if you can't make well this one pass having one again on January great only use them indoors yeah that is a great thing in terms of bringing it another thing besides the holl my friend who also started they have thing called The Meta heard I have ch yes I haven't had a chance to play with it I don't have access to one yet okay yeah just yeah thank you yep so in terms of making the more real were you able to and does the hollow lens support this at all get a sense of where the light is coming from and you talked about like the strength of the light either to give like a shadow of the object or also to give like a little like glare right on the surface of the object like you know the sun is shining on um so it doesn't know where environmental light is coming from but you can uh uh put programmatic lights which simulate the environment to give you those kind of Reflections and glares yeah yep it's a completely selfcontained so the hollowland has the power source inbuilt into it um I have used it extensively and it lasts about maybe two two and a half hours um which is not that bad yeah yeah have you started playing with ar ar kit AR kit yet uh I haven't started playing with AR kit yet because I don't have have access to an iPhone so so yeah once I get an iPhone or if someone gives me an iPhone hint hint hint um then I will start playing with the arit Yeah you mentioned multiplayer yes that to work um well multiplayer essentially I would have to dress up as Luigi and play the game again yeah yep how mat would you say the tool software tools are right now you to do things not ready um I I think Unity 3D like the unity engine is amazing um and you can do like a lot with it but at the same time perhaps the Hance documentation is not the best yet um it's not put together yet though they do have a forum which is relatively active and they're quite helpful over there so it did require me to go to the Forum several times and dig around in different places to get everything to work but Unity itself is quite a mature uh platform yes you mentioned computational intensiveness in your programing how like intensive was the time of your final product okay I forgot completely I should have been repeating these questions so um so he asked what is the computational intensiveness of running these programs um this was not computationally too um intensive the hollow holand says that it can render up to 100,000 uh polygons without any problem um and especially in the case of Mario because it has this pixelated kind of boxy look uh so the assets themselves were not very heavy yeah so this ran without any problem yes nintend reach out to you after is out okay did Nintendo reach out to me after this is out no they haven't reached out to me yet um a lot of people have said that they will only with a season desist so um um but that hasn't happened yet I I I don't know if I'm happy or I'm kind of bummed out but yes has an option where you can suspend the laws of physics did you play with that yeah uh so the question is uh Unity has an option to suspend the laws of physics um so I did play with that uh the thing with ar as compared to VR is that AR still has the real world anchored around you so it doesn't create the illusion um it doesn't create a good enough illusion to play with gravity cuz there were some suggestions that for me to climb up the pipes maybe I could lower the entire level as I was jumping but because I would my I would also be seeing the real world anchored where it is it would not be a good effect at all yeah yeah so from the bio you had your entreprene okay um from my bio what have been my past businesses uh so so one of them yes was related to VR and AR um it's called surround um and essentially it is a platform to create and share uh Live 360 augmented videos so you can take um 360 video 360 video is video that you can look around in all directions and it is captured with specific 3D cameras uh and we allow companies and Brands to live stream uh 360 distributed through a platform but at the same time augment it with elements so you can layer it with any kind of uh 3D objects 3D elements any kind of rich media images text tweets um websites and send it out to your consumers all right yeah so just building off of that not everyone can you know get to Central Park to play your game yes like would it be possible to use like 360 video to just film Central Park really well and then you know put it in in a VR situation where you have the the element add-ons from your code and you have the 360 video and then anyone around the world so long as they have the 100 meters of straight flat space can now see Central Park and can see the objects that you put into it and play the game as if they were in Central Park okay so uh the the question is um how do you transport someone in who cannot uh go to Central Park into a VR thing is there some technique to maybe recreate Central Park in VR and then integrate this game into it um so the problem capturing a 360 video is that 360 video does not have any depth so essentially it is just a sphere and you place the camera in the center of the sphere and look around in all directions so you would not be able to create the 110 M depth uh what you could do um is maybe do a really large uh photogrametry capture of uh of Central Park so photogrametry is taking pictures of an object from multiple directions um and then it uses that to kind of map and create um an actual actual 3D representation of that thing so either a huge photogrametry capture or using something like a liar sensor um to recreate the entire space import that into a VR space and then integrate your Mario game into it yeah it's possible yes okay so um applications of AR Beyond video games and possibly in educational space um yes definitely I mean for education and that's why Microsoft also has been using the hollow lens a lot is educational spaces because and shared experiences I feel like in educational experiences being able to like share an educational experience itself is really powerful so using them in classrooms um using them to kind of they examples of U interaction with the solar system right and you can't really interact with the solar system but you can take like a 3D representation of the solar system place it on the table have your students kind of uh stand around it look into it and then actually see how everything works or there similarly there are a lot of um a lot of use cases in human anatomy and human anatomy and understanding the body is something which is really cool because you can actually take the body split it up into all its different subsections walk around interact with it and see all the different elements so so definitely education like the possibilities are endless in education right okay looks like that is it