Building VR and AR experiences using HTML | HTTP 203

Key Takeaways

so is this going to be weird for you what do you mean so you've been doing this with summer for years right and um you're friends aren't you yeah but you know people leave all the time it's perfectly normal don't really think about it oh here we are welcome to the studio uh what's that oh um this isn't mine um this is something that crew use for because it helps them position for the lighting it it helps them with the shadows sure um i'm just going to move sir oh don't touch him so sorry sorry i'm just a little bit fragile i'll i'll i'll deal with it i've got him it's okay you're safe with me it's okay it's okay it's okay don't worry i got you [Music] so hello ada hi thank you for having me on your show that that is fine it's great because it's episodes i don't have to write and that's what this is going to be right yeah so i have a few things i'd like to cover about webxr today okay excellent stuff um so you're um you work on the working group for yes so there are two groups for um dealing with the um standards from webxr there's the w3c immersive web community group and the w3c immersive web working group they're basically the same name and do basically the same thing and work really close together but the community group is open up to anyone and for the working group you have to be a member of a company that's a member of the w3c cool so like usually when i do these episodes i i have some slides i have some code on the screen and that's as exciting as it gets you've brought a vr headset and you've got your phone connected to the screen and all of this sort of stuff so various toys yeah so i'm going to feel totally inadequate in the rest of my episodes this series but come on then come on then show cool xr vr stuff so yeah web xr um we have a cool logo the logo looks like the dart logo oh no i never thought about that it's supposed to be like a w and an x and it's like 3d it's a different color it's a different color that's fine it's fine um but yeah so the the the idea of the web xr device api which is a bit of a mouthful so i'll probably just call it webxr for short what what was it what's the x stand for oh that's a good question it doesn't really stand for anything in particular great perfect does the r stand for anything reality okay so we've got reality because it covers like augmented reality virtual reality mixed reality all these kinds of like general kinds of immersive hardware um we just put the x in there to be like it can be for any of those it's a placeholder i suppose yeah the main hardware we're focused on at the moment is um so virtual reality headsets and like the like the quest like i've got here um augmented reality headsets like the hololens which i can't afford um then we have um virtual reality like in handsets this is usually like inline mode in devices so this is the only mode where you can embed it in like a web page all the other modes run externally and you also have like handheld augmented reality to kind of like the thing you do on pokemon go but even though it still runs on the same display as the device itself it's still treated as an external display it just goes full screen takes over the device and you can't display like normal web content on top of it so let's let's talk about the web focus of this then how does webxr work on the web that's actually a really good question um so i have an example here and you won't be able to see this but i think the the very clever folks behind the camera will be able to superimpose this on the thing i'll make a series of noises as if i'm watching something amazing right now it will just come together in the edit really well that sounds like a really smart thing to do so here we have a normal website it has a running in the oculus browser it has a url bar tab bar you know normal browser stuff um so when you land it does take you to a 2d web page which is running a webgl scene you push the enter vr button because webxr requires a button press to start because it's kind of similar to full screen in that respect is it it takes over the whole device you want the user to be aware that that has happened because it can't yeah duplicate ui that might be privacy sensitive that kind of thing exactly okay you don't you don't want the user to feel like um they're on their secure banking website when actually they're inside evil.com which is spoofing their whole operating system looking forward to the day i do my banking in vr that's that's a terrifying thought you've got to catch the money as it flies around else it's permanently gone um but yeah that's it works you just you land on the site you push a button you're in vr so it's um if you're yeah actually let's stop the recording and take it off um so if you're um if you want to deliver a vr piece of vr or ar content then your main options are to build a specialized app for every single piece of virtual reality and augmented reality hardware that's out there where the user then has to go to the app store for your particular site find it download it and click on your icon when this beat saber right there next to it to play that instead and the alternative is um you send them a url they open up their browser they open up that url and then they share it to their headset put the headset on and then push the enter vr button and they're straight in your scene directly can i see some code can you see something yes i want to know how it works okay so i'm going to just show you um i'm going to show you a frame because a-frame is pretty cool okay webxr i mentioned it briefly before it specifically works with webgl um which as weber things go isn't great like webgl is very powerful and it's good at one thing it can draw triangles really fast um yeah every time i've tried to learn webgl i go to one of the tutorials the scroll bar becomes this size and i scroll to the bottom and it's a red triangle and like right i'm gonna go do something else yeah my my main um way of making this point is showing people the um the code for drawing a cube in webgl from in raw javascript and it's it's impossible like it's 300 lines and people don't need to know it to that level of detail so when working with webgl there's a wide variety of frameworks you can work with that simplify the process okay so these are the main ones um there's 3js which is probably the oldest and most well-known yeah it has an extremely active community and is it's the one that i like using the most and then there's a-frame which i'm going to talk about more in a sec which is really interesting because it's it's a web component wrapper for 3js so you write html and then under the hood it connects up all of the 3js parts so you get vr and ar ready webgl writing 3d scenes using markup yes now i went to university in 2003. i think you're about to mention vrml i am i did a module on vrml at university and even then it felt like it was out of date but yeah i loved it i actually really enjoyed it it's amazing how accessible being able to write it in html is i am talking about a-frame um has a really like large community like lots of people it's so simple to get into lots of people have got into it and so it's it's pretty active and this is the full hello world for a-frame like i think it's what like nine lines of code including the html and body tags nice um and it pretty much does what it says on the tin it does have a very like a svg vibe but obviously 3d yeah there's similar kind of like uh presentation attributes and yeah so like you have you include the script and then you have the a dash scene which is kind of like if we're gonna take your metaphor further it's basically like the svg tag like your a-frame tag is the a scene um that includes like a camera some lights and other things and then you start adding the other things yourself so here we have a box that's um a meter by a meter by meter by default and it's placed um one meter to the left half a meter up so it's not in the ground and minus three meters backwards forwards yeah so it's it's the words it's negative z is the direction you're looking right i see yeah yeah actually this is an interesting thing to mention so normally in graphics you don't care about units um if something's like um a a thousand meters wide and a thousand meters away a wet way that's pretty indistinguishable from something that's like one unit to you and one unit big um but here you are two meters tall precisely right okay like i see in webxr your users have a height and so and because there is this relationship you need to ensure that everything makes sense so the default unit for distance is meters and the default axis um y is up yep x is across yep yep and zed is into the scene um where negative zed is into and positive zed is behind you right got you you define it for all these things um so we have a box a sphere a cylinder and a plane um i'm not going to work out the color of the hexes in my head i guess i could you wrote this you could have known this in advance i could yes um okay no don't do it the sphere the box is cyan the sphere is red the cylinder is fc6 it's quite red um yeah that's like yellow yeah it's size cyan like yellow orangey yeah like pink pink maybe yeah and the plane is this is great well let's just do this the rest of the episode anyway i'm a great sky yes and so the sky is is an inside out sphere um right because for the texture mapping i guess yes so um 3d objects only have one side so for example the floor here is a plane and what we've done we've rotated it minus 90 degrees in the x-axis which means it flops down right but if we did it minus if we did it plus 90 degrees so it was the other way it would be invisible because it only has one side and if people have done work with 3d stuff in css the equivalent would be backface visibility where you toggle laughing right okay um and so in this case the sky is a sphere that's i think it's five kilometers big and inside out right um and that's why you can see it and so it sets a color for it nice that's what it gives you um and so we'll write a cyan then oh hang on the cylinder is yellow we were yeah okay i was yeah i can't read hex is what we've learned yeah fine yes but nice thing is you can actually use html color names oh cool um because it it just parses the color string as um as a css color and gets you the actual color so you could do laminar blush brick red um hot pink nice kind of thing another cool color um light golden rod like goldenrod yeah so yeah you can use css color names too um which is what i do most of the time because i'm lazy and i've spent way too long building html web pages before css was the thing i suppose in dev tools you could play with those values afterwards to get heard you can yes you can change the values with dev tools and it will update when you change the values in the html very good and that's the thing that would be much harder to do with like 3js yeah you don't get that yeah that instant response from playing around with dev tools yeah it's really nice and it even has a built-in inspector so if you press ctrl alt i in an a-frame scene it opens up an inspector and you can move things around and then you can copy the values into your own code nice um which is like a makes working with it very quick so the default like the the div for a frame um is the a entity okay and so the way it works is that you attach components to a entities to make it do stuff so to make a pink box we would set the geometry to box and the material to hot pink so it has its own extensions to css there is what we're seeing now like it's css like syntax primitive yes right okay um yeah so for the so components tend to um the most simplest components you can will just have a like my component is value but the moment you start to get more complicated than that you start using this css like syntax so you so for example for primitive box also has a width height and depth property you could also set to make it a more oblong shape okay um but you can um define entities to be a um you can define your own entities which map the components so for example the a dash box component has the geometry already set to be box so this is like a shortcut this you could do entirely as an entity with all of the properties yeah so it's a i guess it's a bit like you know you could create a div you can give it a roll of button you can add the stuff or just use a button and this is the sort of mapping right i see and the nice thing is that you can define your own ones of these so if there's something you use a lot with many like tens of components so a really complex thing you could just pre-define it and then you can just use like my dash component all over the place and um it'll just work brilliant it's yeah it's really nice so if you've used web components before like it's um it's a really nice way of doing it and if you poke through the source code of a-frame or you can see that it's pretty much a very friendly api and it gives you hooks directly into 3js underneath nice yeah yeah so that was kind of my level overview of a-frame excellent stuff and that's so we'll put links to all of that in the in the description yeah so people can like yeah because it's an open source thing it's free to use and yeah so if people want to get get started with webxr stuff that's a this is a nice easy way of doing it exactly yeah cool um i also want to show you um so i showed you some virtual reality earlier i just want to show you some augmented reality because it kind of shows off some of the more advanced features of what web xr can do all right so here we have a um that's not augmented reality no this is just um this is like the magic window kind of thing okay um so i pushed the augmented reality button and it's gone full screen um but notice how that uh there are still html elements over the top so this is oh yeah okay i see yeah so um normally um webxr is webgl only but we do have an api that lets you define a single element to be placed full screen over your ar scene and it works on handsets unfortunately trying to describe what going full screen with html element means in a in an augmented reality headset is a very difficult concept yes that wouldn't make sense right where where's the top yeah left bottom right stuff right okay um but here it works really well um and then the other feature we've got here so see where it's like trying to place this reticle on the floor yeah so this is um trying to work out the the 3d position of um well 3d position and rotation so i can place stuff on walls as well or the ceiling if i wanted um and it lets me place objects in 3d so i'm going to try placing it somewhere let's put it remove this bowl yeah and then we'll pop it on the table um so boom actually that's not very good because then the shadows like don't fit very well that's but it seems like it's picked up the lighting of the scene yeah so that's quite an interesting one so it does work out the direction intensity and color of the lighting is that is that the library doing that or is that built into webxr that's built into webex that's really cool um it's also showing that although the set looks very clean at this angle it's a mess the other way yes um so although it doesn't apply it to your 3d models it basically tells you all the information you need that you can do it yourself and in this case a-frame has it built in to take that information and apply it to the models nice um and then yeah gives it a shadow and it works really well so yeah one cool thing which is very subtle and hard to notice is that it doesn't just um let you know where um where you place the object when you tapped on the screen it also continues tracking that position as what's known as an anchor so an anchor in 3d space and so it keeps telling you where the anchor is so when i pan around and look over here say look at you for a bit like look around and come back my object stays where it is because it's maintained the tracking of that object nice so yeah it it's got a point in 3d space where where that stuff is and it remembers it and so because over time um the shape of the scene in the device's head or it doesn't have a head in its like memory will will evolve because it'll initially make some assumptions like i'll just assume it's in like a flat environment and the floor is this far away um but then it might realize there's a table there or there's another surface or maybe the room is bigger than it thought it was and so what it does instead of letting objects drift as they as they move further or closer to the origin as the size of the environment changes instead it works out where you intended to place the object and ensure that that anchor remains in that place even as the rest of the scene adjusts oh nice so if you if you placed multiple anchors say one on a table one on a chair and one on the floor um and 20 minutes later they should as long as we're using the um the anchors part of the web xr api they should still be in the same place um as a lot provided you haven't done something weird like move the table or start moving furniture around then better off then yeah that's fair that is fair so yeah it gives you a very stable augmented reality experience so i've had my phone pointed at this thing for the whole time so you shouldn't have seen it drift around the floor or anything like it should look pretty it's pretty good into the environment yeah and so all that together works well to give you a really decent 3d experience in the web for not like a ton of work on the developers part and so using a-frame you can do this like for ar as well as doing it for vr in theory um like it it takes a bit of thought from the developer's part but you can make something that works with augmented reality and virtual reality together so kind of like how you build a website with aggressive enhancement and responsive design in mind so progressive enhancement for when certain features are available you use them to give a better experience and responsive design you design your scene so that it can like change its shape accordingly to the size of the display you're on you're not having to build it entirely twice it's just a little bit in the middle to to deal with the differences of the entire device so that if you view it in virtual reality you can see the 3d environment around you if you load in if you open augmented reality then it hides the environment so the sky element that you had before does that automatically hide on it doesn't automatically hide but you could there is a component called hide in hide on enter ar you put it on that element so when you start the sky gets hidden yeah and so it is it is possible to build for this and then you add support for controllers so that if the user is using a device with controllers like an arv or vr headset it uses those you add support for hand tracking and do from a device that supports hand tracking it uses those um excellent yeah it's um so it takes a bit of thought but it's kind of amazing what you can build with the web and if you build something like that you can reach like the widest possible audience of people with immersive hardware you're not just focusing on one particular like piece of hardware you're able to support devices that haven't even come out yet exactly as long as they have a web browser right precisely which i really i really hope that we continue to have this this um this thing where as more immersive web devices come out they have a web browser like there's even a new device coming out soon um which you will even have a have a new um a new browser designed for xr so the new wolvic browser was forked from mozilla reality mozilla reality by igalia in order to carry on supporting vr on immersive headsets and so it's really great to see that as new hardware comes out they really view the web as a as a high priority excellent yeah hold it hold it don't you dare