Bose AR Unity Workshop

Key Takeaways

all right so welcome everyone to our bosear unity workshop my name is philippe baba i'm a senior developer advocate at bose so what that means basically is i work with developers enabling them to build experiences for our platform i host workshops we do game jams hackathons also public speaking on behalf of the platform and if you do end up like building something with the bosear unity sdk specifically i'm on the unity side you'll most likely be talking to me and i'll be you know helping you get your idea and your product out there and you know trying to bring those on board to to help you out with that as much as possible so i am your voice when it comes to these things um so i'm just going to talk a little bit briefly about what is bose they are some of you may already know or not but i'm just going to get into it so we have a new concept called audio augmented reality now many of you probably already heard about augmented reality first thing you probably think about is you know hololens or ar kit arcore magic leap stuff like that so we are actually audio first augmented reality now what that means is think like spatial audio for example um where you have sounds placed in space around you and you're kind of like virtually hearing it spatially so what enables us to do audio augmented reality are these sensors that are located in each of these bozear wearables you all have a pair of frames but then we also have our headphones as well which have the same sets of sensors in them those sensors are an accelerometer which allows you to get impulse and you know sudden movements gyroscope which tracks rotation so typically what we use the gyro for is to track the user's head orientation and magnetometer which is basically a compass and we use that for our navigation apps so if you know the gps position of a user based on their phone and if you use the magnetic heading from where their head is actually like pointing at you can kind of know what they're looking at like on the map pretty accurately actually so we have gestures as well on our devices so the frame specifically have an input gesture here on the right side it's a double tap so it's not a specific like touch sensitive sensor it's actually based on the accelerometer so it's like the little impulse tap and the tap actually works from from any side but it's recommended to do it from the right side because that's where the sensor kit is we also have custom gestures that we actually encourage our developers to kind of explore so and it's a mixture of the other sensors so a custom gesture would be something like uh you're creating an experience until you look up and you look up and then you track that rotation change that would be a custom gesture i know some some of the games that have been built sometimes it tells you to duck and there's a bullet flying over you or something like that so that that would be a custom gesture so heads up hands-free is one of the models that we have um so the idea behind heads up hands-free is a lot of the applications today whether they're ar or just standard mobile applications you're always staring at your phone you're always walking around you know you're an eczema crane because you're always looking down at your phone the whole idea behind heads up hands-free is that we want to create applications that you don't have to stare at your phone the input is the wearable and the output also goes through the wearable so you don't actually have to look at a screen for most of these activities so i'll give you an example um let's say uh like navigate for example what it does is it knows your gps location based on the mobile device and it knows you're heading based on where you're looking with the magnetometer so what they can do is they can tell okay you're looking at this restaurant for example or this cafe you double tap the input gesture and it'll actually pull a yelp review so now think about the non-heads-up hands-free way of pulling up a yelp review what are you doing you're most likely googling the name you know or you're typing in the street address but it requires you to pull out your phone launch the app find a name type it in or the address or or on the pin on the map and then you get to that yelp review now that's it takes you out of what you're doing um so if you want to look up like five or six restaurants walking down the street you're basically probably staring at your phone for like 15 minutes the idea behind heads up hands-free is you just walk by a place tap you get the information you move on no no phone required at least not looking at it um now it's it's a really cool concept i think um and a bunch of our apps kind of like use that as its guiding principle it's not a must but it's it's kind of like a design philosophy and some of the developers have been really clever about about implementing it so the way that it works is the sensors actually the data gets sent to the mobile app so the wearable has to work with the mobile phone so ios or android um and that that goes for the frames and the headphones as well so some of the products that work with it are like the frames for example which you all have here i'm going to show just a little brief what kind of like intro and frames [Music] so [Music] [Music] is [Music] so frames were actually a concept that came out of our san francisco office so it's a very different kind of idea behind it i personally think it's a super cool idea i think frames are really great like outdoor wearable it's great while you're biking for example when you don't want to kind of be interrupted um if you don't you don't want to get a call you can just pick up straight through the frames you don't have to pull out your phone or you can do turn-by-turn navigation again you don't have to like be mounting your phone putting like a waterproof case or whatever you just have your frames on also if that doesn't plug up your ears it's open your audio so you can still hear the sounds of the street around you and it's not unsafe in that way like wearing like noise cancelling headphones while biking for example uh i just realized i gotta like do this quick time thing might be cut there there we go next up qc35 twos now this product you might be more familiar with you've probably seen people wearing these um usually maybe on an airplane so this is like the classic um nice cancelling headphone which also has boseaw sensors in it and a lot of people actually don't know that so i think i believe it's all the models after october of 2018 were were manufactured with the bosear sensors in it and then the nc 700s which are these so these are the the brand new noise cancelling headphones that just came out uh i get asked like what's the difference between the qcs and the ncs there's a few obviously there's a there's a redesign it looks a lot more modern it has a little bit more kind of like a sleek aesthetic i kind of think that the the qc35s have a very kind of like uh dad road trip kind of look to them especially the gray ones um but yeah it's a brand new redesign and one of the main um upgrades i'd say is the microphone so for conference calling it actually has a noise cancelling microphone because it has eight beam forming mics so it can actually pick up your voice as opposed to the to the environment so i found it really good in like loud environments they've actually done like comparison tests like in a loud environment and you can barely hear anything in the background while you know while you're talking with them so um i personally really like that because i do a lot of conference calls and sometimes noisy environments um there's a little promo video associated with that too if you want to see and these were the first the first product that was kind of the first bose product that kind of has been marketed with the bose ar features in it so the whole idea behind the nc's is again the heads up hands-free is [Music] [Music] so i don't know if you saw him doing it in a video but there's a little capacitive touch screen here on the side so it's not touch screen it's a touch surface yeah and um it's uh yeah you can do volume up and down skip track or or go back and then there's also a touch and hold which you can actually um get as a gesture inside bose ar so when you're building your app so again the whole idea is you're not pulling out your phone you have your wearable your your ar enabled wearable which is your headphones which you carry around with you if you're like me i carry around my headphones everywhere so you know you can talk to your to your google assistant or to siri through it receive messages it'll actually read out the messages too so the whole again the whole idea is to have this wearable be pretty much like your main input and output device for your daily just you know mobile uses so one of our projections that we're going to have 1 million ar devices in 2019 across all three of the products that i just showed you now that's actually really huge because um most are is is single kind of purpose ar it's like you if you get a hololens it's more enterprise you're not really going to be wearing it out on the bus you know or magically same thing you know they only work in certain environments you know it can't be in bright sunlight and ar typically is is a very niche thing that's found in in specific places for specific use cases but the idea behind bozear is that these wearables are actually being worn by people every day already just for their regular life to listen to music to do their conference calls but now we have this ar platform built into it as well and developers can actually leverage a very big install base because there's there's a lot of these devices out there and they're constantly being sold and the you know the user base is growing and they're going to be looking for for bose ar-enabled apps what can i run on these new headphones that i just bought um so we have an ios and android native sdk and then we also have a unity plugin which is what we're going to be using today which actually works with both ios and android so i'm just going to go into a few of the use cases a few of the apps that are here that might be a little bit inspiring so you can kind of get an idea what people are doing um so autocast is is a really cool concept behind like um how you can for example let's say you're a music festival um they could set up a tour where you actually hear these audio pings so if there's like a point of interest as you're walking by you'll just hear like a spatial audio pane and it's very unintrusive it's not like it's yelling at you like hey there's a thing here it's just kind of like it's a ping and then if you look at that ping you can actually double tap to enter it and it'll tell you oh this is um you know this stage or maybe if you're doing like a historical tour i'll tell you this is this landmark would you like to enter this tour and then you you enter it and then you can go and actually explore and it'll it'll kind of like tell you what what these environments are uh walk is kind of a similar concept also to taylorcast navigate and never got explained earlier it's kind of like you're using the gesture to yelp things basically so it's really cool to kind of like not bring you out of your travel experience you're just kind of like walking and exploring and getting the information you need without again staring at your phone so all these kind of use initially in the sensors but i said so use the gps from the mobile device and then use the magnetic heading from the wearable and what's really cool about using magnetic heading from the frames people always wear their sunglasses perpendicular you know it's not like your phone where the magnetometer sometimes you ever use the map and you're trying to look at the compass on the map and it's opposite or it's not very accurate you gotta like do this and you gotta like recalibrate your phone so with the frames it's actually a lot better because you're not you're never really putting that in like a wrong direction it's always kind of like facing perpendicular so it's it's really good to get the heading um sport and wellness so another use case is somebody sport apps so golfshot is actually a really great example uh people wear frames when they're off out on the golf course it's sunglasses they might be listening to music as well so it's a perfect fit so golfshot is an already established app um that golfers use to get information about about their shot you know stored game details i'm not much of a golfer so i don't really know like all the details but i do know that uh they have a beaujar enabled features that actually allow you to measure things like distance to to the next hole and again without having to pull out your phone and typing in some information it uses all the data from the sensors and from the mobile device it knows where you are and knows what course you're on and then you can kind of have this seamless experience again without pulling out your phone you're just already wearing sunglasses because you're playing golf and it's sunny and you know yeah i i hope that makes sense so new balance i think is is a really cool one this is a demo um so they kind of created like more of a uh like a workout app it's kind of like a motivational workout app where it'll count your sets and reps so think like you're doing sit-ups and you're actually doing that custom gesture and you're counting how many sit-ups you did and it gives you like positive feedback oh you're doing great two more you know or something like that um so again how if you weren't doing heads-up hands-free you know maybe you'd be checking your phone to see how many you've done or you know there's really no way to kind of do it seamlessly unless you're kind of wearing a little extra piece of sensors on you headspace is also a demo and this is actually is getting a big revamp but the idea behind this is it's a meditation app so it's a perfect fit for things like spatial audio they even have a little bit of an exercise you kind of like stretch your neck and then again it's using kind of like the the sensors in the wearable to actually be able to tell if you did your your neck stretch and when you did your next stretch and then it kind of has a soothing voice it's like all right so now we're going to close our eyes and you know and it starts to play like a little soundscape and it helps you kind of like meditate um another idea i heard brought up is you could actually check someone's posture so if someone's falling asleep you can kind of like slightly ping them to wake up you know so things like that um like i said developers have been really creative with it and then music of course um i think music is is one of the best use cases because i think spatial audio which is what today's workshop is mainly going to be about is a very unexplored area of of music or sound in general um and bojar is a perfect fit for spatial audio especially because of the sensors that i was just mentioning um because you know the head orientation you can actually simulate spatial audio just between the two stereo speakers you can actually blend the sound between based on where where your head is looking so radar is a great example of this a radar is an app built by bose which actually connects to something that's called creator tool that's currently in development it's in it's enclosed beta but the idea here is that radar is going to be kind of like a almost like a marketplace like area where you can browse lots of different spatial audio experiences and they're going to be tied to certain areas like on the map so maybe if you're in a certain place there's certain spatial audio experiences that wouldn't be in some other place and it can do things like it could ping you when you enter let's say some park and maybe there's a spatial audio experience tied to that park and it'll you know you'll be able to enter it if you just kind of have the app running in the background uh traverse is a cool example of a mixture between our api and existing ar apis i think they're using ar kit and what it actually allows you to do is it's kind of like you're you're in the middle of a concert and you're hearing like the different instruments playing around you so what you can actually do is while you're holding your phone up you can actually kind of like close your eyes just walk around the room and you're actually exploring all the different instruments that are on you can walk up to the singer you can walk up to the piano and it's using ar kit to do positional tracking and uh bossier for for spatial audio and your head orientation so it's actually using both the phone's positional and the head's rotational tracking and it knows between the two and it kind of simulates as if you're actually in the middle of a band or you know a concert that's playing um iheart radio is is a cool demo app i'd say uh it's basically another take on like how you can um tune into different radio channels so instead of pulling out your phone and and seeking you're actually using your head orientation and you're kind of like your head's the tuner and then when you like something you just kind of like double tap it and you go into it and then you can pull out of it at any time and tune into something else and if you you get a little preview and you like it you tap into it and you're listening to it without having to like look at oh i'm 22.3 fm or whatever you're just kind of like listening to it um now games uh is predominantly like in in the realm of our unity sdk so we have some cool games that have been built the the three that i have up here are what i would call representative experiences so we actually have a lot more games that have been built especially this year um and i'll show you actually some videos of the games to kind of like get inspired so all these games are actually out so you can download them on the app store um if you know if you have a bose ar enabled wearable uh and like i said they're representative experiences meaning that i believe these developers uh kind of like created some design patterns out of these they were one of the first to make games for it and i think these three are are kind of like in their own category so overheard it's a really funny game you basically play as an englishman that's taunted by french men it castles at night it's very kind of like monty python s funny accents the voice acting is hilarious and what you do is you use your head as like a catapult and your head orientation like this and you double tap and you basically launch cows into into these castles and you kind of hear them move and you're like and it's all spatial so you're kind of listening for where they're at and when you're turning your head you hear the castle's over there and then they're the frenchmen are like taunting you like oh i bet you can't hit you know the broad side of a barn or whatever my you know i'm not going to try to do the accent but you should hear it it's it it's pretty hilarious um and uh it's a great example of it's it's kind of like using the spatial audio and using your head orientation as like the controller basically so comrade is a cool uh design pattern so it's basically like a spy novel um and it's it's a choose your own kind of like adventure spy novel so you can actually make decisions in the story so the story goes on and there's you know yes and no kind of like decision trees that you go through and at some point i think you have to duck and dodge a bullet um and it's got this whole ai theme to it and the ai is talking to you and you know you're making decisions and you kind of change the outcome of the story so that's a pretty popular design pattern and i one that i'm a huge fan of i love choose your own adventure games and i think that the medium is perfect for it um instead of just doing yes or no on the screen you can actually do yes and no by nodding you know and shaking your head so that's that's it's very immersive when you get asked the question and you're like yes and then it kind of like goes on um dead drop desperado is a multiplayer game uh it's like a local multiplayer game so what i actually really like about this one is it's got a little bit of a party game kind of uh aesthetic to it so you play with a friend and you one of you holds the mobile phone and the other one gets handed off like the the wearable so let's say the headphones and what happens is he's standing in front of you and you shoot bullets at your friend and your friend is trying to dodge the bullets by listening to where they're coming from using this facial audio and you're kind of like leaning left to right like that and then it gets switched off and then you're the one is dodging and he's the one that's shooting and then in the end there's kind of like a score tally and it's like a leader board and it's kind of like has a little bit of a competitive aspect to it what i personally like is that you could let's say you know you you have your headphones and you know you want to show someone something cool on them you can just kind of like hand off the headphones and now now it's this multiplayer interaction so you have the headphones i have the phone and you switch it up and it's it's also a little bit competitive with the leaderboard and such so like i said like representative experiences um you're definitely not limited to these design patterns we're actually excited to see what our developers kind of come up with and there's always like you know cool stuff that the people come up with and i just wanted to show you a little bit um so we have this video that was produced so we've hosted a bunch of game jams this year and uh this is like one of the first game jams that we hosted and you know these games that i just showed you were actually built at it so i'll just i'll go ahead and show you so this is from our play crafting partnership uh play crafting is like a small like kind of like game developer collective but plus they also kind of like host events like play nyc and they you know bring developers um they teach unity classes and uh they just have a general kind of like game developer community across the country so we've kind of partnered with them to have some of these uh bozair game jams and yeah i'll just show you the video we're here this weekend and we're doing a game jam with their new ar technology we've given the developers a brand new version of the bose ar sdk and unity plugin it allows unity developers to easily add rosear integration to community-based apps they've built a very intuitive system for our developers like me to be able to just pop an immunity and just go we give you access to all of the sensors that are available in the device the accelerometer the gyroscope two different rotation sensors they provide orientation of the user's head and world space we can actually provide spatialized audio which just gives you an entirely new level of context when you're making any sort of experience we're able to get all sorts of different types of information from the frames glasses which we can use to help develop new types of interactions we were challenged to create an augmented reality experience that put audio first it's very interesting to look at games from like a whole different angle usually people think ar they go it's all visual and this one it's both so it's audio the challenge really made us put our heads together and be like what can you really do there's a whole world of sound and we shouldn't just be paying attention to just the visual components so we've had a lot of really creative ideas that the developers have come up with so far and i think some of the more interesting aspects are the different game mechanics or the user interface experiments that they're doing the game we created we're calling sonic samurai you're the samurai and you can't see but there are all these monsters around you and you basically have to use the bose headset to determine where the monsters are and then you actually use your mobile device as your sword to strike at the monsters it's a game where you essentially try to dodge bullets that are sent from one user towards you imagine the matrix but you can't see the bullets that are coming at you you can only hear them our game is called overheard you play a medieval catapult operator catapulting poultry and cows at french castles you're doing this all at night under cover of darkness so you have to use your ears to hear where the castle is when you point your frames in the correct direction you can hear the french defenders taunting you daring you to throw things at them your efforts are always the perfectly good livestocks by turning your head and elevating it up you actually modulate the angle that the catapult will fire at you are being bombarded by vegetables in a 360 degree radius you can tell what direction the vegetables are coming from based on spatial audio and i you need to find and face the correct vegetable and eat it we're working on kind of a spy thriller game when you put on those frames you start to get radio chatter from secret agents you start doing instructions to locate radar pings the fuse bombs like james bond and mission possible in real life augmented reality is really one of the most important future mediums for design we're getting the tools set up in a way that allow people to start experimenting and using something like unity allows that iteration to happen very very quickly it's an honor to be chosen to be here to work on this technology for the first time and also give developers a bigger idea of like what's possible with this stuff so that they can use our games as like a stepping stone for what's next here we are at pax east 2019 all five of these games are debuting for the first time anywhere people are having a lot of fun here this weekend lots of people dodging vegetables and throwing cows i always enjoy like finding new games that are animated i've never played anything just based entirely on sound i don't think i've seen anything else here that relies completely on audio and motion no visual at all but it felt really immersive and cool people are really getting a sense that the audio is like real spatial you feel like you're really there it's just a whole different experience it's more social it really brings us back to where games really are where it's with other people it's a fun ride on these classes we're changing the landscape of ar and games in america and this is just a taste of what's to come that's very exciting for us to have that chance to impact other people's lives with a new technology the things that people are able to do over a weekend really inspired me to think we're going to see some amazing stuff this year we have actually seen some some pretty amazing stuff this year i feel like the more of these experiences get built and the more kind of developers get creative with it you know it's an early platform so there's plenty of room to kind of like set set the standard at what what could be created um so what else do i have here um yes so i have a few um actual like gameplay footage that i'd like to show you on some experiences that i personally like um developers cut up these videos to kind of show what what the games are like so just gonna get into these again just some some more inspiration the worst grim reaper soul mates is an ar music t-shirt in which a singing grim reaper plays a song that responds to you and your world grim reaper sebastian is a character from another one of our games he's tired of his dead-end job collecting souls and wants to pursue his dream of being a songwriter seeking inspiration sebastian asks you to let him write a song about your life [Music] you were looking out maybe lost [Music] there's magic in almost every place [Music] would you like to share your thoughts with me [Music] wanting to learn about the inside of the human skull i ask what's going on in your soul okay um [Music] humankind okay so if it's not spaghetti or chocolate cake is it maybe the fact that the sun will suddenly explode and everything we know will go up in the ocean of flames [Music] is [Music] [Music] we imagine players using this app while commuting exploring or just relaxing through interacting with sebastian and his song we want to show players that even the smallest things in life can carry meaning sometimes we just need someone to talk to cool right uh it actually works it's actually it's actually really fun so one of the things that i personally um have learned over over doing some of these jams and seeing what developers uh build is that when you're not focusing so much on the visuals you have a lot of room for story and and substance if i if i could say so myself it's like uh this developer uh it was we had all teams of four so three of those people were were just content creators so one was brought as midi he was he was actually producing music during the gym one was just doing like the singing and the voice acting um i forgot what the third guy was doing i think he was doing like ui even though you do still you know i do recommend to have nice polished ui but it doesn't have to be crazy that's the thing like most games that are very 3d and one of the things is that's that's a problem i'm a game developer myself nowadays there's so many tools and so many things you can do it's kind of hard to like figure out what you want to do because you're like oh my god that's just overwhelmed with you know all these all these things that our modern devices can do um so there's a lot of room for for this kind of content to be produced so this is kind of like a little ai kind of like companion game you could say and yeah a response and there's other things he didn't mention in the video like it actually changes based on the weather so actually checks the weather if it's a rainy day you start the game he'll have like a rainy day mood you know if it's a sunny day he'll have a sunny day mood so it's kind of like it's a replayable thing and i don't expect people to be running this app like non-stop but you i could probably see people running this for that one hour of their commute during the day or maybe like for their lunch break or something like that which is actually a lot in terms of like mobile mobile app usage um you know so it's fun and it's music so you're you know you're probably already listening to music why not have like a soundtrack to your life for example that's kind of the idea behind the game this one uh is also like a really cool example so i'll just go let him explain it that is about popping and grouping your head to the music similar to other rhythm games you score points by staying on the beat at the moment we have about a dozen custom written songs in the future we want the possibility for players to download hundreds of songs or play with their own music library [Music] we're going to play party the washing machine our foxy friend here is grooving along as well you could follow it and he helped you stay on the beat the line below is showing your head movements it also changes colors to indicate how well you're doing we really intend this game to be a phone and pocket experience we added both audio and tactile feedback for example the high pass filter turns on when you're doing bad and your phone vibrates with the beat when you're doing great so the idea of hip-hop hero came from us wanted to make a game that helped naturally play with the bose ar devices something you can play while competing on the train in public yeah it's like um i'll play on words of guitar hero kind of you know you're kind of staying on the beat you get rated based on how well you did how well on the beat you stayed and you don't have to have a specific head gesture it's just kind of like how you would normally bob your head and they have a pretty good algorithm for detecting that change they're using both the accelerometer and the gyro but i'm not i'm not 100 sure on that um but yeah i mean it's uh i thought it was great how they visualized it as well so that's an actual like graph based on your your head movement and then you get rated and you know it's yeah you've probably seen games like this and this this is the last one this one i like uh because you're kind of producing music in a way now it's it's it's basically song loops that they've created but the amount of song loops that are in this app for example i mean i'm someone here can probably do the the mathematical permutations of how many combinations of unique songs you can get out of it but it's a lot um when you when you include like 10 loops for example for you know bass lines as you're kind of adding complexity to a song and if you have like you know maybe like 10 of each samples and in the end you have like maybe like 40 samples in total you could be creating unique music on every play through choose your own adventure musical experience of layering songs set in a world where flowers produce music players listen explore and yarn along to the music collecting song loops like palm in order to blossom a musical garden the game is played by looking around to find new sounds then nodding your head along to the beat when you find when you like this process can go on indefinitely allowing players to explore the countless new ways of layering together song tracks it's basically she totally nailed the explanation like in the beginning um but basically yeah it's like you look around you hear a preview of a loop you like it you nod it and then and you go on and you're just layering loops and and it it works really well i mean the developer really figured out a way to make it you know if you've ever tried to make music sometimes you could just mess up and it just sounds terrible because you're mixing way too many dissonant sounds but they managed to figure out an algorithm to layer it and it always sounds good and i i've tried to break it whenever i play in these experiences i'm like pressing everywhere and nodding to everything trying to to make it sound bad and i couldn't it always sounded good so super cool so that's our developer portal it's developer.bose.com forward slash bosear um that's where you'll find the sdk i'm sure most of you have already gone on here and downloaded it but i'll show you where all that is in a sec and uh the other the last kind of like but not least um important thing that i wanted to say so we have a new program that we just launched recently and that's bozear certification so what bozear certification is is say you build a bozear app and you submit it for certification our internal team will actually help you get across the finish line and publish the app to the app store or the android store making sure you have all the right requirements and then if it does get certified it's in within our brand guidelines which again will kind of help you get through that finish line you can get a possible feature on the bose music app or the bose connect app and you might even get featured on the bose website so it's a really great showcase and a really great way to get noticed especially as like a early kind of like game developer it's it's an early platform so obviously you know we're looking for great content to feature on our portals and if the content's great and it's again it passes certification which all these games that i've shown you they're all in the process of of getting certified and all be like available when you have you know your your frames for example when you hook them up in the bose connect app there's actually a section there bozear enabled apps and then you click that and there's a directory there of apps yes this works right now so it works on android as well yes it does um we had a little bit of a like earlier start with ios sdk so so a lot of stuff was mainly ios yeah now we're rectifying that there's going to be a lot of android experiences coming out sadly it's still you know some developers just build for ios and we kind of have to prob them like hey could you also build an android build and um sometimes they need to be incentivized to do that other times they just do it if it's unity then you know most of these experiences are going to be on both um but sadly that's just kind of how native app development is they sometimes just want to work in one and not the other but we are fully android supported now as well um yeah and then we also have like a get inspired page this is a place where we have some case studies written um golf shots there if you want to read about it dead drop desperado's also up there i think overheard and it's just kind of like a little interview with the developers they kind of talk about their process of developing and you know it's a good place to get inspired and give me like this guy really happy like ah [Laughter] the uh yeah that's that's the basically the intro uh hope that didn't take too long um but yeah i just wanted to kind of give you a little bit of an understanding of of what this is um so does anyone have any questions for nope cool all right so now that we've gone through that uh what i'm gonna do here i'm just gonna close my unity and uh let's go ahead and actually fire up unity and we're gonna get like our little beginner workshop started so you should all have a copy of the unity package of the bozear sdk for unity which you can find it on our downloads page let me know if you have trouble finding that we're going to be using this 4.0.1 beta when you download that it's just a unity package it gets downloaded so i'm just going to let everyone kind of like get to that and the version of unity that i'm going to be using is just the latest one i have 2019.2.5 f1 um and i have android and ios build support as modules in it and i recommend that you you know if you're going to be building to mobile that you have your desired platform modules installed um i'm just going to let everyone go ahead and do that it's always it's always a thing yeah yeah it's just apple and iphone but again if you if you don't get a chance to build you'll still be able to preview the app and play with it directly from your laptop so that's you know for for most of this workshop we're actually going to be sticking to the creative side of things and and concept and trying to build an experience and then like i said towards the end we'll we'll try to make sure that everyone has built um and if you don't have the right device i'm sure we have some extra devices and if it comes to it and you know we could we could potentially also build or maybe share devices if needed um but once you have it set up once then you're good to go i have my iphone here i usually have to have both all right everyone got their their sdk and ready to go so what we're going to do is we're going to actually make a new project in unity and i'll just make sure everyone has this before we do that there we go i'm just going to do the latest and i'm just going to call this project mit bose ar unity workshop and i'm just going to pick 3d project and create bless you yes it is yeah i think i just missed a little bit in the beginning all right so i'm just going to switch to the default layout here so anyone not have their unity new unity project open yet all right just take your time well good thing the internet's fast how old what what's the version that's fine yeah it's fine i'm downloading the new one anything that's past the the lts the 2017.1.4 should be fine anything with like an f at the end of it it's final now beta is should work okay if for whatever reason it doesn't i know sometimes there's been some some edge cases but it should be fine yes but i'm sticking to it so when you do have this um i just want you to take that unity package that was downloaded and just click it or double click it to open it and you'll get an import dialog and i just want you to import the entire thing into your project everyone imported just you should just be able to double click the unity package and it should just pull up unity if for whatever reason that doesn't work you can always right click in your project here and then import package custom package that also works as a backup instead oh yeah um don't worry about that everyone imported ready to go give it another minute so yeah when you import the unity package you get the entirety of the sdk um there's samples in here sample scenes there's a lot of cool stuff in here that i'll get into and the setup itself is actually really straightforward and simple we've tried to make this like as easy as possible to either put into a new project or integrate with an existing project so if you're familiar with unity or if you've worked on a unity project before you'll find that a lot of the um the i guess the the the user flow and the way that the sdk works is is typical kind of like to to a unity workflow um so once you're all imported you'll see these three folders show up so the first one that i want you to look at is the bose folder if you go inside the bose folder there's wearable and then inside wearable there's modules you go to modules and the first module that we want is connection so we're going to go to connection and then prefabs all right so that's bose wearable modules connection prefabs and i want you to pull in this first prefab that's available here uh so just drag that into your scene and you should see this this white rectangle pop up now the first thing that i like to do um when i start like a unity project especially if i'm developing on mobile is i like to pull my game view which is usually up here you might have a different layout and i actually like to take the game view and i like to make it visible so i actually drag that game view over here to the right side of the project panel so i always kind of have a little preview window here of of what the what my preview looks like basically so if you go ahead and do that then you'll see that the panel that we just pulled in actually shows up here so and i'll explain what the wearable connect panel does it's basically the entire menu that shows up that that you can browse all available devices you click a device and you connect to it um so this handles all of the the connection for you there's some cool features on here like auto reconnect and and such so for example if during your app for exam maybe bluetooth connection turns off or or for whatever reason the battery runs out this panel will pop up again it'll say you've been disconnected you know searching for a device to kind of like keep the user experience clean so you have the wearable connect ui panel and the next thing that we actually need to make the wearable connect ui panel work is we need an event system so what i want you to do is i actually want you to just create an empty game object somewhere in the scene and call it event system okay and on that empty game object i'm going to add component here in the inspector and i'm just going to start typing event system there it is it's a built-in unity thing and once you have your event system there's a button there called add default input modules so just click that now explain what this does this actually allows you to click the ui of the connect panel now the reason why we do this separate is maybe you have an already existing unity project and maybe you already have your own event system that you've set up so we don't want to mess with that but since we're starting a new project we're going to create this event system and add the default input modules like i said that just allows you to actually click the ui and and select the panel so that's pretty much it the last piece that we need to actually um set up like our entire bose ar is we actually go in and we're going to create another empty object and we're going to call this one wearable control now what this will do this will actually stream the sensor data back and forth from our app so wearable control and i'm going to add component now if you clear the search here i'm just going to show you where the bose components are so when you clear the search there is a bose subcategory here and you can click bose wearable and we have some sample scripts here that you can use so the one that we want is wearable control we're just going to click that now one other useful thing that we added semi-recently is uh you can actually go up into let's see here it's yeah so if you go to asset for assets create and then there's a bose wearable category and there's an app intent profile so i'm actually going to take that app intent profile i'm going to put that like in my top directory and and assets because i was in the in the prefabs folder here you don't have to but it's just cleaner that way and what you'll see here on the right is you'll actually see the sensors that are available to us so for this first app that we're building um we just need the gyroscope so i'm gonna check the gyroscope on here and rotation sixth off and then the update interval i'm gonna make it uh let's say 40 milliseconds that's all we kind of need in terms of sensors and what this basically does is it just kind of syncs up what what the app requirements are and what what the app is going to be using so back to our wearable control that we created earlier you'll see there's a active app intent profile empty slot right here you can actually just click that and just pull in that app intent profile that we just created so again that's uh app intent profile is in assets create bose wearable app intent profile in the app intent profile you want to select gyroscope rotation sixth off and 40 milliseconds and then in wearable control just you know put that app intent profile there the last step to get this to actually work is where it says here you see it says editor default provider it says debug provider we're actually going to switch that to usb provider and what that will actually allow us to do is we'll actually allow us to send the data over usb bridge directly to unity which is great for debugging um while you're developing so that's actually what we're going to be doing first so your wearable control has a usb provider and that's pretty much it so your app will actually now connect both using usb and if you built this to android or ios you'll actually be able to pair with bluetooth the only problem is is we're not getting any of the sensor data so we're not actually doing anything with it yet so the first thing that i want to do is let's say i just want to uh visualize like the head orientation like on the screen so how do we do that right so what we're going to do here is i actually want you to go back to the modules and in here there's another folder called model loader and prefabs so if you go to bose wearable modules model loader prefabs in here you'll actually see we have a 3d model of all of our bose ar enabled wearables which is actually really cool and then there's also a default for if it doesn't know or maybe for a future product or something like that so the wearable model loader actually pulls up the appropriate model automatically so it knows which one you're doing but right now let's just do it manually so pull up the one that you have so i have rondos here so i'm just going to take the rondos prefab i'm just going to drag it into my scene and if i double click it you'll see it's very small so i would recommend to scale it up let's say to something like 40 by 40 by 40. so we just have our roundels like that in the scene and the last thing that i want to do is on this prefab i actually want to add component and in here we're going to go to bose wearable and then there's a script called rotation matcher so we're going to select that and we're going to make the update interval 40 milliseconds okay so what rotation matcher does is what the name implies it'll match the rotation of the wearable so it'll actually take the gyroscope data um and that's actually pretty much it so this should already work so what we're going to do is we're actually going to test it out so i want you to do is i want you to take your frames hook them up with usb to your computer if you're on windows you'll hear like the connected sound and i'm actually just going to maximize on play here so you can see what i'm doing i'm going to press play it's the picker is there i have my frames i select it in the ui it's going to connect to it and there are my frames tracking the rotation as intended this is all happening over usb so the cable disconnects yep it will stop so the idea is that this is for for development you're obviously going to be using the cable but if you took this app and actually built it to your phone it would work over bluetooth so i want everyone to just get to this point and then we'll move on to the next part and what's really cool about this is we're already pretty much you know just a few steps away from doing spatial audio now that we have the head orientation we can actually move our point of reference for the listener it should be plug and play yeah um does it work for you so make sure you have um the wearable control script so you you skip this step so there is uh you have to have the wearable control script so what you want to do is you want to create an empty game object yeah wearable control there is um oh so event system yeah so on press play so yeah stop the app and then add default input modules there you go now you can press play and you should be able to click the ui this time so yeah there it is um so let's then try let's try to update i mean it might be faster just to swap it with another program it's i think it's it's something that we've also been been kind of told to do is like people should know how to how to update the firmware so if for whatever reason um it's just not working um chances are sometimes that you just need the latest update so where you find that is if you just look up bose updater like if you just google it there's a it's btu.bose.com um just try to follow those steps it's usually pretty fast you just download the little little tool and it's all through through online so you just have it hooked up and it'll pick it up so if i try it on mine here so it looks like this i'm gonna connect my frames i see it it sees it there's an update ready for my product and that's and you just click update now and it'll do it like in in a minute um if you're just working you don't have to do that um but if for whatever reason it doesn't that's usually the problem has like an old version of the firmware um if you need to know how to turn them off you you actually just flip them upside down and you put them on the table and they'll turn off after a few seconds bless you if you get the um filled connection uh usually reconnecting the device works it's it's just a usb debugger thing yes so i'm actually playing the sound through my bose frame so when you connect over usb you can actually drive the audio also through usb now on mac you'll see it pop up like in your sound settings and on windows it's like in the bottom right in the sound sound devices you can set like the the bose frames as your as your default audio device so it's a great way to preview with now what i did here is um i placed the sound on this sphere now this sphere is kind of like just in the middle of my field of view here but uh in the next exercise i'm going to actually kind of show you how to how to spatialize that sound so you can hear it so i'm just going to wait for everyone to make sure that everyone's got this so far yes all right so what we're going to do here there's a couple of ways to do this but so typically like your camera is your frame of reference for audio so as your camera turns if you have 3d sounds in the scene it's going to take the frame of reference from your camera um to spatialize sound right so what we want to do here is i'm just going to actually take the camera that's staring at the glasses now that you don't have to do it like this there's a few different ways to do it but we'll just do it for simplicity's sake we'll take the main camera and we'll pull them into our frames now the way i have my scene set up is my frames are just at at like zero zero zero and they're scale 40 and the camera is kind of like right behind them like that so i'm just going to take the main camera and parent it to the frames so it's kind of like inside the frames and if i press play i'll show you what that looks like so now if you're turning this is kind of like this is your frame of reference so this kind of simulates your orientation in space basically right so here's how this is going to work right so in unity doing spatial audio is actually really really straightforward so what i want you to do is and i already did this i'm just going to delete it so i can show you how i did it um it's very straightforward so now let me what does it come in there we go so what we're gonna do is i'm just gonna use uh a sphere to represent a sound object just for visual sake um so i'm actually gonna create a new game object so 3d object sphere and you know i'm gonna place it at zero zero zero and i'm gonna add component to the sphere and i'm gonna add an audio source okay so that's a built-in unity thing now i take this sphere and let's say this one i'm gonna put it in front of the glasses like here okay maybe a little bit further and oops and the audio source in audio clip if you click here um there's a few different just sound samples that we have in our sdk you could pick like chord lp or something it's up to you which one you want to pick this is just just for the demo um so we're going to pick a sound effect we put it in front and the only thing you got to change is make it loop so there's a little loop check mark here so the sound is you know when it finishes it'll start from the beginning again and then there's a drop down here called 3d sound settings i don't know if you see that we'll get to that in a second but the only one we want to change is right here you see spatial blend okay so i'm actually going to drag spatial blend from 2d to 3d and that's all you need to do to turn audio source into a 3d audio source now you'll notice that there's cool things here like there's actually like the range of the sound that you can tweak and stuff so we don't have to do any of that just yet um we can just kind of leave it as is just make sure it's 3d and that you have a looping sound effect and that's all i want you to do right now so now if you press play and you let it connect mine is failed connecting if that ever happens to you again it's just it's a known bug that we have with usb just reconnect make sure you know you're always saving your scene so you know no crashing happens unity tends to do that sometimes but you know rarely but usually it doesn't um so yeah if you get the failed connecting thing just reconnect and then try again so again so i'm gonna unpress play i'm gonna reconnect it's something to do with that it remembers the previous device and there's a there's a little bug with with the usb in that and connected so so when i turn my head i can kind of hear that sphere in front of me and when i turn this way i hear it here now i hear it on the left so in order to get that make sure you're driving your audio through the bose frames which on mac you go into the sound settings here and on windows i can also show you like in the bottom right in the where the volume mixer is if if it's saying that connecting to pixel 2 go into the bose connect app and to the the settings and you can actually remove any previous device that was there so it doesn't always do that auto thing so there is a little bit of a like a doppler effect i don't know if you can kind of tell when you're doing that right so these are all things you can actually change here in the in the 3d sound settings so i think if you just turn off the doppler it won't do that i think so yeah um if you just go into 3d sound settings and you turn off the doppler you won't get that kind of like warp effect when your head turns um that is an interesting effect that can be played with but again this is just the the built-in um unity spatial audio we actually have a couple of different ways you can do spatial audio so we've gotten to the to the point where you've set up your device you've gotten a little bit of an intro you've seen what what the platform is about and this is basically uh a first example of spatializing audio right so now as an exercise what i'd like you to do for for the next 30 minutes or so or maybe even you know maybe let's say 20 minutes um go online find some sound samples set up your own little spatial soundscape around you and uh let's see how that goes yeah make something cool later you can get some beach sounds maybe like water crashing on the side birds chirping or it could be music samples maybe you get a drum loop um i can show you a few like i like freesound.org is cool i believe it's resound.org yeah uh you need to make an account but it's free and you know it's all like um free sounds there's also a decent amount of like um websites that offer like loops if you want if you're going for like a more musical approach so yeah go ahead and do that and let me know if you have any questions i'll be you know going around and helping you out you