FOCI GenAI Users Group (31Jan2024) : The Large Language Model for Mixed Reality (LLMR)

Key Takeaways

you can take it away standing here is nice all right hey everybody thanks for coming to the first phai uh now rebranded gen AI users group meeting uh last semester he did the llm users group meeting but being more inclusive this time around um our speaker today is Bill long he's a fifth year applied math PhD student here working with wrong G Li and he had the opportunity over the summer to go inter turn with a a small group in Microsoft and work with uh jiren laner if you know who he is and if you don't know who he is you should and he can talk about it a lot better than I can so I'll let him take it away hello so uh thank you for all thank you thank you for all for uh showing up um today I'm going to talk about how you can use large language models to uh So-Cal speak the world into existence which is extremely ambitious and this is kind of you can think of it as like a Northstar goal so to speak this is like the the holy Grill you know you kind of um as Arin has mentioned um so let me show you okay so let me first introduce you um to kind of the team that worked on this project so um um like eron mentioned so this was uh uh my summer project last summer and uh it was at Microsoft and they have this kind of research Branch that's kind of experimenting with like different um aspects and kind of exciting applications that one can one can use with large language models and in particular I was working with uh two other interns so um top left that's me and two other interns Fernanda and Cassie uh they're both from MIT one's from the brain and cognitive science lab the other uh from the uh uh media lab um we had two mentors uh Andy and Judith they are both uh principal and Senior researchers at Microsoft and then finally our manager Jaren lar um I don't I don't think a lot of people know him these days but he was once so yeah I I think he is very popular among certain subgroups of uh subcultures and subgroups of people um probably most famous for um inventing the word VR virtual reality and popularizing it by creating the first commercial kind of commercially viable company to create VR products um he is um kind of old now but he's still active he's still very sharp uh which is extremely impressive to me um but anyway that was the the team and this talk the general structure is going to be uh the following so um it will be kind of it will kind of have two parts um the first part I will kind of introduce you to this kind of overarching framework that we have kind of developed during the summer we call it LMR large language model for mixed reality um kind of the tldr one sentence summary of this model is that it's it's kind of a framework that can allow you to do real time generation and modification of virtual scams objects worlds what have you um using large language models and this is done in unity uh for those of you who are not familiar with it um Unity was developed as kind of a game engine so it's like a platform you can like write and Implement video games um I think prob I don't think anyone will distribute uh dispute this um probably the two most famous um most kind of popular and famous Frameworks out there is unity and unreal um Unity I think have more people um there are more people who knows how to use Unity because of its uh Simplicity and that's why we chose it um so the first part of the talk I would kind of introduce this whole system that um can do many different things um a lot of it I will show you just in just a minute uh I will talk about kind of briefly the um um kind of how the uh framework functions um as kind of an orchestration of different language models um and then I will move on to the second part which is um you can think of it as a particular component of this framework that we have developed uh in particular we're using this framework to uh do animation Creation in real time and that's something that really uh excited me because me personally I really like video games I U I think any kind of application between Ai and video games is extremely exciting um so yeah um couple things couple more things I mention so um uh oh one thing I mention is like two weeks ago we got the notification this was accepted to Kai 24 pretty happy about that um and uh I will this talk will be um light on the technical details but um you're more than welcome to ask me about anything um down to the tiniest detail um I will more kind of focus on just showing you like a lot of videos of what it can do and kind of applications um that we kind of have dreamed that maybe this can do um yeah so without further Ado let's get into it so let me let me start with kind of the minimal example of how uh you can use use our framework okay so on the screen um which I hope the definition is good enough but essentially this is like the um this is kind of a screenshot on the unity uh the gang engine that I have mentioned and what it is is that so these kind of boxes are uh we put into are the code we wrote Into the uh gang engine so you can think of what we're doing is we're kind of embedding a language model inside a Unity engine which supports all sorts of stuff like object creation physics you know like lighting like just all sorts of these uh many people have worked on it over the years and we're kind of trying to push the large language model to see how much it can leverage and harness the ability um uh that's embedded in the unity engine to create and modify these kind of virtual seams so this first example what I will show you is um you you can use our framework for just simple object creation that's certainly possible so how it works is that there's this uh text box on the bottom so yes uh so um the main code is not open source yet we're working on it they're legal te I don't know um but but part of it um uh for the second part of the talk is um I I can put those links up I can certainly give you the link to the paper just want to uh M okay uh what will say yes so uh how how you will interact with our frameware is that you basically just type there in the text box uh here that what you want to do in this uh existing scene there's nothing here so maybe you can just create something so here let me just show you hopefully you can see what the text is so here I will just do a really simple thing what I will what I were about to type eventually is I will ask the framework to create the cube and the cube should be rotating and while it's rotating it should change its color every second so very simple you can program this yourself um with some effort um but you can also add trib to do it so then I send the requests to the um uh uh framework that we have the framework does some processing including like coding and doing inspection the details that will going into just in a little bit uh in about like 20 seconds you will finish everything and then you will execute that code and then the result will be shown on the screen there you go rotating queue changing color so this is a very minimal example so let's see how far we can push this so I'm about to show you a lot of videos you will have a lot of questions um let's um I will appreciate it if um let's wait till all the videos the demos are sh I'll take questions then first thing I will show you is um you can do object creation that's a little bit more involved so in this particular demo I will I will show you um kind of the creation of a very simple toy car um it should have like a you know semi- believable appearance but also you can kind of drive around with it so this is a sped up version of that video um so here we just prompted GPD to create a car then we prompt it to color the car we prompt it to make the car like interactable and drivable and then we can put the car into like a Terrain and then we can actually drive in the uh d in the terrain to see that it is actually functional the terrain was not generated this is like we put it there but a car everything all the steps into creating this like semi functional car um is is completely generated by the language model uh here's another example um here I'll call it the scene creation so what we've done is we've created a kind of a semi I would say a moderately comp complex scene uh in this case it's a kitchen so um I've cut out the part where uh it's actually like I gradually promp the language model to do this because that took a while this is the finished product uh I would just highlight a few things so um to me what's really interesting here is that so everything here is created by gbt including um all the so all the objects their appearances they relative sizes positioning and colors so you can see that you know there's like this that's kind of an oven with like the furnace stove tops on top there's a fridge over there there's a table with Sal and pepper shakers you can see that all these things they're kind of placing actually kind of a rational like semi sensible order and their scales are believable so PBT or language model certainly has some like kind of physical intuition as to like how these things should look where they should be placed right next to each other you can take this further so let me show you kind of a different set of examples um here you um instead of kind of creating objects from like the vacuum you're uh what we have is kind of an existing scene so this scene is a kind of a bathroom scene which I grabed from I think sketch Fab which is like an online repository for 3D models so someone else made the scene and it looks pretty and stuff but nothing in this scene is kind of uh interactable so for example um the lights they're all fake they're just visuals um uh the faucet here is also fake you cannot turn it on obviously but one thing I've tried to make gbd do is okay I say okay can you make the water um flow out of the U faucet when I click on it and uh I also give gbt a little hint you can do it without I said use particle effects you can do it without it's less reliable but it can still do it here I used the hint and gbt go processing in the backand I'm just focusing my camera on the actual faucet speed up to when it's done yeah so once it's done you can click on defet and then you can see that it has produced some kind of party effect on defet and the other thing is that um if you let me go back a little bit well I guess you I guess you can't actually see it without the uh um the other the other thing so I guess it's blocked by this like box here but one thing is that um the water is coming out of the faucet it's not actually colliding with anything in the environment so for example there's the sink that's like uh below it right so in physical reality the water coming out of it will like collide with the sink and the sink will collect the water right so you can further from gbt to like make alteration such like these uh GPD will try in this case I may add maybe not super well but you can see that it is colliding with something so at least it's TR uh let me show you another example this one the idea is that uh I want to make it so that my um I can kind of interact with the environment in different ways so for example here my idea is um I want to askb okay can you make it so that you turn my mouse on the clicker I have in the uh virtual environment into kind of a texture stamp so the idea is I ask it so that if I right click on a certain object I will kind of capture the texture on that object and if I left click uh I will transfer the saved texture with uh of the object I right clicked onto the object I left clicked so it's kind of a it's I would say it's kind of a moderately complicated task that I've asked gbd to do I'll just skip over to where it finishes okay so here it's finished so then I right click on the wall I left click on the bathtub there you go and left and I right click on the painting on the wall um and I left click on the bathtub there you go uh one last thing I click on the plant uh and I click on the bathtub so there is I'll show you a few more in this um Direction but I will one thing I'll emphasize is that these we kind of have a we have more of a application in mind um I think the previous few were more just maybe playing around these I want to invite you to think of as um the potential application is maybe accessibility so maybe if there's someone uh using these kind of you know apps or like um these kind of um they're they're having this virtual experience right in the world and they want to um H many times right everyone has their personalized kind of preferences and kind of accessibility needs that it's very hard to cover like have a CLE kind of programming design uh in the original software and our idea is well maybe the pr based technology can kind of cover much more basis um much more kind of naturally so one idea is for example here um this is a scene I grabbed from um I believe uh AI to thore or author is from the AL Institute it's originally used for training like robots for like reinforcement learning um but here we're just using it for our purpose the idea uh for this video is that um okay suppose so I wrot the promps here so suppose you're you're someone who's red green color blind like me um and you look at this room and uh you know you most of the colors are fine but if I may be honest I don't know what this color is I think it is yellow but I'm not sure about that um it could be green could be yellow could be orange to me so just all mushed up so I could promp gbt um to say okay can you make the room more accessible to me um I didn't tell it to how to do this but I just I just given my demands um and uh GPT goes coding after a couple seconds there it is so it knows how to I think it's yeah so it definitely writes some code in the background to go through all the objects and then change their color scheme and the cool thing is it's able to understand like what kind of colors are more friendly to a red green color blind person I'll give you a few more example so maybe you have terrible eyesight um also like me um so you um maybe you want to uh have some kind of like customized magnification tool so in this case the idea is okay can you I'm asking the framework okay can you make it so that my mouse becomes a magnifier and wherever I click you're going to uh the screen the camera is going to zoom in to the place I click um moderately complicated I would say I don't know how to do this if you ask me I probably look up some tutorials um but um turns out that the language models do um so let me maybe skip a little bit to where it's finished okay there we go so it's finished now and then I'm testing it yeah so somehow someway is able to figure out kind of the perspective shifts and like this zooming and zooming out kind of Dynamics um one last thing also kind of goofy um so suppose suppose we want to so we want to take this room we're having a virtual experience and we want to put this room in kind of a party mode all right so so I'm asking the framework okay can you kind of put this room into the party mode and uh do so through kind of manipulating the lights you know do a little light Shi something like that and gbg will go coding and I will show you what it looks like so there you go I don't know if that's my idea of a party or but gbt certainly think so well it's an accessible um here's kind of another kind of category of applications we're kind of maybe dreaming about um is uh what we call remote assist the idea is that you have a lot of these Frontline workers in like hospitals or like firefighters um and a lot of times they need a lot of uh lot of training to be able to do uh the things that they do kind of effectively um and virtual VR as a whole has been like very uh um useful in this in this respect in terms of like training them to do certain tasks but you know producing these kind of training guides and like videos experiences are very kind of like manual and like very um difficult um takes a lot of resources so we thought okay can we kind of generate these guys automatically and this is one such example uh here the coffee machine and the hand are not generated but the movement on the hands as well as kind of the posture or like the gesture the hand is currently have are all determined by GPT so we're just prompting GPT to do okay can you make the hand can you can you can you generate a guy that tells um this person how to make a coffee and then it comes up with the steps and then it comes up with the movement of the hands as well as as cups two more um since we said this model is in uh virtual reality or mixed reality we also have a few VR uh applications one such is um if you know VR there's this app called I think just magic ping brush or Magic Brush uh that allows you basically just to draw things and like uh just kind of like express yourself with like the pink brush that's like in the uh uh that's on your controller so we we were able we were able to kind of create a very simple version of this app just by prompting basically and you can make this the idea is that well you can make this very customized and what used to be kind of like a very U complicated programming like process making these VR apps you can maybe now make them much more more easily I think there's the last one I will show which is um kind of a funky example again um you can you can uh the idea here is that you can you can also do some kind of gesture recognition with it so the uh the hand gesture is recorded by some kind of like uh video device however uh the recognition of this particular gesture that's determined by GPT and also linking the recognition of the gesture to the C being started um that's all done by just PR based technology okay I think that's that's all the videos I have um I'm I'm going to highlight a few things uh which I've given this demonstration to uh uh uh a good number of people now and every time people kind of have the same uh amount of questions which I will kind of cover uh just as a uh per measure so the first thing is um the uh we uh one thing that people ask us a lot is whether we're using uh any kind of vision model um because a lot of these tests would appear that you need some kind of model to do kind of uh object detection or like kind of segmentation or whatever have you uh in fact not so everything I've showed you before I said except the AR demo but I didn't actually show you the AR demo so everything before um is actually all the videos I show you is actually only done through just language models um everything that I show you before has no other modalities involved other than text all that the language model has to do is to code in unity and it can make all of those happen so the second thing I really emphasize is um we did not find tuning on gbd4 as it was back in the summer um when we built this framework so in other words we prom engineer the heck out of it um but I'm kind of using promp Engineering in a liberal way here uh I will show you I will show you just in a little bit what I mean but essentially um what we are what we are building here I think a large part of our technical contribution is that we actually have uh kind of a multi-agent orchestration going on that's eventually be uh that kind of eventually contributed uh to uh the success of like being able to do all those demos so if you just use the standard the Shelf gbt you will not be able to do most of those so I will discuss in just a little bit um how we're able to actually do this um two other things I want to highlight is um uh I think it's also very cool that we're kind of exploring generation of kind of these Virtual Worlds that's beyond the visual aspect of it so you know I think that by now we can generate images extremely well I think we're also beginning to generate videos very well um but a lot of these efforts are still very focused on just the visual aspect which by itself is already very hard um but I think the more ambitious goal here is to kind of um dive into different aspects of what constitutes kind of a meaningful uh virtual world such as it is so it has its internal degrees of logic uh sorry internal degrees of freedom its own like inner logic stuff like that um last thing I want to emphasize is that uh the framework is built not just as kind of a development uh tool so um large language models a lot of times I just use in development time meaning that say I'm a programmer I don't know how to write some piece of code I use the large language model to help me with that piece of code um here we're kind of we want to move beond we want to move beyond that in the sense that we want to make we want to give power to the users in the sense that um we want the user to use this framework and this user user doesn't need to know how to do any of the programming stuff um they can just use this in real time think about accessibility if they have needs they just ask the framework on the spot and the framework will do the bidding so that's that's kind of a perspective change in this sense okay um so now I'll move on to kind of down into the details of how how exactly um this all works so what exactly did we do to make this happen so to explain this I will also kind of um a little example to go with it again a very simple example so uh to start with what the I think the most General setup that we have is that you have some kind of current scene uh that's uh some kind of existing virtual world object or what have you and as the user you're in this scene and you want to kind of uh do something with it so let's say that you know on the right we have uh this virtual scene which looks like a children's room with there's various toys in the scene in particular you have this teddy bear that's in the corner and let's say that you know maybe you're a kid using this uh virtual uh software and you're like okay I want to make this teddy bear twice as bit okay so very simple request um let's see how let's see what our framework does in order to be to be able to um uh carry out this request so the first thing is that there's has to be some level of understanding that's happening so because if you think about this request right so it says make the bear twice as big um in some sense our framework must be able to kind of look at the existing scene and then determine what the user is referring to when they're talking about the Bear right so how that works is the following so when we see this virtual scene because I mentioned there's no vision models there has to be some kind of textural representation of this scene that's being fed to the language model and that's something that looks like the following so um this text um it's kind of readable so what it is it's like a Json file of the hierarchy of objects and their relations in the scene and here the this this is parsed by just like a manually written script um going through the unity back end so as you as you can see that you know it names it gives the scene uh the name of the scene and the various objects positions relative scales orientations Etc and what our framework does is that there's this first um yes generate this one yeah this one is uh uh automatically P parsed so there's a fixed script that does this um there's no no AI involved in this step so so um what our framework does is that you get this text from that virtual scene and what you first do is to feed this text into what we call the scene analyzer which is uh uh which uh by itself is a language model but properly pred so that it knows you know what its rle is and all that what it will eventually do is that it will take this whole string of test representing the entire scene and then only outputting the part that's relevant to the user's request and in this case that's the bear so the sing analyzer is gonna take this uh as its outputs so the relevant object bear that's the output and you will um kind of pass it down uh to another GPT that I will soon cover um there's a u there's the skill Library here which um is not actually used in this particular example I will cover more of it uh later but you can think of it as like kind of a collection of plugins for the language models it's like kind of a specialized library of uh like a specialized um skills uh so this is a simple task so we don't need this so the scene analyzer has looked through the scene and determined that the relevant object is the beay so what it then does is to pass this information down to what we call the Builder which is another language model and the um the role of this language model is that it just writes code it's going to write the code that does what we want it to do which is to make the bear twice as big um and in this case it's the code that it writes is something that something like I'm showing on the right um but before we kind of go ahead and execute this code um what we found is that a lot of the times even as good as of of a coder as gp4 is um it makes a lot of mistakes because um kind of the space of the request that user can can ask for is Extreme big and uh programming in like a gang engine is no easy fee so um it make a lot of mistakes and what we found that is that there are a lot of for example repeating mistakes that we will still make even if we caution it against them um so what we have finally arrived on is this design choice of introducing another gbt we call the inspector which checks the Builder's code before it's being executed so when the Builder has come up with a draft of the code that he wants to execute it this draft is first passed to to another GPT called the inspector and the inspector will look through this code and see if there's any like potential errors and in this case uh it has determined that there is and then it will kind of refer back the um uh suggestions to improve this code back to the Builder and the Builder will try again and this iterative process will continue until the inspector's happy so after two times the inspector is happy and then the code is passed down to a compiler execute it and then uh you will do what it supposed to do so so that's the general framework uh again lot of lot more details in each of these components I'm happy to answer if you uh uh want to ask about them one thing I would highlight is that we we have done some quantitative experiments on all of this so one graph I will show is that um basically you can compare this kind of multi-agent or multimodule orchestration to just using a single GPT and what we found is that uh and we do this in a lot of different scenarios across different requests and promps and we found that is that if you use all the modules that we have previously mentioned uh you can achieve up to like three or four times less of an error rate while carrying out the user request an error is defined as so the user gives a prompt and then the error is when if the uh when the code is being generated um the code either contains runtime or compilation errors yes yeah that's is a very that's a very good point so that does not mean that it you did do what the user want you to do like completely but that is very subjective so we opt for like a more kind of objective a conservative way of um measuring this yeah yeah right so but uh but yeah but towards that point if it if has a run time or compilation error then we can always say that it has failed so that's why we use this metric uh sa is the uh uh C analyzer SL is the skill Library I is the inspector skill librar fun yeah yeah that's sound it um uh it also has like for example when I show you like the the whole VR kind of set of demos there's some like additional kind of like apis with the VR libraries and you may not like know about those or like um call those very consistently so we provide an in this uh external scill Library is the insector build Builder module interaction is that like an adversarial or and also the question is how does if one them makes a mistake how does the other one sort of know that's a mistake yeah yeah it's a is a really great question um so uh let me let me answer one by one so uh so the first question is the inspector actually is um the the underlying model is all just GPT they're all the same what's what makes them what makes them different being one is inspector one is Builder is that they have kind of a a preamble paragraph that I use as part of the prompt uh and for the Builder I the Preamble I say okay you're kind of a coder that's supposed to write the code that executes the user's demand whatever that is and for the inspector the Preamble I wrote is something like um you know you're a careful inspector who look over a piece of code and determine it has any errors and stuff so so you kind of give them different personalities P like so to speak by doing this um to your second question it it it's this this is a very good question basic which is basically if the Builder didn't know that he made a mistake why would the inspector know right because you're the same model you're all gbts right um I think there are a couple there a couple ways that I can answer this um intuition wise I think one is that which this this one um people have argued with me uh with it but but I think I think by and large is true I don't know what your intuition is which is it's easier to check a snippet of code rather than to write it this but I think even taking a weaker stance even if that's not completely true I think at least checking the code um has different kind of error modes uh like like different ways that you can go wrong compared to actually generating the code so we're kind of hedging different kind of sources of er error together ex sense to me that if you're prompting it differently inter yeah yeah uh but it's a very good point that ultimately neither of them know if this is right or not but um it's uh yeah but we did find that this is actually like pretty uh robust ac across different scenarios um give it back and say hey this was uh which part Spector it back to uh so it will send a verdict uh which is success or fail and it will send the suggestion as to how to change the code um to uh to pass so I guess actually will send a reason and a suggestion um so yeah yeah exactly yeah exactly that happens a lot yeah yep y I think there's some kind of my interpretation partially is that there's some kind of attention deficit in the Builder is trying to do too many things at once's yeah yeah I I mean um when I built this my intuition was not very mathematical or rigorous but I think there's definitely like more work to be done in this respect and perhaps something rigorous you can say about it um yeah okay sorry yes so I'm wonder if you could put the the illustration of the room right so know when you're building Unity models you either have it so that they they actually know that there's a boundary in the boundary goes through okay I think I know so now in in asking for the bear to be twice as side that bear could fit or that couldn't fit did you play with that at all if you make it aware of like you want to make the bear three times as big yeah um so uh you you can uh I guess I'll answer it this way so uh uh what will probably happen if you if you ask exactly that make the bear three times as big it will make the bear three times as big and it will go partially will go out of the room but if you don't want that behavior you could maybe say make the bear three times as big but mind like the boundary so that it's still in the room um I think it will try to check some kind of um Collision or overlap between the meshes and try to like uh have a little uh room between them so the other the other question I had your bar yeah so this this answered a lot of questions essentially the leftmost bar is just raw raw GPT code generation right yeah raw just have it generate Unity code and that's what you get exactly tons of errors and stuff like that tons of Errors yes you added these both from prompt engineering standpoint and then your tal model you've added that yep yep exactly yes amaz game comp but you can create a video game now the LI of the of the of the of player and according to his thir Advent really not even a procedural generated game but I gave that the first designs and programs as it plays along yeah yeah yeah just that's the dream wow yeah mind loone um we just had a speaker today this is a um now Co side talk Mato he's a PhD and in the comary depart Masters in the comary department he was talking about using the same thing that he did and um Builder and Checker but he was using it to check for um adversarial attacks in terms of uh C passing the guard rails gener uranium and getting structures for that so he was Da chaining combine yeah yeah I I think it's definitely a trend to uh kind of for PRP engineering techniques to move towards more kind of multi- Asian interaction uh Frameworks which I think are [Music] yeah rules things like for instance with the S and the Collision uhuh uhuh to like continue correcting for mistakes yeah yeah yeah yeah so I would say uh so you can tell me if I answer you correctly so um our Our intention of uh the inspector is mostly in terms of kind of uh mostly in terms of runtime and compilation errors but also maybe checking some like typical behavioral errors but for the example that you mentioned which is kind of more qualitative like the particles that bouncing but they don't look visually quite as good as you would like right those ones we're not really checking for because um if you think about that space of problems they're very subjective right so um I think if you do want to check for that space of problems probably some kind of vision model will be nice um yeah yes yes indeed not not the case when we did this but yes yes now it does say is the waterink yeah yeah I I would love to try this honestly if they give me the keys build aame I is this a sustain way to write C yeah great uh great question um it's certainly not a good design principle from software engineering there's no modularization or anything like abstraction anything like that is out of the window software um well what I would say is at least we made the framework kind of persistent like it is theoretically possible like you could save all those scripts and they're like persistent in that way and you can iterate on your previous war that's possible whether it's good design to do it like that yeah then we don't we don't have anything in that respect yet so with your existing framework if I just put in one prop and said make a Gollum video game would it be better or worse I don't know um that's a good question we have to try it [Music] out all this stu oh um yeah that's a good question so we um when we did it we just used like just we just coded it ourselves there's no kind of orchestration package that we leveraged but but I would say we like we heavily you know borrow inspiration from like cemented kernels and that is that because Unity there wasn't anything to use to apply to Unity yeah I think yeah so yeah because we're embedding a Unity I think uh it's maybe cleaner to just write ourselves so um to that note I have made uh like a like a complete gaming Unity before just a hobbyist so yeah I know Unity first example show umy the only preu assets that there were with the standard Unity ones no other preu assets all the like the the objects on the sink and on the stove that were generated by the like s all it is noled the the if you so these are existing oh yeah all all buil yes yeah all buil that's a great question um there are details to that in fact um uh if you just say create a kitchen it wouldn't be as good um because it will it may be very lazy and just give you some Bare Bones things now you can also be very kind of comprehensive and then say just specify everything in one prompt so like create a kitchen it has fridge fridge should go here um counter should go here their toasters on the counter you can say all of those in one prompt we also try that and that's also much worse interestingly so we buil um which I didn't cover really here but um we also build a system that allows you to kind of decompose very complicated promps into sequential steps um that each step is like small contain and much easier to do and this catching is a result of that yeah yeah it's another language model yeah yeah yeah yes exactly it just have the access to the piece that was picked out um you could do it uh that way like have the Builder have having access to everything um there are a couple reasons we design it this way the first reason is token size limits um back then there was only 8,000 not a lot especially given these um but I think also the performance will degrade um I think there's severe kind of I go back to this turn again attention deficit that the Builder has if you just feed this whole thing as well as all the other guidelines and requests that you needed to do so yeah um I would say uh to that I will say uh certainly if if if the token if the token size just exceeded then yeah that's a problem um but I would say that even like if if the token size is fine then it's actually very good at picking out even SC even like extremely complicated as long as it fit uh fits in the token size so like it's the the so it's either completely broke or it's like pretty much perfect so I was pretty surprised actually yeah the the thing that's really fascinating to me about this is seem obvious but it's doing it in this virtual environment virtual reality so that it it's the language model is manipulating well is manipulating objects that have that can have sorts of properties that that initial scene that you just showed that's a you know 300 you can you can that's you can immerse in that where it's not a just stupid picture you can you can have it cons conceivably POS things you know it can know that hands Got 5 right yeah yeah exactly exactly knows it for y yeah can manipulate that yeah yeah no for sure yeah so fasinating yeah no that's definitely one of the kind of the uh um should I say main motifs that we're working uh along the way which is like this is to us this is a more kind of sensible or intuitive way of generating because the language model actually knows what's going on instead of generating an image well you got to dig into the neurons to see what kind of representation and features they learn but like that's like to me a little bit more inuitive in this way like it actually knows that he manipulating this part and you know it's more interpretable so this person on the call who couldn't be with USIS is bursting he thinks this is the coolest in the world I don't know if you know Eric is but he teaches the game courses oh okay so nice Okay okay but if you like games this next thing you're going to really like okay I um okay so this is the second part of the talk I'm running out the time I'm going to be fast so this part like I said before um it's you can think of it as like a particular functionality of the first like overarching framework what we'll do eventually is that we'll use the framework we built to generate animations in real time let me show you just one again the Bare Bones example of how like what kind of things are we talking about here okay so here the unity window again in the window there's this beautiful whale someone else made it um this is a human-made 3D model it's static currently so we want to make it move so the ultimate idea is well if we want to speak the world into existence if the world is all static that's terribly boring so we want animation and this is the part I really worked on during summer um yeah so I don't know if you saw but I basically I type I typed in the uh front um can you can you make the whale swing forward gbd goes processing this will last a minute um and then when it's done it's going to play the animation on the whale should be any second now can you sort of sketch out from a performance standpoint or profiling standpoint where where is it doing most of the thinking where what's taking the most process in time how's it distributed yes um it's very task dependent um uh for the previous ones I would say the coding um for this one uh which I will talk about in just the next slide it's going to be the part where it's generating the animation uh which I will show you in detail how it's done uh yeah so yeah I don't know if you saw but the animation was generated and it's automatically played on the whale um and uh oh gosh all right well yeah so I hope everyone saw that so now I will just talk about how this is actually done um so I I will remind you again that all we're using are language models no vision model no like uh like diffusion model on like time series not nothing like that language model so all we can do is to manipulate text right so how is this possible Right text and motion like how is how how are those related well it's actually pretty simple so I'll just give you this example so on the left there's this motion of a whale and the only part that's moving is the tail right so we because we're using language models we need a way of kind of representing Motion in terms of text and then kind of process that text with language models and eventually generate the text right so um the the way we do this is uh actually very straightforward um the motion on the left I put the text representation of the motion on the right here and let me just kind of break it down a little bit so what it is that so there are two rows here and in each row the starting part this part uh is what I call the drawing hierarchy or the joint name so what it is is that this whale has a lot of different joints that you can move right so like this so the the part that's moving is kind of the tail joint right so the name of the joint is first specified and something like the following that's just its name in the whole tail kind of bone structure and the next thing is that if you think about what motion or animation is well it's really just you just have to specify how the orientation of this particular joint changes with time right so at time Z it's like in this orientation time one is like this time two is like this and then if you interpolate between you get a motion right so that's what this is so following the joint name is basically just a bunch of vectors specifying this trajectory of orientation change and each of those vectors is a five Tuple the first uh the first entry is just time and the next four is uh Quan determining the orientation um if you know on computer Graphics that's just the best way to represent um rotations yes so you say um XYZ and W for quatern on values so I'm assuming that is a good way to I think like consider the best way to do rotations but what's the reasoning to have these actually before I even ask that question which one is your traditional one I J and K I'm guessing x y and z are I J and K and W is the one or um which one mathematically speaking would be the one I J and KD yeah I J K L I guess the last one the last one yep so and then what was the reasoning to do very interesting stuff but what was the reasoning to do the qu turny on values did you see that as the best way as alternative to like oer angles is that what you're thinking yeah I mean um it so it doesn't you could do warer angles but I guess uh I'm not a graphics expert but I I think I remember if you do Oiler angles you run into problems like when you interpolate sometimes you will do this like really awkward thing like if you seen like FIFA games EA like sometimes the the the football the footballer head will like you go like this and that that's that's what happens if you do like um so so this avoids that like when you interpolate between cians it's more kind of smooth like visually um yeah uh gimbo lock that was the term you can look it up yeah um yes are kind of bad how do you uh great question um the unit constraint I guess were not well that one you can I guess just normalize um you can do like a postprocessing but being bad at math is definitely a thing um I will talk more about that um well actually I will talk a little bit about about it now so um a couple things that really helping that respect one is that you probably only want to generate the cenans at Key frames and interpolate between so you don't want to do a very dense sampling like you don't want to like generate a CIA every like 0.1 second every frame or even um that would theoretically work but language models right now probably wouldn't and the second thing is if you notice all the flow Point numbers only have one significant digits so that's also something that helps because they're treating it all As text right so um this this way it's kind of it'll focus on the more kind of the big picture like this should be like about this size instead of getting lost at like the fifth digit that's it um but yeah so um yes uh yeah this one right the initial position where you have the angle yes um I think I know what you want to ask very good question so we also allow I didn't write it here for Simplicity we also allow I think it's called root motion so the translation on the whole frame is possible we didn't allow translation on like other joints because um those are usually like really unusual right like you can extend this joint like this that's really unusual as movement but if you want that's all also possible uh for the for the um root motion we represent it with like square brackets and it's like a four tle instead of a five where the first one is time and the next three are XYZ coordinates you did that like doing or no no it's also model yeah you just have to tell it if you want rule motion do square brackets and if you for all the other ones do yeah okay um oh well okay yeah so lastly I'll just I'll just mention so yeah so this is a textual representation of that motion so ultimately all we have to do is to Output this text and then par into motion we're done so you just need some kind of language model that can output this text and that's what we eventually did um so how we did this I will skip over some details again but uh no so we didn't do fine tuning although if you did I think it would be better well we didn't do it because we didn't have a training data um so uh what we rely on instead is just in in context learning we just the idea is just that okay let's say for the whale model right someone already have made some existing animations on the whale model right so like swimming for example and then we use that as uh few short demonstrations to the language model and then we can ask language model to generate new animations on the same 3D model um and to me it's really interesting to see like how much you can push this right if you show it how the whale can swim how much more can the uh uh language model generate so I will show you that so this is the demonstration that was used for the whale uh it's a human M animation it's like swimming I suppose I'll show you six that's being made with language models so uh different kinds like so you got Tails moving that one's tail wiggling this one's tail moving left and right that one's turning the head um I'm not sure if wh's head can turn that way but there you go um that one really interesting to me it's swimming aggressively doing this kind of circular motion with the tail this one kind of goofy so it's trying to make the whale swimming like a oh sorry flying like a bird I don't know if it looks like that but it's like doing this um I'll show you more examples on different models so this one is like a humanoid um character it's a pig uh this is the human made is like I guess walking and I made a few more so here this one the first one is very simple I just say can you make it walk but with one leg limping and there you go uh you can make it spinning jumping eating um let me show you a few more uh here like an I think Animal Crossing character uh anyway some kind of raccoon and you these two are made by human animators uh just kind of idle and maybe explaining or talking um I made a few more so here you can you show it disapproval wagon it tail uh laughing uh I've also tried so all of those before are what I what I call the F Shaw setting so you have some existing animation used as demonstration and then you ask it to produce other animations on the same model those tend to work pretty well um I've also tried some zero shot settings meaning that um you don't give any existing demonstrations on this this model and you just say okay can you just make can you just make this happen um this is much more difficult and only works for uh smaller models um and like simple ones I just show you some examples of one one is here is kind of a sakman you can walk can cheer um the other one is Like a Bird model you can attribut to Flat to Flap the wings of the bird um the last slide I have so uh there's also it's also very easy to include kind of humans in the loop for this so one thing is that once you have generated these animations it's very easy to fine-tune them uh if you're a human animator in the unity um uh editor uh the second thing is you can also if you lazy you can also like further promp gbt to do it actually so I'll show you one example so on the left here right I I prompted gbt to do bowing for the and you you see that it's actually trying to do it but the uh the direction is incorrect so uh it's going backwards instead of forward so then I asked you okay you made a mistake can you make the raccoon bow the other way uh and it does fix it so uh yeah so there you go I think that's all I have thank you very good question no I my summer R out but next that's the next thing I will try I think I really like it's kind of some kind of transfer learning I guess like very interesting right like if you if you have a lot of demonstrations on fish like a whale how like how well can you like transfer into like some other kind of marine animals that's like som