🎮 [S21] Improving LLM game UI | LLM game development

Key Takeaways

welcome to another session of the super interest super F lrm powered game development we name our game ready agent one um this is the 21st session we like we started all the way from zero near Zero from session one and this is the 25th 21st session uh today uh there will be only my me myself um we're going to keep building uh uh the UI uh pick up where we've left off last time uh so let's get right to it uh to just give a quick recap of what we have right now we have this more weird interface is looks like kind of stage in here and on top of that is a summary of the plot and in this particular case we are in the alien planet I mean I know there's not yet anything that that kind of vividly represents the scene but we're using tetraedron to represent objects and it's like this weird um figure to represent you know characters inside of the game but just bear with us we're going to improve this just pretend that this is some kind of pretend play right now and then you're are objects on a sort of like a stage and then people um people just kind of there's a bunch of NPCs and in this case NPCs are powered by large language models you see they started generating uh conversations and dialogues it's a little bit slow right now it's kind of slow so that's something that we kind of hopefully try to figure out like how to tackle today but the I guess the premise of the story right now like we give it this kind of kind of seed story in here as you can see and then we hope that wait hold on a second oh never mind I thought I spotted something never mind we we give it this kind of a SE the um context and we hope that um it will uh just keep developing the evolving the story on its own and which it already does it can kind of progress into chapter two chapter three chapter four and just keep going okay uh let's let's take a look here like for whatever reason right now it's super super slow the execution is super slow today I'm just trying to figure out why that that is a bit of a stretch uh this is not not quite right because okay give me one second to check something in the back end real quick looks like my LM server I do have my locally running LM server oh by the way all of this is being run on fully local llm this is like one distinguishing factor of us from from like a lot of other applications or games that they kind of trying to sim whoever is building we try to keep the whole thing fully local and fully um so that you number one you don't have to be built by open AI or anybody else for like uh like dollar bills you can just use your idle GPU to play the game we believe the minimum requirement right now is like 16 gig ram right now we're just trying to kind of just make it self-contained so that it can just keep running in this PR in these parameters and number two um I guess I um number two I guess we just want to make this uh as um oh as um widely used as possible widely played as possible hopefully so um by the way as you can see did you see that okay there was a little bit of character moving right now there the characters do can move on their own and all those movements are actually produced by large language model as well so as you can see on top the plot is actually developing and what are they doing so they're moving towards uh let me see what they're doing they're moving the trail stopes upon Ancient Temple hidden beneath jungle canopy where The Whispers of the ancient civilization secrets of way Discovery Okay so I think they kind of decided to move [Music] away from the cave and decided to go into the jungle I don't know where jungle comes from it's not from the original plots um and there's also a bu on UI here where um the props okay I think they're all moving towards some direction I don't know what what they're doing here guys we're in trouble an ancient Temple about to blow oh okay hey what's going on I don't even know what's going on like the plot just keeps like developing on their own uh okay as they escape the Temple's explosion the trio stbl upon a massive ancient portal hidden beneath the jungle floor ping with an otherworldly energy I keep getting Amazed by this every time you run it it's it's sort of different but uh we probably have a few bugs in here because number one the state like the the the the subtitle is not being updated and number two uh the props every single scene we ask the LM to generate different set of props but right now um doesn't look like yeah we have object massive ancient portal person Jack Tracy and Indiana yeah those were the original kind of uh characters that got carried over uh but you can see here the object is kind of different from the the initial set of object right now like started hallucinate like different objects uh but for a for for some reason the front end UI is not actually updating the new objects along with their coordinates it's it it's still stuck with the objects from this first them so this is what we're going to fix today okay we're going to try to face so uh all that said let me just kind of stop stop the or maybe actually just keep it going for one more time these things are really fun to watch I just kind of help it so how is the story developing right now as Morgan approaches the ptle its energy surges count during a swirl swirling Vortex that threatens to engulf them nice very nice we just have to guys I think we should be careful we I don't know what's on the other side of that portal I think we should just you know just fix the UI and make make the whole in the back end a lot of things are happening but it's not actually being shown right now so let's just go right to fixing problem okay so I'm the first thing I'm going to do is go into this props manager file oh uh I think it kind of failed in here let's check real quick what the error is so the error is uh oh here there's an error here now interesting it talks to a Target yeah this is the wrong hallucination by the way oh okay it talks to Jeremy Jeremy help but for the Target is wrong okay it shouldn't have Target talk doesn't have Target I think we can can tolerate more about okay we're going to fix that later action is wrong to and go to this coordinate all right and all right let let's um let's try to fix this so um let's kind of try to figure out why the objects are not updating okay so the the reason objects are not updating is um maybe they are not being propagated back to the front end okay so the first thing we're going to check I think I checked this last time but just to make sure I think it is actually it is propagating all the state back to the front end it's just that it is not updating but let me check real quick okay I'm going to open up the web socket connection I'm going to restart this game because this one kind of fail all right um open this this game oh wow this is a new okay let it wrong oh sorry I got to start over my bad because um I didn't open the the portal like the that developer tooling time I do need that uh okay all right so let's see uh first time like a lot of data is gets sort of sent back I think this one is sending me back the game State I believe Yep this is the game state so what I want to see I want Let It round for a little bit and then let I'm going to see if the new game state is being prop okay there's another game State okay the game state has props and everything so we do have game state but for some reason it's just not um it's not actually being it's not actually being um set properly do we actually have the logic to okay I I I see why because the game state right now is entirely being set manually okay I see what the problem is well just to not to get into too much details of this but in order to fix this we're we're kind of mixing right now the the front end game state is almost the same with the back end game state but with some differences uh the front end game States has some additional properties such as current position moving or rolling these are only frontend Affairs but they got mixed up so both the back end the front end schema both have these properties which is not very good idea based on like you know from a principal standpoint so I think what I'm going to do first is to move this front and only state out of the shared game State uh and just make it only like kind of isolated front end kind of state dictionary kind of things uh the next thing I'm going to do is uh is then I'm going to use uh Liv Stacks buil in facilities to automatically propagate the state from the back end to the front end it's a two step approach yeah so that's exactly what we're going to do so let me get started um in order to fundamentally change because I I like to kind of start changing schema like from schema first that just gives me a better way of thinking about these things so I'm going to just start by say we don't need these in the shared State we just don't need this yeah yeah I in fact I even made a comment for myself before that says this is front end only we're not going to do this okay so I anticipate that the back end would not be complaining about this uh let me me check real quick so workflow. uh workflow. liflow Define huh can't quite find it okay workf flow.com okay it's workflow. conversation by the way this should be really live flow live flow our conversation because we recently changed the our our name mostly for marketing purposes because everybody's using workflow but we're doing live flow just make things slightly more distinguished dis distinguishable all right and this this I'm going to change that to LIF flow okay next thing is that um yeah this is not complaining this is fine but however I do see that oh oh wait what's going on here oh live flow for some reason I correct didn't work very well so this I'm going to change to live flow and this is live flow and this is live okay this is the test case fix the test case as well now Fran is going to start complaining he kind of start failing to to find these coordinates which is expected which is what we want now yeah we don't need this like moving rolling and all this just going to get rid of this um yep yep is this moving rolling yep we're now going to have this instead we're going to have uh some kind of unified lookup state in me in the in the react state memory what I mean by that is we already have something similar called speech by character which kind of captures the speech of each person uh I'm going to do a new new thing called uh state by prop set state by e e e e e e e e e e e e e e e e e e e e e e e e e e e e e okay we're back uh I'm not sure if the I kind of kind of lost the window in here probably didn't hear me for a very long time well we just got to keep going says connection refused interesting all right so let me try to play this or maybe just it will just work okay if it doesn't let me check oh what Arrow do I have now cannot read property undefined reading data oh I see so in the beginning there's no data so game State could totally be a now we got to sort of handle the situation here uh props manager this could potentially be very now be totally now actually H let me think how I should handle this so in the typing actually I should just make it super explicit that this could be undefined output this could be now and this be oh I see so I'll just say if job job and and game State okay I I think there's no escaping of you know G State just give you the question mark and then here game State equals State D question mark data all right I think this should be safe enough it's a little bit awkward but it should work all right all right this is rendering let's see if the game has kicked off all right initial state was fed but uh nothing happens sorry give me one second okay maybe uh just I'll give it a little bit of moment wrong oh what happened maybe this has to do with my internet something is not right I think my internet is kind of failing right now all right uh for whatever reason I do have feed input here and it's feeding the input but for some reason it's not um initializing the job anymore the job comes [Music] from wait there's nowhere that kind of initialize the job huh so where do I kick off the job