🎮 [S26] Consistent LLM Outputs | LLM game development

Key Takeaways

[Music] hello everyone hello Emily hello cat oh cats yes all right so what do we do today we're gonna keep debugging the LM uh the cor LM engine um as we progress we find that the quality of output starts to deteriorate significantly like it just get progressively worse uh we tried many things just to kind of tame the the shape of the output to know OFA so far uh mostly I think because we we kind of we we run into some kind of trap we probably need to get ourselves out out there by by just shake things up a little bit maybe just fundamentally change I'm prompt uh to achieve that we're going to try a few things uh first we're g to I guess set up like a console kind of thing like test B I think we had that but like we haven't run that for a while and then we're going to going to now we have all the prompts and that's got spit out from the output so we're just going to you know uh save all those prompts and then just try to test them out manually and start improving them maybe the other thing is we go back to instruction mode instruction mode has been working like the Mel instruction mode has was working until we kind of abandon it in favor of some kind of few shot chat mode which I don't even know is a thing but um otherwise another thing we could try which we got inspired from one of our interns is just to just throw our faulty prompt into chat gbt and see what it come comes up with or Cloud oh actually I think I think cloud has some kind of meta prompting thing going probably Yes actually like okay let's just just explore these things okay uh so I guess um let's run a test script first so at least we can get all the prompts printed out yep uh let's see is this script still usable what script is still usable the LIF flow. conversation. test.ts if the yes as long as there's no like uh compilation error uh test TS uh where is the Run TS script oh got to go into game oh you're already in game all right so what's happening right now oh looks like it's still working okay oh never mind let me just copy the whole thing cool let's see what we can use here um actually a better way would just be to go to the dashboard if I can find the dashboard link live. deev awesome uh let me see if I have a good browser window here all right that's a nice browser window all right so where are our data flow for a character worker this is where the prompt is coming out are these still clickable maybe they're not anymore uh maybe they are ah yeah they are oky doy I'm not sure or if there's a way for you to copy and paste those things uh we'll see okay does it no h if I I just drag this whole thing is it going to copy everything oops not good uh ideally there's [Music] some there should be something printed out here as well wait wait how did you get that I missed that did you did you just copy and paste no I didn't copy paste it doesn't work yet so I guess that's something we'll have to add okay why does it say thoughts to you uh I have no idea it's just some sort of in it script some kind of virus okay cool jeez uh let's just make sure this is the right prompt we want user um context Story begin scene okay user Story begin scene Okay cool so this is our character prompt now we would want our good old pal CH GPT to take a look new character actions next activities got to sort of describe the problem problems and then so what is the problem oh maybe just tell it like oh um I I want okay let's just give a try because it needs to be aware that it's we're operating on a smaller model number one number two is the schema has some issues that's another thing all right we're using a small LM and model and output scha doesn't always okay okay it is trying to generate it just doesn't wrap text doesn't wrap text so you all have to keep scrolling left and right okay uh actually this is hard to see what changed explain to me what has changed yes improve Clarity the instructions were concise but could be cleaner improved added detailed instructions about the Json structure reason to ensure explicit Json structure requirements okay the JS schema was can you make make that everything bigger because is it so not clear yeah for some reason no that's clear okay uh okay let's let's try then let's give it a try inclusion allows for no value mandatory talk action context and examples uh you know what I'm just going to try um there's no easy way to check the diff but at least we could take a look here uh let's see what we might be giving we might be giving it a hard time because you know um that's not how it usually act process prompts and such yeah this conversational style is pro probably problematic okay let's try let's see what what improves all right how come it didn't even generate the whole prompt because I think it's confused it's got It's got it it's got to juggle with the prompt the formatting and proper you know new line characters and all that wrapped within a jsaw an array of jsaw that's a lot of things to to take care of what LM um what was the rationale for existing on a Json structure again wait what do you mean existing are in J like why can't we use plain texts you okay this is not a prompt that you actually send to the LM this is actually the thing you send to the open AI API is it different right you could you could just copy and paste the whole thing concact concatenate concatenate together in in in in the format of one input one example one input one example the actual input the end okay nothing I don't think anything stops you from doing that okay but either way let me copy these back to our gen prompt should be gen action prompt here so we have the user the assistant the user the adant again and this is where we want to replace uh did I copy it correctly let me see let me see uh it shouldn't be displaying where is the diff okay let's see what changed I don't think anything changed because you copied over from that there's no like backtick string can like like there's all the uh Slash end not notation uhhuh if if here you see any like sort of string uh templates sorry back to string that means it's not copied over yet uh okay this one is copied over but you got to reformat the whole thing and do do the it's totally not readable now oh okay let me uh is there any easy way to yes you can do that really no no you got to check the ex the regular expression uh switch oh no not sorry not not you got to turn that off and then oh actually the way you do it is turn on the regular expression switch and in a search you do SL double slash in no double slash SL slash in yeah that ah okay and then you just got to manually replace all the all the double quote with back tick yes include subject reflection activities okay let's give a try but I'm really not sure what just improve but let's try this yeah um H where should I nope I feel like it should be here but how come oh okay it's complaining about the next thing which is fine subject next activities hopefully this would work kind of looks messy though content uh what is this still complaining about SC down yeah that one but then what was the Orin you replace the the original I think you need a comma right if you missed the comma uh comma there was a comma I don't think so that was the auto suggest really yeah yeah I mean objects in scene shouldn't there be shouldn't they be eliminated by commas uh let's see oh you are absolutely correct okay okay okay try let's see what else oh actually yeah this one doesn't need the comma okay okay so should be working now let's just double check the diff log okay looks like it's just some spacing change uh and this one was okay some [Music] reformatting Ro user all right what else oh okay so essentially it tried to make the schema consistent by allowing them to be null that might actually have Merit okay interesting okay so the rest shouldn't be changing the other thing though is we don't know the Baseline like is this thing completely you have the latest code right uh yes I I think last time it was like basically completely broken but let's Che let's see and if you have latest code why you still printing out the the the the entire prompt I thought I commented that part out [Music] uh okay something still off here right is see the see the error what's the error oh okay so it's still trying to put it under next actions I thought we we got rid of automat mention of next actions wait do you are you sure you have latest code Yeah a different branch did I forget to commit last time oh goodness I forgot to commute everything did you make a lot of ch yeah I did but I basically render the whole thing completely broken do you want to keep the changes from last time some of them okay for example I got rid of all mentions next and what not uh hold on next actions next activities uh okay let let me commit my my thing but then it's going to be in conflict with what you're doing it's all right might I because all I did is is uh well actually um nah no I I think you should you should just keep doing what you're doing um CU my my attempt was kind of pointless last time okay so what I did um I think one easy is okay so this is this is fundamentally broken to be honest you can you can fudge words and and make it nothing do this and that but this is still fundamentally broken right um we might as well we might as well do this we might as well um just kind of start from scratch and um um give maybe give the gbd a few shot examples those those are hard right those are things we want give them the photo example give them the EM cat example and we ask this is what we want help us construct the prop uh one thing I'm rethinking though um I guess if the goal is to make the game more playable should we even try to have all these complicated actions um most of the time before when we were testing the script it's essentially generating one or two actions one of those are one of those is actually talking so if we allow it to generate one activity and then say one line of message for example so essentially your activities won't be an array of object it would just be um it could be as simple as a strin for example attack the bear or hide and then you'll have an additional property saying message or speech or something okay let's do that because because we I think we're spending enough time on this for for doing something that's kind of pointless anyway the only tricky part with that would be how do you decide on destination I guess it could be an additional property for example movement or I mean if your actions move to then what's the reason for distinguish a Target and destination just just Target and move to Target oh the reason was Target uh we target is more like an object destination and coordinate but in never got confused by that part uh that was okay right okay um I guess if it's never confused about destination we can make it a single object so removing the complication coming from array yep I agree we try to scratch it doesn't work let's go back and simplify yeah so I believe the easiest way in the Liv stack world is to change the schema definition yes as in any type Square project while very while typed type script project I guess so history entry schema instead of and I suppose we can also change this to uh action and we would also want message is that simple huh uh hopefully and then here we basically take out message is it simple enough can we do s poer ah I suppose we can do Target as well so it would be also is reflection the right word is be should be like thoughts yeah I was going to get to that next all right so if we take out destination as well do we even need this well based on let things step by step paper yes giving a rationale does improve things okay among other research I guess I guess we'll make it nullable you just um no can we can we just just not make a notable until like we hit the wall cuz like previously if it's aable it normally means something's wrong it unable to to to provide a reason so it's not a thinking yeah it's not legit you know because it rarely it rarely is incapable of generating some thoughts M yeah okay so now is the time to fix everything uh we'll fix the front end in the end let's try to do this part first recent history uh oh so we'll have to modify some other schema as well wait recent history do you have to though uh I don't think so okay so I should have changed everything why the way why do we have thinking in recent history uh uh just I renamed the reflection into thinking I guess thinking is an easier word than reflection then still do we need to include thinking to recent history uh it is technically there but it doesn't get factor into the prompt I don't think okay right so let's fix it okay recent history ah that is because I changed these thinking and action you know what to make it easier for future changes I am just going to do easier than this way cool history and trees why is it so complaining oh is it because it's missing message which should have been generated by data D message cool all right so this one was relatively easy to fix okay now let's come to the front end for const action of ah one sec [Music] so now we don't need a for Loop anymore for action and and that would just be const action equals data dot data do action and oh it's actually trying to append the conversation which means we don't even need action we can just get it from data data do message cool so that part is done but then the heavy lifting is updating the prompts again uh actually do we even want to apply the changes that chbt just did I almost want to revert it back yeah sure yeah one pointless like um if it's still generating issues then we'll try to optimize The Prompt then so you seeing wait this is G props which is not what we need here we need gen action your helpful assistant your job is to determine the next action in message Json response now we have the thinking and we also have action oh this is where it gets interesting it's flatten out I okay uh I guess now we don't need message but we so need it here okay uh my my question is though doesn't this lead to confusion because 169 and 170 you both have actions yeah that's what I just realized action action schema doesn't feel quite necessary so maybe this would just be type for example why why do you even need a nested object to begin with uh again don't you need the coordinates in a separate attribute so you can parse it somehow yes but action and Target these two properties does not need to be nested into an action object ah okay okay I see your point so it would just be this is just as to unnecessary um complexity yeah yeah now we essentially have Action Target subject thinking message State changes yep that makes sense this shouldn't be oh wait uh so we just fixed cops manager let me uh you have three problems right now yeah yeah let me just clean up the terminal a bit um it's kind of like a personal I don't know pet peep okay so here we would not need these as nested they will just all get flattened out instructions put all activities yeah that's that's out got to rewrite this part actually you each subject allow activities the The Prompt could you know the instruction could determine the next action is is the action notable it could it be like if there's nothing then just just output now uh it should still populate something but what if someone just want to stand there and just keep talking then the action will be stand fair enough so there's always in the action is not notable mm okay use one of the subjects allowed activities as action is that clear you have like one one thing you have activities and the next you have action I guess we can change it to always be I changed that to these two actions subject allowed actions subject allowed actions or should be just allowed action subject you what I shouldn't even be doing this myself what you you going to gbt it again sure I'm trying to I'm trying to teach a smaller version of your brain to do my bidding but I fail here is my failed attempt at parenting please instruct as to how I should proceed so that so that it does not completely grow up as a total disaster the end okay uh sorry what were you saying I was trying to distract you okay this is kind of annoying I wish I can wrap text wait wait wait no no no no okay you just ask it to you know not use uh Slash and use it just asking to use back tick maybe oh instruction space f in the following Json schema fing the following Json schema with the subject's next action thinking that what what the is what is that oh this is the prompt that is not Json sure okay whatever it was uh so should all of these go into example response thinking Action Target message I successfully changed the life title to Parenting fail seeking guidance for raising kid goodness make we paren I actually want to change the whole thing So based on the context provided fill in the following Json schema Jason schema uh okay now this is the system prompt user would be context objective object SE SC recent activity log subject name subject allowed actions which would just be walk toad Fe talk uh should you uh further clarify that if it's walk through or or walk to it it must always be a XY coordinate uh let's see one step at a time so this will be the next thing to try I mean we must have it or otherwise okay let's let's do yeah you should have something working first before adding something else I'm thinking response would just be J stry I am silvered thinking [Music] and action oh actually I don't need these Action Target message okay so this is one and then again user context objective object in scene recent activity log subject name subject load [Music] actions shoot a tag hide talk and response this would just be thinking action hide Target null message oh no okay so now again we'll have recent activity log con actions uh why do we need ah so this will just needs to be maap in and we can just do history dot uh okay so we want the format to be return history. subject history doaction history do tet and then ah h H cool so now we want I guess we don't need talk but we do want oh also I changed this let me just quickly change this before I forget yes uh so where were we now we have walk to run to Trump examine operate punch kick yes that's basically what we have add subject next actions shouldn't that be response oh yeah okay so now this should be all good all and if we were to run the test script I suspect we'll have to change something here I'm surprised that you you wouldn't have need to change anything in the game engine uh let's take a look if you open it up it's going to be hell yes right that that's exactly what I was was was concerned about so if you if you do free form because you do not um here's thing you need to handle here um Target could be a coordinate or could be a string right right if it's a if it's a yeah this is complicated so can I do if okay how about we do this how about we just instead of x y and all that we just have a string with you know two cordinates uh with a parenthesis separated by comma but it's a string yeah then we we we worry about that later so then let's go back here because now we only want a string right uh which means ah let's keep it optional for now yeah because because you don't do anything with jumping and all that right but then I also want to make it so that walk to five did I ever give it an example where Target put the coordinates and then yeah right okay doing that okay so let me just go through this again now we have the Target and we have hide so this naturally doesn't have any Target recent activity log h. Target response okay so far so good let's go back to fixing the game engine now if Target I think we'll want to try I probably use some kind of red Jack patching yes uh if Target all maybe add a common n no you don't you don't need that just uh um do a comment and say um test if Target is in the form of X comma y yes uh typ script comment okay okay brain freeze um if Target follows the format of oh actually shouldn't yeah I was um shouldn't give you a little bit more guard rail like instead of just like a naked number common number you brace it with yeah maybe bracket or parthis that so that it's easier to parse and also uh there might or might not be a space in between the comma like like right after after the comma that's also another thing to consider uh I would do no space I would probably do no space use reject but I'll print out the target just in case so we can take a look use but in here like you're pretty close just do um like slash S SL ask question mark I think that do uh right slash S yes Aster yeah that work uh as means means zero zero or more that's fine yeah let's do either zero or one uh and I don't want multiple numbers it should just be oh actually digits never mind the rest looks fine okay Target but then it's surrounded by bracket by the way uh it is yeah so then you accordingly need to change the prompt oh no no I think these are just capturing the group no well right before that there's slash bracket stash open the bracket uh right which means are you concerned about the parenthesis I mean it's already this is fine you just need to change the prompt let me just quickly see uh o do we want to consider dots or always integer yes right um just one more thing wait how come this didn't get captured the minus 10 comma any wh space character matches the previous token between zero and one time uh how come this didn't show up it should have captured minus yeah minus because you don't have minus no no I mean the case where you have a white space how come this didn't get the second one or I can't quite see because it's kind of small oh okay so you is that a white space or is that a minus sign I can't I can't really see that's a white space that's a space so it should be a uh just try to instead of question mark just try Aster that's more broad that's more General nope let's maybe debug this quantifier zero or one zero or more zero or more greedy uh mattres the character literally okay so this is matched now match any y space character okay so this worked did I have to set Global ah dum dum okay so that was originally fine okay so now we do want to add potentially a minus here and a minus here and we could have I think you need to escape the minus uh I don't think you have to Mattress a character you don't have to but sure okay if you do that then okay I guess it doesn't hurt it's a special character is special then safe it's safe to to escape it even Dash is a special character yep Dash is like say A to Z 0 to 0 to5 so oh okay okay okay makes sense so now we definitely have to escape the dot this is probably uh okay sorry hold on one sec is I know we're doing it but is it easier to just uh just allow for either string or or bracket of of two numbers let's just do string let's just do string uh in this format so now it would be this uh so if Target match X [Music] Y split do map looks okay from location to location okay and then else instead of throwing new error we want to do return uh wait where is this nested El if okay so it is everything up until this point okay so if Target match then we do this else we do this pop look up okay now we don't need the Eli Target okay um it's always like good to include that just just in case I only have one condition I don't need the else if Target that was from before if destination uh but the what what if in the case so okay fine okay so what is this doing from location I guess this is like a default case um wait this is very confusing so from each action and Target what you want to do is if action equal equal equal walk to or run to um so we're here if there is a target okay okay so this is basically the else condition let's just wrap this so we know and which condition is a slink to okay so now if action ah okay now everything makes sense much clearer so that means we will have to what is this we will have to go back here to update our coordinates I believe there were two places I had to change cool what about instru um sorry what about a schema Chas on schema instruction I think there's one thing oh true in the format of maybe maybe just uh yeah oh yeah um if you do that should wouldn't that like give you a stringing St I'll give you a writing St string I think actually this is dangerously close to confusing but let's let's see um let's see cool so now that we have let's just open up all files to see if anything through there summary of course stand for example Target all message and then thinking why why does the lzard say I must I must fight to survive does that make sense uh one sec one sec now this is a tricky part I want z.0 um I must stop lizard one shouldn't they be talking to each other yeah I can just remove this wait wait why uh it was kind of confusing confusing how it just just just um change the message like make them Converse to each other and that's how or something like you're dead uh what's the premise of this well this is very very very very confusing what am I looking at here lizard War has been going on for centuries not only two lizards for me you have samples Warrior yes okay lizard Warrior lizard bad nihilist you have the Flames so lizard Warrior like fear my strength something like that yeah girl I might fear my strength I and then the not L9 72 is changes something El maybe just make it auto complete you need nice okay so nothing else from here should be breaking anymore what about supervisor huh surprise is actually not red cooler doesn't concern with actions in general right so this is our initial input maybe let's take a look here of course um we can do uh what did I okay so why don't we do action Y Target I oh y the message will be what what a what a boring day let me just another day at the store oh all right while we're at it might as well double checking the other this the other problem with this is like okay we we tried to add too many props I think it was confusing right uh okay let's see uh what what is so trying to give what's so hard about this no no no it's [Music] um it still has reflection and what I I think there there's some lingering things otherwise it would would wouldn't wouldn't even be aware of the word reflection there's a bug somewhere uh Let There are actions there are actions somewhere what if I restart my olama you sure you don't have Reflections left in your code reflection character output schema mhm what the hell is this I think this is actually being used oh yeah it's being used by instructor okay and and we are actually using instructor to produce output so it's kind of try to force into that format so this is not being used character output schem oh so where is this being that's the default output that's a default and it's being used by schema two places that's actually very important that's that's actually the schema uh either way let me comment this out so it's less annoying now this is schema okay so hopefully I can just change it here wait why do I have to redefine It Whatever thinking uh message yes now game manine is complaining again why I finish the business character actions where is activities being used const out put equals ah I was actually wondering about this earlier why would there be a map now that explains it so basically instead of [Music] having uh you just get rid of that map on line 27 for in some way right or I do this oh okay okay okay you know what let me try to make sure I am removing the right level filter so I don't really need the filter either um okay but this one does require some sort of array so let's do kind of a hacky way but yeah that will work all righty uh message it's not outputting [Music] message so sometimes it takes the action and doesn't do any write any message wait why why is it still actions yeah it is still quite messy walk to examine dock training area why uh there must be action somewhere we do a like a uh uh full full case match match search yep somewhere okay so this is getting a bit annoying how can we simplify even more what's the uh what's the issue so it is still trying to generate a whole bunch of things what if in the Gen prompt you tells it's not like we need a list of actions anyway let's just remove it and see uh you do need that otherwise it's not it's not even renderable uh let's try let's try oh I can easily bring that back maybe just just open up uh okay if still fails let me just open up to check the prompt it actually generate just just take a look at it uh right you want to check the prompt um yeah after if this fails one thing I wanted to actually I don't even um what did I wanted to do again I think there was still subject loud actions uh assistant user prompt okay right so there is no allowed actions here if there's no objective we can move it um something else I wanted to add was action is one verb subject mandatory thinking Action Target message message so why is response here all right let's [Music] see so essentially put message under response right because you do response comma uh I think few shot is fundamentally broken we'll do it this way okay maybe just bring everything back to one message then it just fix itself okay so if we and also maybe try completion mode instead of chat mode cool because it shouldn't be this hard uh I'm thinking maybe the context we provided is too much it wasn't it was at some point it wasn't working it just kind of totally stopped working but it at certain point it worked okay um let me try a quick thing it is getting even more creative what's temper uh and and and temperature it is something is fundamentally wrong because um like instructor is totally stopped working it instructor was working too before to just stopped huh that is very interesting so what if we start small uh I remember there was an instructor test let's see maybe let's start from debug so user extract schema client um main so this is the main function extract data main or extract data or this okay let's first see if the instructor is still working properly uh run yes instructor. test.ts okay so far so good uh this should not happen oh final extraction I'm assuming we use string [Music] here I'm going to commment these out console.log okay so far so good character character output schema messages where is this message is coming from system okay maybe we'll test here now let's copy the over just to see test prompt [Music] user assistant and this is the final context so it should be the block before here oh wait did I actually copied okay now context objective object SC SC recent activity log uh whatever I know this is in the wrong format but well double check on this later and then we did say response here um let me just this we might bring back in a bit so let's save it here now if I go back and run my test case oh crap crap no no no this is work uh is this using the stream mode yep J's on patch streaming mode it's working okay so ultimately it is working but but it isn't for so what's the difference between isolated use case and how it works in the system got to find out something something's wrong here yeah is fine yeah I mean at least it's good to know one single case would work let me just rerun this it should work most of like like every single time yeah it works okay so why don't we maybe let's parse through our workflow a bit I feel like it could be even our original actions right I think it should work but okay let's try uh actually just doing a bit of time check it is 10:08 next time may just commit what you have and then we just keep doing it to the next time yeah honestly I don't know uh I guess it wouldn't hurt to commit right it didn't make anything worse it's not working anyway yeah sure all right let me just quickly double check on the change history all right so far so good single test case work but not but not in the okay app need to look at work live flow next time uh did you commit no okay all right okay I guess we'll continue this on Tuesday on no Saturday what day is it today Saturday today Tuesday yes I can't believe it's only Tuesday feel like it's ages already since when since this morning it's been a long day all right bye everyone thanks everyone