Angie Chen Learning From Natural Language Feedback

Key Takeaways

[Music] um hi everybody thanks for joining um today we have Angie Chen from NYU um she is presenting on training natural language uh sorry language models from natural language feedback she's a rising fourth year at NYU advised by Kingo uh and broadly interested in two directions of research online training of LMS from Human feedback and evolutionary strategies and understanding the learning strategies and training dynamics of llms she's previously interned at Google brain and research on applying evolutionary algorithms to llms for neural architecture search and streaming disfluency detection uh prior to NYU she worked at Google search and on developing neuros semantic parsing algorithms with the selling Lab at Princeton where she did her undergrad for ML applications to healthcare um I encourage everyone to leave their camera on just for the purposes of making it more interactive and as you said you can ask questions during the talk um yeah so take it away Angie thanks for joining great great uh thanks so much for having me uh so as Alex mentioned I'll be talking today about training language models from natural language feedback and this will actually cover two different papers one of them more focused on code generation and the other one more on Tech summarization but I'll start talking first about like the highlevel uh approach and rationale for it um and yeah of course feel free to ask questions throughout um I'm not really looking at the chat because I'm looking at the slides right now so feel free to just like shout out yeah um okay so uh as you all probably know actually like language model pre-training is traditionally offline in the sense that the pre-training data is collected prior to training and it doesn't involve any interactions between the language model and the environment and it's also off policy so meaning that we're learning the task independently of the agent's own actions uh so again no interactive guidance based on the language model's own outputs um so without these types of training schemes it's a little bit more difficult to Target the model's current weaknesses in its given environment right so there's some current work that attempts to solve this with schemes like um rhf reinforcement learning from Human feedback but the preference data for that typically provides only one bit of information per pair of model outputs right because the humans are just rating like which output is better um so instead we're proposing learning from natural language feedback which is uh richer and a more expressive form of feedback than preferences and we also think that this is advantageous because it's process-based rather than outcome based so it provides supervision on like why the model out like why and how the model output needs to be improved rather than merely just focusing on the output and there's a little bit of pass empirical work actually showing that process based feedback can in general be more effective than outcome based feedback um and of course natural language feedback is also more natural for language models since they are trained to intake language um yeah and there's a little bit of evidence showing that language models can incorporate natural language feedback at inference time so like this image on the right this is from the Cod gen paper the first code gen not the second one um and in it they show that if they provided um at inference time feedback like uh in uh one piece of feedback at a time um Cen was actually capable of incorporating that feedback into its code um and also our algorithm um as you'll hopefully see is relatively simple compared to rhf and it doesn't require like an extra reward model or uh extra RL optimization algorithms like uh popo okay so I'll start out with some preliminaries so our problem setup we have our pre-trained language model I'll refer to it throughout this talk as Pi Theta there's a refinement model Pi refine and um we basically assume access to such a model in this paper but we do also like illustrate like how we actually Implement that on our own and what this model does is it incorporates uh natural language feedback into an uh an original model output so it basically comes up with a revision or refinement based off of that feedback so we have some reward function R it's just scalar valued and we assume that this is usually non-differentiable uh we have a small pool of human annotators and our general goal is that we want to fit Pi Theta so that it can approximate a Target or ground truth distribution so I'll call it piar um that fills this property that uh the probabilities output by a p star are um proportional to this boltzman distribution uh where the exponent is the reward uh for any given test T okay so I'll go through first a little bit this highlevel algorithm and then we'll delve later on like deeper into the details for each of the tasks um so to start out with you have your py Theta right it generates multiple outputs for a given task and then we ask our human annotators to give natural language feedback about the model outputs and remember this is usually done um with very very small samples of data because we know in general human feedy is pretty expensive and so in our very first version of the paper we actually wrote this feedback ourselves but as you'll see later on we ended up scaling up to um and using Serge AI instead um but part of like the appeal of this algorithm is that it should work even if you only have like a very small sample of natural language feedback and next we have py refine generate multiple refinements per output that uh incorporate the feedback into the original model outputs and lastly we use the high quality refinements to fine-tune Pi Theta uh so those that's the general highl algorithm um any questions about that before I get a little bit more into details oh may ask is the the goal to replace step two with a model at some point step two yeah so there are works that attempt to do that actually um as you'll see in some of my analysis the models we were working with at the time didn't work great at providing um the type of detailed uh correct feedback that we wanted so I'll show you some examples of like model versus human generated feedback actually um but of course the pace of language model progress is constantly accelerating so who knows we might have language models that can do this in the near future um and there's some work from anthropic called constitutional AI that does explore this but not with these particular goals okay cool moving on then so a little bit about whether or not this is theoretically Justified because I know like a lot of language model training schemes nowadays can feel like kind of hacky so we did try to provide a little bit more of like a grounded ex um justification for why this works so let's say that you have um a Tas T So t might be like a document that you need to summarize or a uh program that you need to generate um it comes from a task distribution p and we have this reward function that I mentioned before and uh this is the same goal as before you want to um get our PI Theta as close as possible to your target distribution um okay so in order to get our uh Pi Theta as close as possible to the Target distribution we essentially need to minimize the KL between those two distributions right and that's actually equivalent to minimizing this uh following cross entropy loss that I have here in the last line um and you can actually just move from the second to the third line because we just removed the term that only had Pi star of t in it because we're only optimizing data uh yes Andy I think that's correct yeah it's very similar yeah um yeah so the problem with this is um of course piar sampling from piar is kind of intractable so how do we get around this um so instead what we can do is important sampling and what that means is that we're going to attempt to assign higher weights to higher quality outputs and so we're kind of like shaping the distribution towards what we want it to be uh and so this like shaping of the distribution is sometimes don't a proposal and so how do we actually Define what the proposal distribution is is um in this case we use feedback and we can actually approximate it um and this is derived from basically just the probability chain rule um uh from like originally The Joint distribution uh so essentially we're using an empirical approximation of the marginalization of this proposal distribution over all pairs of possible feedback and the original model output and I will explain a little bit more what each of these terms means um yeah so the first one is just the probability of the original model output for a given task and that is something that we can very much sample uh next term is the probability of the feedback given the original model output that we're sampling from by just asking our human annotators to generate the feedback for us and then lastly we have this term that represents the probability of the refinement given both the feedback and the original model output and so this is sampled by from py refine okay uh any last questions about that before I move on to the code generation portion hey I've got a quick question um so can you go back one to the PF given x0 uh yes let me so this is oh yeah um so that is the feedback um and then you just sample from the you get the probability from the model that of that feedback so in practice we don't actually need this probability right because we're not Computing the probability of like Q itself which we just need to sample from this distribution okay interesting yeah so um it's a little bit confusing but yeah we're just sampling feedback from our annotators um who are given the original model output okay cool could you go where what P fun was again sorry oh yeah of course that is just any generic model that can intake uh both the original model output from PI Theta and a piece of natural language feedback and then it generates something known as a refinement which basically incorporates the feedback that was given into the original output so it's supposed to give kind of like a revision of the original output okay great uh any other questions okay cool uh so let's talk about code generation that's my favorite part um so this is covering uh one of our papers improving co-generation by training with natural language feedback um with co-authors uh Jeremy uh TCH John Anders who I think is here actually hi John uh June uh Sam Bowman K yuno and Ethan Perez um okay to start out with I'll describe a little bit about like the task that we're tackling so we used a mostly basic python problems data set which is um almost a thousand Python Programming tasks that are designed to be solvable by entry level coders so there's an example of this on the left side um and they also come with three uh unit tests so um because this data set sometimes the natural language task description is actually a little bit ambiguous so what we did in this paper is we provided one of the unit tests um as part of like the description or the prompt and then the other two are kept uh are held out and used for evaluation um yeah so this is not a perfect training scheme I do wish that we had like more unit tests or more evaluation but for our purposes for now it sufficed um and we also used a Coen 6.1 uh B model um we selected this one mainly because it was one of the best open source models available at the time and it actually demonstrated pretty similar human of Val scores as codex 12b which is pretty impressive um on this table below just sort of gives you like a uh a little um view into what its initial performance was on this Benchmark so this is zero shot the pass at K value basically just estimates that um in a given group if you're sampling multiple Generations from the model in a group of K Generations what is like the expected number of them that will be uh H what is the percentage of the time that like uh an expected number of them will be correct um yeah so we have 31% % and 63% for a k equal 1 and 10 and then we also have the separate metric that is like of all of the generations that we have which we also always generated 30 Generations like what is uh the percentage of the time that at least one of them was correct okay so to collect feedback um so the annotators were instructed with uh two specific things number one they need to describe what was wrong with the code sample and number two how it should be fixed and we also instructed them to not use any like uh entire lines of code they can reference individual variable or method names but that is like the extent of the amount of code that is allowed in the feedback um yeah so this is an example this is actually kind of a fake example because um the real feedback examples we have are often too long and the code samples are also pretty long so they don't really fit on slides but this gives kind of an idea of the type of feedback that we collected why the restriction on ability data reference code just it would make it too easy yeah we didn't want um the model to just directly copy the code in the feedback especially because um our like longer term goal for ilf was for task where providing feedback is easier than providing the refinement so you can imagine for example a case where like you're working in a really large code base or like with really longer long complex pieces of code where it might be easier to say that like no this sort algorithm is wrong and need to be like more efficient like maybe like uh I don't know use a data structure other than a list to sort and then um that might be an easier piece of feedback to give than like rewriting the code yourself okay so the next part generating refinements so we have P refine it uses the original code and the feedback to generate the refinement so here's an example um so we know that the original generated code was wrong because it didn't pass all the unit tests um and uh the human annotator um gave the feedback that um the list variable was sliced incorrectly and so the refinement basically just like adds this slice um generally in our real data we usually had like many more bugs in this to address in a feedback but it's just an example um okay lastly uh we just fine tune pyth as mentioned before um but we only use the refinements that uh pass the unit tests okay so we compared ilf between U various different baselines so first of all we compared ilf against uh fine tuning on two gold standards so the two gold standards here that we considered was the um mbpp gold programs so mbpp the data set comes with like gold like correct programs and then also we had asked our human annotators to provide human uh their own refinements just so that we have like a basis of comparison even though those are not actually required into algorithm itself um in general um oh yeah and then we also compared against zero shot inference and also learning from instruct GPT generated feedback and refinements um okay so compared to the uh in general ilf outperformed most of the other methods but performed somewhat similarly to fine tuning on human written refinements which I think is not unexpected um because human refinements should be like basically your gold standard um yeah and ilf does uh outperform um fine-tuning on mbpp gold programs um at on the passet one metric specifically um it yields like 14% absolute and 3% absolute increases in the pass at one rates and in a field like software engineering that actually really important because if you want like autocomplete in your IDE for example like you don't really want to have to sort through like 10 different completions and evaluate yourself whether or not what which one is correct so the passet one rates here are particularly important um in my opinion for applications um so sorry did somebody say something well could you explain the difference between mbpp gold completions and the human refinements oh yeah so yeah sorry this is a little confusing so when we uh recruited human annotators to provide feedback we also asked them to write their own refinements of um the model code and that's a little bit different from the mbpp provided gold code because as you can imagine there's many different ways to write a correct program for the same task and the annotators were asked specifically to revise the original model output and they're instructed to not only incorporate their own feedback but also to make only the minimal edits necessary in order to make the code correct um so and I'll talk about this a little bit later something that's interesting about that is that those refinements are probably much closer to the models original outputs than the mbpp programs far so is it not kind of surprising that like training on the gold standards does not beat the zero shot performance or only imally does so right yeah so that is surprising I was actually going to talk a little bit about this yeah no no no worries at all I'm glad that like this stuck out to you too um yeah so we do think that like that is probably because um mvpp is like maybe a little bit like out of distribution especially with such a small training data set because we controlled all of the experiments here so that like we're only training on the same number of examples for each experiment um so that that wouldn't be like a compounder and so that means that because we didn't have that many like refinements that means that we only sampled from the mbpp gold program so probably if you find tuned on like the entire MPP gold program data set like you would probably be able to do a lot better but here like you have a set of programs that are probably a little bit out of distribution um in comparison to what the model itself is already generating and so it's probably harder for it to learn with such a small data set yeah and really the mbpp um gold programs they are M like they're very weird they're very weird because they're automatically generated so most programs will look something like T1 equals t0 + 4 T2 equals 2 * T1 so they're very weird which yeah as you're saying like they're very out of distribution unless model very weird code yeah and in general I think a lot of code benchmarks like the gold code that's provided is like not always very high quality so good cool than yeah we can dig a little bit deeper into these sources of training data um because yeah I did no this this as well that merely training on just a small sample of the mbpp gold programs didn't make a significant difference in accuracy over at zero shot inference um and so as I mentioned that might be out of distribution so one thing we looked at is what if you calculate the perplexity of the mbpp gold programs the P refine generated refinements and the human written refinements but using the pre-trained Coen model um and so as you can kind of see here from the blue bars there are some higher blue bars to the right of the histogram um that means that the mbpp gold programs uh according to the pre-trained Coen model are like higher like has more higher perplexity outliers um and so like this does indeed seem to be a little bit out of distribution compared to these other two data sets um so that means that it's likely easier for this Coen model to learn from the orange and the green data sets since they're closer to coen's like own original distribution before find tuning um but that's really interesting because that says a lot about like what kind of training data is um helpful because these are all programs that are considered to be functionally correct right but they just have different styles and um a couple of them are much easier for this model to learn on than others okay and um what I thought was really cool I think somebody already anticipated this particular analysis is um we wanted to see like how much can model feedback um yield the same benefit as our sample of human written feedback because human collected feedback is great but it's so expensive and you know generally difficult to collect um so we randomly selected K tasks from the set of mbpp test tasks for which Coen didn't originally output a correct answer and prompted instruct GPT to generate both the feedback and a refinement um and so that's what we're comparing here with the uh purple and the magenta lines um or the solid lines basically um the dash line here is the zero shot performance and then the black X represents the um I left performance um and so like even if you continue to increase uh the instruct GP number of instruct GPT generated like examples of feedback and refinement um you the the rate of improvement is still much slower and it like at least for this number of K and never actually ends up reaching the ilf performance and the reason that we have to keep like basically like maxed out at 200 is because we only were using original tasks for which Coen had not generated any correct outputs so we are limited in like the number of examples we have here um yeah but of course again language models are rapidly improving in quality so we do expect this Gap to narrow but for like a qualitative um analysis of like what was actually going on here um I do have some examples of like differences between like model and human feedback so for this particular example the instruct GPT feedback is not um not correct or relevant because we specifically asked for how to fix bugs or how to make it correct and this uh instruct GPT feedback is attempting to address efficiency rather than correctness um versus basically our H human feedback was like always correct um okay so another example um sometimes what we notice really often was that instruct GPT would just uh simply restate the description of the method instead of providing feedback so that's really not helpful and so finally um I think we had a number of takeaways from this paper um in general we showed that like ilf even though this is kind of a smaller scale SC study we have a larger scale one following up right after this um it can significantly improve the quality of a code generation model even with just a small sample of natural language feedback it's not model specific and it provides um an online training signal um and ilf can be thought of as a method for generating training data that's closer to the model's original outputs in data representation space and that specifically represents the kinds of bugs that the original model generates so it's like a very targeted um method of training um but of course there are some limitations it is overall meant to be a smaller scale proof of concept study um and we didn't really get the chance to study what would happen after multiple rounds of like feedback or with multi-turn feedback um in general we discovered the hard way that annotating code data is also very expensive and also um this in general only works if you have access to a sufficiently powerful Pi refined model okay so that's it for the Cod gen side um I'll move on to summarization next but are there any question question here could you talk a bit about like the code annotation experience like we certainly think about anot data a lot um what were like how did what is like the process like you just ask surge or I guess you guys wrote feedback in this case is that right so in our preliminary studies we did write the own feedback but then what is finally represented in the paper is uh from search feedback um I think what I found from like the annotator experience is that code data is pretty well language data is difficult to annotate too but with code data what we encountered a lot of the times is that sometimes um aners won't actually check if their code is correct unless you explicitly like make available like an interpreter to them and like tell them explicitly you have to check that this is correct and also another interesting thing is that um you have to like um people write code very very differently and so when we were trying to get them to write refinements that were not too far from the original model completion it took us a couple of rounds to get that correct and we had to use like edit distance to check and everything in some cases the annotators said they actually found it a lot more difficult to um create that like minimally close correct implementation than just Reit the code themselves so is like is the feedback stylistically really varied um or is everyone forced to write in like complete sentences and like have certain code style or I other like particular formats or guidelines that make the feedback better oh yeah so for the natural language feedback um we didn't actually enforce very many constraints on style so I didn't say like oh you have to have complete sentences or anything like that we really only enforced those two requirements I mentioned that it has to mention specifically what is wrong and how to fix it and um there was a variety of lengths as well um especially because and we have further analys of this in the paper that I didn't mention this presentation um there was actually even feedback that had to address like four or five bugs at a time even though these approach generally like not super long um and so sometimes the feedback would end up like varying a lot in length depending on how many bugs there were to address yeah I just had a small question um what what happens if the refinement like model or policy is the same as the original model that you're trying to tune can that work as like what I could imagine is maybe the model knows how to take in the feedback and then write a better response it just didn't realize it from the original prompt so that's so making errors but more like maybe stylistic feedback yes yeah um so we're actually trying to explore this in a like follow-up study that I'm contributing to um I think it's an open question of like whether or not like does your Pine model need to have like more capacity or can it have less Capac capacity um and it probably differs a lot depending on like whether or not the model was like instruction tuned for example but you're right that I think we have seen this phenomenon where the model didn't originally generate the correct output but once you give it feedback it can generate the correct output yeah yeah yeah that one equation you showed with the three like um components to it like the marginal probabilities I I just noticed like if we set the two models to the same original model I wonder like what mchi was go on on the math side but either way it's it's a pretty cool Direction thanks yeah um I don't know I would have to work work the details out on that one but um I think we're trying to explore that in like a follow-up study of more generalized across domains and data sets Okay cool um so summarization um all right so so this is actually led by my co-author Jeremy um and we also wanted to explore summarization because it's marketly different from Coden in many ways there's not as precise a ground truth as functional correctness um which you have in code um and human evaluations tend to be the gold standard for a task like summarization but even those evaluations are noisy so um you have this extra problem of like how do you select high quality refinements we would hope that Pine always generates high quality refinements but that's just generally not the case um so we wanted to see how well um ilf could be adapted for a task such as this where we don't have that additional signal of like passing unit tests for selecting both the best refinements and the best final samples okay so I wanted to give an example of like how we collected feedback so the annotators were asked to give feedback about these uh multiple different attributes so Essence is like is it does it basically describe like is it a good representation of um the post oh this is from the Reddit summarization data set that was um originally introduced I think in uh it was adapted from um Steen in 2020 had adapted this original RL uh Reddit summarization data set so Clarity is a summary reader friendly does it express its ideas clearly accuracy are there is there anything false in it purpose um does it have like the general same like idea of like what it's trying the main message I was trying to get across is it concise enough and um is the style similar to the original post um and so this is a real example here I did cherry pick this one because I wanted one that was short enough to fit on the slide so this is one of the shortest ones that we have but they're generally much longer than this um yeah so in this case the annotator gave the feedback that the summary isn't accurate because it didn't mention that uh the Poster's girlfriend got blackout drunk and cheated on him okay so for the general highle algorithm um so it's somewhat similar but with a few tweaks so we actually generated our refinements this time not through a separate fine tune model but through zero shot prompting of uh Da Vinci o1 because we found that it was in general good enough um and the instruction was write an excellent summary that incorporates the feedback on the given summary and is better than the given summary um so to evaluate the like reward of each example if you recall in Cen the reward is basically whether or not it passes the unit test but here to do that we actually use an instruction tuned um instruct GPT model we prompted it with the question of whether the refinement incorporates the feedback and then we took the probability of the yes answer and that's averaged across five different prompts and we compute that for each of these five refinements so then we selected the refinement that had the highest average score um and in the paper we actually refer to this like refinement like selection ranking process as instruct RM Ensemble um and so that like highest ranked uh refinement is then used to fine tune okay so first so this plot has is comparing a few different things but let's first look at just comparing just the fine-tuning algorithms so first we want to compare ilf against uh both fine tuning on the initial summaries versus fine tuning on human r and summaries which were collected for our human annotators so initial summaries I'm talking about is um the uh Reddit summaries and then we have fine tuning on the human written summaries which we collected ourselves because we determined that this was actually like a higher quality Baseline than the Reddit summaries um yeah and so ilf does um outperform both of these and it also outperforms simply just sampling from DCI o1 which is this uh Brown Line at the bottom okay but what if you actually want to compare ilm ilf to learning from binary preferences um which you can consider to be kind of a standard approach from learning from feedback because that is essentially the approach the approach that's used in rhf so one way of learning from binary feedback is to train a reward model and use it to do best of end sampling um and that's been shown in the literature to often be competitive with rhf and so that's why we take that approach instead of tuning RL HF ourselves because that is uh quite the black box that most of us don't quite know how to do that well currently um yeah so to train the RM we first fine-tune an opt 13B model to classify whether a summary is high quality or not and then at test time we generate 64 summaries um using the ilf trained model and then we rank it based on the probability of uh rank the summaries based on the probability of being a high quality summary according to OPM um yeah so to look at the results okay so now we're comparing against two different methods here uh we have the methods where OPM simply just does best of end sampling from um the Fe me is the same thing as like d o1 here uh we're just using OPM to do uh best of 64 sampling on it and then the last bullet here is ilf combined with this technique so the summaries are generated by an ilf train model and then filtered and ranked by OPM um so as you can see here that's uh the pink line compared to the green star um so uh and since the y- axis is like win rate versus human summaries that tells us that basically um the combined method where we're combining ilf with uh learning from binary preferences actually outperforms everything else um so that kind of suggests that both methods are independently learning valuable information about human preferences and that they can be cumulatively helpful when used together okay um and to talk a little bit about like future directions um so model generated feedback as we talked a little bit about earlier um maybe we don't necessarily need human written feedback and there's a little bit of work um on this already um it's often referred to as constitutional Ai and it's from biodel um and there's also this um selfrefined work which explored a similar scheme with model generated feedback on a variety of other tasks like acronym generation Common Sense generation code readability and so on um and of course I think some interesting future work um that could be done on this is also how you can generalize this approach across multiple tasks and domains because here we did specifically adapt the reward function and the refinement models for each tasks and data set and that is um it does take some tweaking and um generally more labor that we would like so it could use Improvement on this front for sure okay thank you everybody uh let me know if you have any questions and if you want to reach out my email is here and I've provided the archive Links of both of these papers here um okay I I think there are some hands raised I don't know how to tell who well there's a classic problem of trying to clap and accidentally raising your hand oh oh yeah thanks so much for having me um and I'd be curious to hear if anybody has any like thoughts or questions in general um I can lead off with a question and I'm gonna go back to the kind of feedback aspect again yep I'm curious like one if you can get a sense of like how much feedback costs for you did you search also for this portion or yes yeah okay can you give a sense of how much it costs and also like I guess I want to try get sense of is like how much time should I invest in a single piece of feedback that might be like should I go for one really long detailed piece of feedback or like really short and quick and just get a lot of data I'm curious have experiments or insight into this yeah okay so for the cost um I only remember off the top of my head the cost for Co code annotation which was I think it was like $23 per sample which is uh quite a lot in my opinion and it actually took them an average of like 27 minutes per sample so it's it's quite laborious as it turns out but I guess the good news is is that you don't need like a very large sample of that feedback in order to you know make significant improvements and I'm hoping that in the future this will also become more streamlined and cheaper um as for shorter versus uh longer feedback I think with Coen we did observe that it was fine if the feedback [Music] um 23 bucks per s yeah I know right yeah um yeah we did observe at least for code that like it's fine if like you're not using complete like long sentences so long as you're just hitting all of the bugs that need to be addressed um and John I don't know if you might feel like you can comment on like the summarization feedback so I don't have the numbers of the money on of my head I was checking the paper right now he said that the data set collection was like 40K so we have 5K annotation so yeah it's just doing the math yes uh how much was the total 40K but we were like Gathering gold summaries uh feedback and also preferences so they were doing the three tasks at the same time because we wanted to have like the same data for for all the examples in the summarization data set uh and also we were like checking uh how costly was in time in terms of time for the annotators to give like feedback versus binary versus the gold and we found that like giving the gold summary versus giving feedback was like really similar in terms of time that for the annotators so I guess that uh like I mean we had in in our minds the idea of like having a task that it's much easier to give feedback than to solve it but I guess that summarization wasn't one of those uh so yeah it was like similar in terms of like time cost for for annotators to give like the feedback not language feedback or to give the the gold summary I think for Coen in total we spent only like7 to 8,000 um because it was a little bit more we had different sources of funding so Coen was a little bit more of an academic source of funding so that's what happened there um yeah uh Andy what you were saying about I'm actually surprised that radiolog just only got a $100 I thought they would get more it's something insane yeah that is pretty wild um GPT learned how to respond to my monkey level feedback why chart sideways uh what was that feedback too like just conversing with it as you code it can somehow handle like pretty poorly constructed feedback yeah yeah so a lot of models nowadays are pretty good at like um I guess doing what you want it to do but when it comes to actually analyzing whether something is correct uh that's a little bit harder for these models um sorry there's a raised hand um Claire hi so I work on the human data team here at coher um I just have some questions like what was the background that you saw um amongst the code annotators and did you feel like they had to think a lot about the feedback quality and data validation steps um so a lot of them had I think like some software engineering experience because you can code then why not right sorry what was the other question like did they have to think a lot about feedback quality and data validation uh yes definitely because I think we actually did in total maybe three rounds and we didn't use the data from the first or the second rounds because it just actually high enough quality feedback so as I mentioned we had to go back and iterate with um initially um sometimes the feedback would just be like oh like we're just going to show you how to write the correct code but it's be completely different from the original code or um annotators would just describe what was wrong but not how to fix it or how to fix it but not what was wrong classic yeah cool thanks and do you guys just like manually reviewing a lot of the feedback too yeah for the first two rounds and then for the third round um surge actually did help us a lot with this so I consider that as like part of the overhead of the costes because I think different annotation services are quite different in terms of cost um that they were able to help us a little bit more with uh just communicating with the annotator so that I didn't have to constantly be like responding to annotator messages I was thinking with the the code feedback project you can look at like feedback or like what features of the feedback are correlated with more tests improved as a way of proxying like the effectiveness of the feedback um like the length the question or like I don't know like if they give you specific like mentions of parts of the code to fix and the with help well that improves the models oh yeah that's a that's a good question actually so we have a breakdown of um what types of feedback are provided like in the paper of like uh there's a lot of stuff related to like dynamic programming and arithmetic um and we also have an analysis showing that like the more bugs that are addressed in the feedback the harder pie refine uh or the harder it is for pie refine to generate a correct refinement but we don't have that correlation and that's something I would be interested in looking at for sure had one idea like what if you uh purposely got your annotators to like like start with something that almost works but doesn't quite work and then you know create the actual correct solution and then get a different annotator to like give a feedback that would lead from you know this subpar solution to this right solution but because it's easier to maybe make up a feedback that does does that transformation but then again now you have antiv everywhere instead of a model doing the refinement so yeah and I do suspect that part of the reason why this works so well is because you're not straining too far from the model's original outputs yeah yeah yeah you're right yeah but that would be an interesting idea okay um well you have Angie's contact info here and thank you so much um this is super informative thanks everybody thanks Angie it was good chatting with you all thanks thanks see y oh