AdaTest

Key Takeaways

building good natural language processing models is hard consider a team building a model to detect to-do's in meeting notes they fine-tune on some training data then use held out test data and get great accuracy should the team celebrate and deploy the model no models that work great on standard test sets often fail in the wild you need to test your models beyond simple trained test splits but how there are two types of approaches those that help people write tests and those that automatically test the model the good thing about people is that they know when the model is right or wrong but writing tests is slow in contrast automated methods are fast but they can only test limited aspects of model behavior we propose a human ai partnership called adaptive testing that combines the strengths of both humans and large-scale language models like gpt-3 where the ai generates lots of tests designed to highlight bugs while the person decides if model behavior is right or wrong here's how it works you start by typing in a few input examples in a topic representing an aspect of model behavior such as to-do's that are in the past tense if the model is good it will probably pass all these initial examples then you can ask the ai to generate a large batch of similar tests in the current topic sorted by how likely they are to reveal failures you can then label a few of the top suggestions that are in the current topic and then repeat the process this results in a growing set of organized tests optimized to break the model being tested the ai can also suggest new topics and fill in empty topics enabling you to efficiently build out a large tree of tests often revealing new aspects of the problem that you didn't anticipate the team now knows something important their model has serious bugs and we can fix these bugs by adding the tests to the training data and refine tuning our model but has this really fixed all our bugs no it turns out fixing one bug can easily create other bugs we must re-test after a few rounds of this debugging loop it gets harder to find new bugs and so we have more confidence the model will generalize well in deployment in fact when a real team at microsoft applied this test fix re-test loop for a real to do detection task they were able in just four hours to double the in the wild performance gains they had previously achieved over the course of months check it out yourself