STEALING OpenAI models data 🥷

we do also serve people building on top of models and one thing that I think people are interested in is prompt injections prompt security that kind of stuff I feel like you the relevant version of yours your thing is can I steal the rag Corpus that it might be proprietary to a company there's two kinds of stealing there's model stealing and there's data stealing data stealing is exactly this kind of question in many ways the answer is is yes even without rag you can often steal data that a model was trained on we have shown that for production models okay in this case um in the most extreme variant we showed a way to recover training data from GPT 3.5 turbo he figured out that if you prompt chat GPT to repeat a word forever and it will repeat the word many many many times in a row and then like explode and like just start doing random stuff and it was doing random stuff maybe a small percent of the time maybe two% of the time it would just repeat training data back to you which is very confusing but like you know this is a thing that like happened and was like an exciting kind of thing and like we've we've seen this in the past yeah