Taming LLMs: Using Executable Oracles to Prevent Bad Code

1. 1Identify areas where LLMs have degrees of freedom that can lead to poor results
2. 2Develop executable oracles that can test and validate the output of LLMs
3. 3Integrate executable oracles into the testing loop to provide feedback to LLMs
4. 4Use opposing executable oracles to pinch LLM results and improve quality

Developers and researchers working with LLMs can benefit from using executable oracles to improve the quality and reliability of generated code, particularly in areas like compiler development and dataflow transfer function synthesis

💡 Executable oracles can help prevent LLMs from generating bad code by providing a clear and constrained set of goals and validation criteria

💡 Use executable oracles to limit LLM degrees of freedom and improve code quality