LLMs Lean on Priors, Not Programming Language Semantics

1. 1Investigate the role of formal semantics in program execution using LLMs
2. 2Analyze the impact of distribution shift on LLMs' reasoning
3. 3Evaluate the conditioning of LLMs on explicit rules versus statistical regularities
4. 4Apply the findings to improve LLMs' performance in tasks that require formal semantics
5. 5Test the robustness of LLMs in executing programs with altered symbolic transition rules
6. 6Configure experiments to systematically alter distribution shift and measure LLMs' response

AI engineers and researchers benefit from understanding LLMs' limitations, as it informs their design and application of these models in various tasks, such as code generation and execution

💡 LLMs' reliance on statistical priors rather than formal semantics hinders their ability to condition their reasoning on explicit rules

💡 LLMs rely on statistical priors, not programming language semantics, limiting their reasoning capabilities