JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees

1. 1Convert fault trees to textual representation
2. 2Train LLMs on the proposed benchmark
3. 3Evaluate LLMs' ability to track and analyze malfunctions
4. 4Analyze results to improve LLMs' performance

AI engineers and researchers benefit from this benchmark to improve LLMs' performance in complex system maintenance, while product managers can utilize it to develop more efficient diagnostic tools

💡 Textual representation of fault trees enables LLMs to process and analyze complex system malfunctions

💡 New benchmark for LLMs: JFTA-Bench evaluates ability to track & analyze malfunctions using fault trees