Using Custom Evaluators During Training | Datawizz

This video demonstrates how to implement custom evaluation metrics during model training in Datawizz, allowing you to monitor domain-specific performance metrics alongside standard training/validation loss. Technical Overview: In this example, I train a conversation summarization model (Qwen-0.6B base) and track ROUGE scores during training. The feature supports both built-in metrics and custom evaluation functions, including LLM-as-judge implementations. Key Features Covered: - Configuring custom evaluators in the training pipeline - Real-time metric visualization during training epochs - Accessing individual inference samples from each evaluation step - Correlating metric improvements with loss curves Timestamps: 0:00 - Feature introduction 0:34 - Dataset structure 0:52 - Training configuration 1:15 - Custom evaluator setup 2:03 - Metric analysis during training 2:40 - Evaluation sample inspection The evaluation samples are exportable for offline analysis. Multiple evaluators can run concurrently, and the platform supports both predefined metrics and custom Python implementations. Try it yourself: https://datawizz.ai