Langfuse Tracing in Python: Turn LLM Failures into Eval Tests

Professor Py: AI Engineering · Intermediate ·🛠️ AI Tools & Apps ·1mo ago

Skills: LLMOps61%Prompt Systems Engineering53%

About this lesson

Turn production failures into repeatable evals by capturing failing LLM runs with Langfuse in Python. Learn how to record structured traces, assign severity, promote failures into a dataset, and replay regressions as deterministic tests for CI. Demonstration uses the Langfuse Python SDK for tracing, scoring, dataset creation, and replayable evals to improve LLM observability and triage. Subscribe for practical AI engineering tutorials on regression testing, observability, and production LLM workflows. #Langfuse #Python #LLMops #AIEngineering #Observability #RegressionTesting #MachineLearning

Original Description

Turn production failures into repeatable evals by capturing failing LLM runs with Langfuse in Python. Learn how to record structured traces, assign severity, promote failures into a dataset, and replay regressions as deterministic tests for CI. Demonstration uses the Langfuse Python SDK for tracing, scoring, dataset creation, and replayable evals to improve LLM observability and triage. Subscribe for practical AI engineering tutorials on regression testing, observability, and production LLM workflows. #Langfuse #Python #LLMops #AIEngineering #Observability #RegressionTesting #MachineLearning

Watch on YouTube ↗ (saves to browser)

Sign in to unlock AI tutor explanation · ⚡30

More on: LLMOps

View skill →

LLMOPS 06: CI/CD Deployment with AWS ECS & Fargate | End-to-End GenAI Project Deployment

LLMOPS 06: CI/CD Deployment with AWS ECS & Fargate | End-to-End GenAI Project Deployment

4. LLM Ops Infrastructure: Model Serving, RAG Pipelines, and Observability

4. LLM Ops Infrastructure: Model Serving, RAG Pipelines, and Observability

Analytics Vidhya

Cloud Run functions with Gemma 2 and Ollama

Cloud Run functions with Gemma 2 and Ollama

Google Cloud Tech

Model CI/CD Course: LLM Evaluation results

Model CI/CD Course: LLM Evaluation results

Weights & Biases

OpenClaw is open! Run your 24x7 Clawdbot on a Secure VPS!

OpenClaw is open! Run your 24x7 Clawdbot on a Secure VPS!

Build and Deploy a GenAI App with RAG on AWS Cloud | Step-by-Step Tutorial

Build and Deploy a GenAI App with RAG on AWS Cloud | Step-by-Step Tutorial

Analytics Vidhya

Related Reads

Transform Business Operations with Robotic Process Automation (RPA) Services

Learn how Robotic Process Automation (RPA) services can transform business operations by automating repetitive tasks and improving efficiency

I Didn't Expect an AI Tutor to Beat My Favorite Online Course (But It Changed How I Learn)

Discover how an AI tutor outperformed a traditional online course, changing the way you learn new skills and subjects

How to Talk to Your Database Using AI: A Practical Implementation Guide

Learn to use AI to interact with databases and simplify business question answering

Dev.to · Erwin Wilson Ceniza2

ROGup Explained (4/4): The ROGin AI Token That Powers It All

Learn about ROGup and its token ROGin, which powers the platform with four active utilities

Advanced Tutorial NotebookLM Slides For Powerpoint

Russell Stannard (TTVideos)