LangFuzz: Redteaming for Language Models

LangChain · Intermediate ·🧠 Large Language Models ·1y ago

Key Takeaways

Tests language models using LangFuzz, a library that generates edge cases through fuzz testing and metamorphic testing

Original Description

One of the hardest parts of testing your LLM application is coming with a good dataset of edge cases to benchmark it on `langfuzz` is a new experimental library that helps with that ❓How? It uses a fuzz testing technique similar to metamorphic testing. It generates pairs of similar questions, and then runs them through the LLM application. If the answers are drastically different - then one of them must be wrong! From there, you can add both, one, or neither of the questions to a LangSmith dataset so you can iterate on your app and continue benchmarking on these datapoints. GitHub: https://github.com/hwchase17/langfuzz
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from LangChain · LangChain · 0 of 60

← Previous Next →
1 Chat With Your Documents Using LangChain + JavaScript
Chat With Your Documents Using LangChain + JavaScript
LangChain
 2 LangChain SQL Webinar
LangChain SQL Webinar
LangChain
 3 LangChain "OpenAI functions" Webinar
LangChain "OpenAI functions" Webinar
LangChain
 4 LangSmith Launch
LangSmith Launch
LangChain
 5 LangChain x Pinecone: Supercharging Llama-2 with RAG
LangChain x Pinecone: Supercharging Llama-2 with RAG
LangChain
 6 LangChain Expression Language
LangChain Expression Language
LangChain
 7 Building LLM applications with LangChain with Lance
Building LLM applications with LangChain with Lance
LangChain
 8 Benchmarking Question/Answering Over CSV Data
Benchmarking Question/Answering Over CSV Data
LangChain
 9 LangChain "RAG Evaluation" Webinar
LangChain "RAG Evaluation" Webinar
LangChain
 10 Fine-tuning in Your Voice Webinar
Fine-tuning in Your Voice Webinar
LangChain
 11 Tabular Data Retrieval
Tabular Data Retrieval
LangChain
 12 Building an LLM Application with Audio by AssemblyAI
Building an LLM Application with Audio by AssemblyAI
LangChain
 13 Superagent Deepdive Webinar
Superagent Deepdive Webinar
LangChain
 14 Lessons from Deploying LLMs with LangSmith
Lessons from Deploying LLMs with LangSmith
LangChain
 15 Shortwave Assistant Deepdive Webinar
Shortwave Assistant Deepdive Webinar
LangChain
 16 Cognitive Architectures for Language Agents
Cognitive Architectures for Language Agents
LangChain
 17 Effectively Building with LLMs in the Browser with Jacob
Effectively Building with LLMs in the Browser with Jacob
LangChain
 18 Data Privacy for LLMs
Data Privacy for LLMs
LangChain
 19 "Theory of Mind" Webinar with Plastic Labs
"Theory of Mind" Webinar with Plastic Labs
LangChain
 20 LangChain Templates
LangChain Templates
LangChain
 21 Using Natural Language to Query Postgres with Jacob
Using Natural Language to Query Postgres with Jacob
LangChain
 22 Building a Research Assistant from Scratch
Building a Research Assistant from Scratch
LangChain
 23 Benchmarking RAG over LangChain Docs
Benchmarking RAG over LangChain Docs
LangChain
 24 Skeleton-of-Thought: Building a New Template from Scratch
Skeleton-of-Thought: Building a New Template from Scratch
LangChain
 25 Benchmarking Methods for Semi-Structured RAG
Benchmarking Methods for Semi-Structured RAG
LangChain
 26 LangSmith Highlights: Getting Started
LangSmith Highlights: Getting Started
LangChain
 27 LangSmith Highlights: Debugging
LangSmith Highlights: Debugging
LangChain
 28 LangSmith Highlights: Datasets
LangSmith Highlights: Datasets
LangChain
 29 LangSmith Highlights: Evaluation
LangSmith Highlights: Evaluation
LangChain
 30 LangSmith Highlights: Human Annotation
LangSmith Highlights: Human Annotation
LangChain
 31 LangSmith Highlights: Monitoring
LangSmith Highlights: Monitoring
LangChain
 32 LangSmith Highlights: Hub
LangSmith Highlights: Hub
LangChain
 33 SQL Research Assistant
SQL Research Assistant
LangChain
 34 Getting Started with Multi-Modal LLMs
Getting Started with Multi-Modal LLMs
LangChain
 35 Build a Full Stack RAG App With TypeScript
Build a Full Stack RAG App With TypeScript
LangChain
 36 Auto-Prompt Builder (with Hosted LangServe)
Auto-Prompt Builder (with Hosted LangServe)
LangChain
 37 LangChain v0.1.0 Launch: Introduction
LangChain v0.1.0 Launch: Introduction
LangChain
 38 LangChain v0.1.0 Launch: Observability
LangChain v0.1.0 Launch: Observability
LangChain
 39 LangChain v0.1.0 Launch: Integrations
LangChain v0.1.0 Launch: Integrations
LangChain
 40 LangChain v0.1.0 Launch: Composability
LangChain v0.1.0 Launch: Composability
LangChain
 41 LangChain v0.1.0 Launch: Streaming
LangChain v0.1.0 Launch: Streaming
LangChain
 42 LangChain v0.1.0 Launch: Output Parsing
LangChain v0.1.0 Launch: Output Parsing
LangChain
 43 LangChain v0.1.0 Launch: Retrieval
LangChain v0.1.0 Launch: Retrieval
LangChain
 44 LangChain v0.1.0 Launch: Agents
LangChain v0.1.0 Launch: Agents
LangChain
 45 Build and Deploy a RAG app with Pinecone Serverless
Build and Deploy a RAG app with Pinecone Serverless
LangChain
 46 Hosted LangServe + LangChain Templates
Hosted LangServe + LangChain Templates
LangChain
 47 LangGraph: Intro
LangGraph: Intro
LangChain
 48 LangGraph: Agent Executor
LangGraph: Agent Executor
LangChain
 49 LangGraph: Chat Agent Executor
LangGraph: Chat Agent Executor
LangChain
 50 LangGraph: Human-in-the-Loop
LangGraph: Human-in-the-Loop
LangChain
 51 LangGraph: Dynamically Returning a Tool Output Directly
LangGraph: Dynamically Returning a Tool Output Directly
LangChain
 52 LangGraph: Respond in a Specific Format
LangGraph: Respond in a Specific Format
LangChain
 53 LangGraph: Managing Agent Steps
LangGraph: Managing Agent Steps
LangChain
 54 LangGraph: Force-Calling a Tool
LangGraph: Force-Calling a Tool
LangChain
 55 LangGraph: Multi-Agent Workflows
LangGraph: Multi-Agent Workflows
LangChain
 56 Streaming Events: Introducing a new `stream_events` method
Streaming Events: Introducing a new `stream_events` method
LangChain
 57 Building a web RAG chatbot: using LangChain, Exa (prev. Metaphor), LangSmith, and Hosted Langserve
Building a web RAG chatbot: using LangChain, Exa (prev. Metaphor), LangSmith, and Hosted Langserve
LangChain
 58 OpenGPTs
OpenGPTs
LangChain
 59 Open Source RAG with Nomic's New Embedding Model (and ChromaDB and Ollama)
Open Source RAG with Nomic's New Embedding Model (and ChromaDB and Ollama)
LangChain
 60 LangGraph: Persistence
LangGraph: Persistence
LangChain

Related AI Lessons

⚡
Building HITL Feedback RAG: Embeddings, Retrieval, and Reranking
Learn to build a Human-in-the-Loop (HITL) Feedback RAG system using embeddings, retrieval, and reranking to improve model performance
Medium · AI
 ⚡
Building HITL Feedback RAG: Embeddings, Retrieval, and Reranking
Learn to build a Human-in-the-Loop (HITL) Feedback RAG system using embeddings, retrieval, and reranking to improve LLM performance
Medium · LLM
 ⚡
The 2026 AI Model Release Race: Every Major LLM Launch You Need to Know
Stay updated on the 2026 AI model release race, including major LLM launches like Claude Sonnet 5 and GPT-5.6, to leverage the latest advancements in AI technology
Dev.to AI
 ⚡
Call GPT, Claude, and Gemini from one API key — a 3-step setup
Access GPT, Claude, and Gemini through one API key with a 3-step setup using Modelishub
Dev.to AI
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →