The Failure Catalog

📰 Medium · LLM

Learn how Visual Language Models (VLMs) can fail when trusting data over visual evidence and how to stress test them

intermediate Published 24 Apr 2026
Action Steps
  1. Setup adversarial conditions to stress test VLMs
  2. Use fabricated or partial data to evaluate model performance
  3. Evaluate the impact of delivery channels on model accuracy
  4. Analyze the results to identify anchoring bias in VLMs
  5. Apply techniques to mitigate anchoring bias and improve visual reasoning
Who Needs to Know This

Data scientists and AI engineers working with VLMs can benefit from understanding these failure modes to improve model robustness

Key Insight

💡 VLMs can be misled by structured data, suppressing visual reasoning, and delivery channels can modulate this effect

Share This
🚨 VLMs can miss the 'Elephant in the Room' when trusting data over visual evidence! 🤖 Learn how to stress test and improve model robustness #AI #VLMs

Full Article

Title: The Failure Catalog

URL Source: https://medium.com/@mraduldubey/the-failure-catalog-fa236eeb25e2?source=rss------llm-5

Published Time: 2026-04-24T14:20:22Z

Markdown Content:
[Sitemap](https://medium.com/sitemap/sitemap.xml)

[Open in app](https://play.google.com/store/apps/details?id=com.medium.reader&referrer=utm_source%3DmobileNavBar&source=post_page---top_nav_layout_nav-----------------------------------------)

Sign up

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Fmedium.com%2F%40mraduldubey%2Fthe-failure-catalog-fa236eeb25e2&source=post_page---top_nav_layout_nav-----------------------global_nav------------------)

[](https://medium.com/?source=post_page---top_nav_layout_nav-----------------------------------------)

Get app

[Write](https://medium.com/m/signin?operation=register&redirect=https%3A%2F%2Fmedium.com%2Fnew-story&source=---top_nav_layout_nav-----------------------new_post_topnav------------------)

[Search](https://medium.com/search?source=post_page---top_nav_layout_nav-----------------------------------------)

Sign up

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Fmedium.com%2F%40mraduldubey%2Fthe-failure-catalog-fa236eeb25e2&source=post_page---top_nav_layout_nav-----------------------global_nav------------------)

![Image 1: Unknown user](https://miro.medium.com/v2/resize:fill:32:32/1*dmbNkD5D-u45r44go_cf0g.png)

# **The Failure Catalog**

## _Ghost entities and false accusations, when VLMs trust data over their own eyes!_

[![Image 2: Mradul Dubey](https://miro.medium.com/v2/resize:fill:32:32/1*gfO8IyU81NLbEnFn2BjV_Q.jpeg)](https://medium.com/@mraduldubey?source=post_page---byline--fa236eeb25e2---------------------------------------)

[Mradul Dubey](https://medium.com/@mraduldubey?source=post_page---byline--fa236eeb25e2---------------------------------------)

Follow

5 min read

·

Apr 24, 2026

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fvote%2Fp%2Ffa236eeb25e2&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40mraduldubey%2Fthe-failure-catalog-fa236eeb25e2&user=Mradul+Dubey&userId=b72e98a18093&source=---header_actions--fa236eeb25e2---------------------clap_footer------------------)

52

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Frepost%2Fp%2Ffa236eeb25e2&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40mraduldubey%2Fthe-failure-catalog-fa236eeb25e2&user=Mradul+Dubey&userId=b72e98a18093&source=---header_actions--fa236eeb25e2---------------------repost_header------------------)

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fbookmark%2Fp%2Ffa236eeb25e2&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40mraduldubey%2Fthe-failure-catalog-fa236eeb25e2&source=---header_actions--fa236eeb25e2---------------------bookmark_footer------------------)

[Listen](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2Fplans%3Fdimension%3Dpost_audio_button%26postId%3Dfa236eeb25e2&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40mraduldubey%2Fthe-failure-catalog-fa236eeb25e2&source=---header_actions--fa236eeb25e2---------------------post_audio_button------------------)

Share

Press enter or click to view image in full size

![Image 3](https://miro.medium.com/v2/resize:fit:700/1*BWhyJEqRI67GV1mjYMXt-Q.jpeg)

Your VLMs may miss the proverbial “Elephant in the Room”

In [VLM: the more you tell it, the less it sees](https://medium.com/towards-artificial-intelligence/vlm-the-more-you-tell-it-the-less-it-sees-c07f33b6a159), I defined anchoring bias in VLMs and measured it across seven controlled conditions.

### The findings were quantitative:

> **_Structured data suppresses visual reasoning, and the delivery channel modulates how badly._**

This post is different. These are not controlled experiments. They are stress tests. I have setup deliberately adversarial conditions (fabricated data, partial data, a weaker model) that I used
Read full article → ← Back to Reads

Related Videos

5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Chapter 3: Looking Inside Large Language Models | Hands-On Large Language Models Book
Chapter 3: Looking Inside Large Language Models | Hands-On Large Language Models Book
onepagecode
Hands-On Large Language Models | Chapter 7: Advanced Text Generation Techniques
Hands-On Large Language Models | Chapter 7: Advanced Text Generation Techniques
onepagecode
Hands-On LLMs - Chapter 1: An Introduction to Large Language Models
Hands-On LLMs - Chapter 1: An Introduction to Large Language Models
onepagecode
Chapter 2: Tokens and Embeddings | Hands-On Large Language Models Book
Chapter 2: Tokens and Embeddings | Hands-On Large Language Models Book
onepagecode
Hands-On Large Language Models | Chapter 5: Text Clustering and Topic Modeling
Hands-On Large Language Models | Chapter 5: Text Clustering and Topic Modeling
onepagecode