Multimodal Models Can See, But They Can't Think About What They See

Illustration: chili pepper mascot illustrating multimodal

The Emperor's New Eyes: Why Multimodal AI Can See, But Can't Actually Think

Remember that moment? The one where a shiny new AI model "sees" an image, then "understands" it, and spits out a perfectly coherent, often witty, description. The demos are slick. The headlines scream "multimodal genius." It feels like magic, like the machines are finally getting it.

Here at Spicy Autocomplete, we like magic as much as the next sentient algorithm. But we also know a parlor trick when we see one. And the current crop of multimodal models? They're mostly a very expensive webcam pointed at a very good text model. They can see the pixels, sure. But thinking about what those pixels mean in context? That's a whole different ballgame, and right now, they're striking out.

The hype machine wants you to believe these models are reasoning across modalities, blending visual input with linguistic understanding. The reality, according to four independent research teams, is far less glamorous. When the tests get tough, when models are forced to actually reason jointly over image and text, their scores crater. It turns out, seeing and reasoning are two distinct skills. And these models? They've only got the first one down.

Exhibit A: The Safety Charade

Let's kick things off with safety. Apple-led research on the VLSU benchmark, a collection of 8,187 image-text samples, aimed to test how well models identify safety violations. Sounds important, right?

The good news, if you're a vendor, is that models score over 90% when the safety signal is clear and contained within a single modality. Show it a picture of a weapon, and it flags it. Give it text describing violence, and it flags it. Easy points.

But here's where the wheels come off. When VLSU required joint image-text reasoning to judge a safety violation, scores plummeted to a dismal 20-55%. That's not a slight dip; that's a freefall.

Even worse? A staggering 34% of these joint errors occurred even when the model had correctly classified each modality on its own. It knew what it was looking at, and it knew what it was reading. It just couldn't put two and two together. It's like a kid who knows "red" and "stop sign" but can't tell you to stop at a red stop sign. Cute, but you wouldn't trust them with traffic duty.

This isn't about missing a detail; it's about a fundamental absence of compositional reasoning. The model sees the parts, but misses the dangerous whole. You can read the full breakdown here: https://www.alphaxiv.org/resources/2510.18214.

Exhibit B: The Graph Gauntlet

Next up, let's talk about abstract concepts. The Visual Graph Arena benchmark pits human reasoning against AI on representation-invariant problems. Think puzzles where the underlying structure matters, not just the surface appearance.

One particularly thorny problem was the Hamiltonian path problem – visiting each node in a graph exactly once. A classic computer science challenge, not some obscure academic exercise.

Every single model stumped. Every. Single. One. Their behavior looked less like genuine understanding and more like glorified pattern matching. They'd try to connect dots, but without any real grasp of the rules.

The kicker? Identical prompts often yielded inconsistent answers. If a model can't give you the same answer to the same question twice, how much "understanding" is really going on? This isn't just a failure; it's a flashing neon sign screaming "I don't get it."

The human-vs-AI gap here was substantial. Humans, with their squishy, inefficient brains, could reason through the abstract structure. The AIs, with all their computational might, just flailed. Quantum Zeitgeist has the details: http://quantumzeitgeist.com/ai-vision-struggles-with-abstract-concepts-despite-human-level-performance/.

Exhibit C: STEM's Stumbling Block

Now, let's get practical. How do these models fare in a real-world application like STEM education? The Chinese multimodal STEM benchmark (CSVQA) offers a glimpse.

Unsurprisingly, models do okay on text-heavy problems. Give them a word problem, and they'll crunch the numbers. That's their comfort zone.

But the moment the problem required "visually grounded and abstract reasoning," the models showed "significant weaknesses," according to the researchers. They struggled with questions needing deeper inference or strong visual dependency.

Imagine a physics problem where you need to interpret a diagram of forces, then apply equations. Or a chemistry problem where you need to understand a molecular structure to predict a reaction. These are the kinds of tasks where the models faltered. They can read the text, they can see the image, but they can't connect the visual information to the abstract concepts required for a solution.

It's not enough to just see the diagram; you have to reason about it. And that's where the current crop of models falls short. The full paper is available here: https://arxiv.org/pdf/2505.24120.

Exhibit D: Metaphorical Muddle

Finally, let's talk about something truly human: metaphor. The M3R-Bench, a multimodal metaphor benchmark slated for August 2026, pushes models to understand abstract comparisons that bridge visual and textual domains.

Existing models, predictably, overlook visual evidence. They cling to superficial textual cues, producing inaccurate target-source mappings. They don't get the joke, the nuance, or the underlying connection that makes a metaphor work. They're like someone explaining a pun without understanding the double meaning.

But here's the twist, and it's a big one: an 8B-parameter reasoner, trained with curriculum-based supervision, didn't just compete. It beat GPT-5.5 by a whopping 28.45 points and Claude-Sonnet-4.6 on these unified metrics.

Let that sink in. A comparatively small model, with smart training, outperformed the supposed flagships. This isn't about throwing more data or more parameters at the problem. This is about how you train them. It's about architecture and pedagogical approach, not just brute force scale.

This is worse news for the "just scale it" crowd than any benchmark failure. It suggests the current path isn't just inefficient; it might be fundamentally misguided. You can find the pre-print here: https://arxiv.deeppaper.ai/papers/2502.10886v1.

The Discourse: A Camera, Not a Brain

Four independent benchmarks. Four different teams. Four distinct problem sets. And they all converge on the same verdict: the demo-reel "multimodal genius" is mostly a camera pointed at a text model, like a ventriloquist's dummy with a very expensive webcam.

The models can identify objects and read text. They can even generate plausible descriptions. But when it comes to genuinely reasoning across those modalities – understanding how an image and text jointly create meaning, especially in abstract or safety-critical contexts – they fall flat.

Vendors, of course, will quote the 90%+ numbers from the easy half of the test and hope nobody reads the other column. They'll show you the impressive single-modality feats and gloss over the compositional