SIGNPOST-Bench tests text-vision conflicts in MLLMs
The benchmark measures how multimodal models respond when scene text conflicts with visual evidence.
Why it matters
The results show that current MLLMs can be materially steered by conflicting text embedded in images, even when non-textual visual content is preserved. This highlights a measurable failure mode for grounded perception systems used in geolocation and scene understanding.
The key points
- 1.SIGNPOST-Bench contains 25,555 image variants across four datasets.
- 2.Adversarial text raised median localization error 4.8-fold.
- 3.All 20 evaluated MLLMs showed shifts toward injected targets.
Researchers introduced SIGNPOST-Bench, a controlled counterfactual benchmark for evaluating how multimodal large language models resolve conflicts between visual cues and text in images. The benchmark includes 5,111 counterfactual groups and 25,555 image variants from four datasets, using localized synthetic scene-text interventions. In evaluations of 20 MLLMs from seven providers, adversarial variants increased median localization error from 282 km to 1,347 km, a 4.8-fold rise.
⚡ Try this today
Use SIGNPOST-Bench-style counterfactual tests when evaluating MLLMs for tasks that combine scene text with visual evidence.
Sources & original reporting
This brief summarizes and links to reporting from the publishers below.
Enjoyed this brief? Get the next one in your inbox.
More in Research
HarnessRisk benchmark targets agent harness safety
The arXiv benchmark tests how agent harness failures emerge across operational phases.
Agent Lightning v1.0 targets harnessed agentic RL
The framework connects arbitrary agent harnesses to RL training through an LLM endpoint proxy.
Paper studies memory transfer across language models
The arXiv paper tests whether frozen hashed memory can work with a new model reader.