LoT improves KB-VQA accuracy without model training
The framework highlights image regions and text evidence before answer generation.
Why it matters
The work targets a common failure mode in retrieval-augmented multimodal systems: models can miss the visual and textual evidence needed to answer correctly. Its training-free design could make evidence selection easier to test across existing MLLM deployments.
The key points
- 1.LoT needs no parameter updates, auxiliary models, or architecture changes.
- 2.The method uses internal attention to select visual and textual evidence.
- 3.Reported gains reached up to 12.5 accuracy points across evaluated MLLMs.
Researchers introduced Look Twice, or LoT, a training-free inference-time framework for knowledge-based visual question answering. LoT uses a model's internal attention patterns to identify query-relevant image regions and textual sentences, filters distractions, and reformulates the input to highlight selected evidence before answer generation. Across four KB-VQA benchmarks and ten off-the-shelf multimodal large language models from 2B to 38B parameters, LoT improved every evaluated backbone, with average gains of up to 12.5 accuracy points.
⚡ Try this today
Test evidence-highlighting at inference time before fine-tuning a KB-VQA model.
Sources & original reporting
This brief summarizes and links to reporting from the publishers below.
- arXiv cs.AILook Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question AnsweringAug 7, 12:00 PM↗
- arXiv cs.CLLook Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question AnsweringAug 7, 12:00 PM↗
- arXiv cs.CLUniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question AnsweringAug 6, 12:00 PM↗
- arXiv cs.CLRepresenting Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native ModelingAug 6, 12:00 PM↗
- arXiv cs.AIPerception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question AnsweringAug 6, 12:00 PM↗
- HF Daily PapersChronoVision: Temporal Reasoning via Latent State ReconstructionAug 6, 4:00 AM↗
- arXiv cs.AIWhen and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video UnderstandingAug 5, 12:00 PM↗
- arXiv cs.AICURV: Enhancing Chart Understanding Through Curriculum Visual Grounded ReasoningAug 5, 12:00 PM↗
- arXiv cs.CLCURV: Enhancing Chart Understanding Through Curriculum Visual Grounded ReasoningAug 5, 12:00 PM↗
- arXiv cs.LGBayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question AnsweringAug 5, 12:00 PM↗
Enjoyed this brief? Get the next one in your inbox.
More in Research
Vivodyne says AI drug discovery has a data problem
The startup says its robotic labs generate human-tissue data for drug discovery models.
HarnessRisk benchmarks agent harness safety
The benchmark tests safety failures across agent harness lifecycle phases.
Agent Lightning v1.0 targets harnessed agentic RL
The framework connects arbitrary agent harnesses to RL training through an LLM endpoint proxy.