NOLLI benchmark probes English-Korean model gaps
The new puzzle benchmark tests whether Korean gaps come from language, script or culture-specific tasks.
Why it matters
The results suggest that some Korean performance gaps may be tied less to presentation language alone and more to handling Hangul jamo or Korean-specific orthography and culture. NOLLI gives model developers a more targeted way to diagnose multilingual weaknesses than broad aggregate benchmarks.
The key points
- 1.NOLLI includes 7,500 deterministic English-Korean puzzle items.
- 2.Matched English-Korean accuracy was statistically equivalent for 12 evaluated models.
- 3.Korean Cipher trailed English by up to 68.7 percentage points.
Researchers introduced NOLLI, a procedurally generated English-Korean puzzle benchmark with 15 puzzle types, 25 tasks and 7,500 items. The benchmark uses seed-regenerable instances, unique-solution verification and deterministic scoring, with difficulty calibrated against a fixed reference model. In tests of 15 frontier, open-weight and Korean-developed models, the 12 models above a 3% overall-accuracy floor showed statistically equivalent matched English-Korean accuracy within a +/- 10 percentage-point margin, while writing-system-intensive tasks showed larger gaps.
⚡ Try this today
Use NOLLI-style task breakdowns to audit whether Korean failures come from translation, Hangul jamo handling or Korean-only task design.
Sources & original reporting
This brief summarizes and links to reporting from the publishers below.
Enjoyed this brief? Get the next one in your inbox.
More in Research
New papers target VLMs' spatial reasoning gap
Researchers propose runtime memory, RL training, benchmarks and 3D generation methods for spatial AI.
AI stories draw Hacker News debate across research and tools
Posts covered drug discovery, math, privacy, Cloudflare, AI education and lab culture.
New papers probe spatial intelligence in AI vision models
SpaRRTa, SMA and PinpointQA target spatial reasoning gaps in visual and multimodal systems.