Researchers flag ALiBi numerical failure in attention
The paper says floating-point underflow can make some ALiBi attention heads partially blind.
Why it matters
The findings suggest common decoder benchmarks may miss retrieval failures caused by positional-encoding numerics. Teams training or evaluating long-context models with ALiBi may need targeted retrieval tests rather than relying only on aggregate benchmark scores.
The key points
- 1.ALiBi bias scaling can underflow floating-point precision.
- 2.Token retrieval may degrade while standard decoder benchmarks barely move.
- 3.Log-scaled distances were the most consistent passkey-retrieval mitigation.
A new arXiv paper identifies a numerical failure mode in ALiBi positional encodings, where linear bias scaling can underflow floating-point precision and zero out many attention weights. The authors report that the issue appears in state-of-the-art pretrained models using ALiBi and can substantially impair token retrieval while having only minor effects on standard decoder benchmarks. In pretraining experiments with 148M-parameter decoder models, they evaluated four mitigation strategies and found log-scaled distances gave the most consistent gains in passkey retrieval.
⚡ Try this today
Audit ALiBi-based models with passkey or token-retrieval tests, and evaluate log-scaled distance mitigation before training new models.
Sources & original reporting
This brief summarizes and links to reporting from the publishers below.
Enjoyed this brief? Get the next one in your inbox.
More in Research
Paper introduces Travelling Thief Problem with Drone
TTP-D jointly optimizes item selection, routing and drone synchronization for collection operations.
Study finds audit-repair context makes LLM verifiers more lenient
Prior audit-repair episodes lowered false alarms across all tested model-wording combinations.
AlphaEvolve helps improve matrix multiplication exponent bound
A new paper reports an upper bound of ω < 2.371177, down from 2.371339.