#attention
2 briefs tagged #attention.
Research
Paper proposes power law graph attention for LLMs
PLGA generalizes scaled dot-product attention and reports inference-collapse results.
arXiv cs.LG+1 outlet·2d ago
Research
Researchers flag ALiBi numerical failure in attention
The paper says floating-point underflow can make some ALiBi attention heads partially blind.
arXiv cs.CL+1 outlet·Aug 5