PTXBench tests LLMs on GPU kernel optimization
The benchmark evaluates architecture-specific PTX use across GEMM and attention workloads on H100 and B200 GPUs.
Why it matters
The work adds an auditable testbed for assessing whether LLMs can exploit newer GPU architectures, a key issue as AI systems increasingly rely on automated performance engineering. It also suggests current models are not yet reliable substitutes for frontier GPU libraries across demanding kernels.
The key points
- 1.PTXBench covers GEMM and attention workloads on H100 and B200 GPUs.
- 2.No evaluated model consistently matched frontier libraries across the suite.
- 3.Fine-tuning Qwen3.6-27B improved some tasks but generalized unevenly.
Researchers introduced PTXBench, a benchmark for evaluating and adapting LLMs to use architecture-specific PTX for GPU kernel optimization. The benchmark measures functional correctness, runtime execution of selected target instructions, and speedup over frontier libraries across GEMM and attention workloads on NVIDIA H100 and B200 GPUs. The reported results show uneven capability: success drops on complex attention backward workloads, target instruction execution does not necessarily produce competitive performance, and no evaluated model consistently matches frontier libraries. The authors also fine-tuned Qwen3.6-27B, finding that repair-conditioned training helps some tasks but generalization remains uneven.
⚡ Try this today
Use PTXBench-style correctness, instruction-execution, and speed tests before relying on LLM-generated PTX kernels.
Sources & original reporting
This brief summarizes and links to reporting from the publishers below.
- arXiv cs.AIPTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTXAug 19, 12:00 PM↗
- arXiv cs.AIKernelArc: A Multi-Agent Framework for GPU Kernel OptimizationAug 19, 12:00 PM↗
- arXiv cs.CLPTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTXAug 19, 12:00 PM↗
- HF Daily PapersPTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTXAug 18, 4:00 AM↗
Enjoyed this brief? Get the next one in your inbox.
More in Research
Startup says cancer AI needs better data
TechCrunch reports the company argues data is the central barrier to cancer-curing AI.
HarnessRisk benchmarks agent harness safety failures
The benchmark tests safety risks across agent harness phases, with attack success ranging from 12.6% to 80.9%.
Agent Lightning v1.0 targets harnessed agentic RL
The framework connects arbitrary agent harnesses to RL training through an LLM endpoint proxy.