2026-10-29 · America/Los_Angeles · 社区动态 · #9

TKintegers K5: 3.21× prefill and 3.10× decode throughput vs TorchAO FP8 on NVIDIA L4 [R] [P]

I built TKintegers, a balanced-ternary weight representation, converter, and GPU inference implementation. The result that stood out to me: the NVIDIA L4 has native FP8 tensor-core support, but my K5 implementation achieved higher throughput than the TorchAO FP8 baseline—even though the GPU has no native K5 instructions. On Qwen2.5-0.5B-Instruct, using a fixed matched workload: - Prefill: K5 achieved 3,402.85 tokens/s versus FP8’s 1,061.58 — 3.21× throughput. - Decode: K5 achieved 26.75 tokens/s versus FP8’s 8.63 — 3.10× throughput. The baseline was standard TorchAO dynamic E4M3 FP8, running…

热度 45.8 / 100；排名与评分保留该期记录。