2026-10-31 · America/Los_Angeles · 社区动态 · #18

I built a custom Sliding Window Attention from scratch. Got 4x speedup and 20% VRAM reduction on T4.

Hey everyone. I'm 15, and I've been trying to solve the VRAM bottleneck for long context in local LLMs and video processing. Standard nn.MultiheadAttention in PyTorch eats up VRAM and slows down heavily on long sequences (e.g., 15,000 tokens). The standard library tries to compute the full QxK matrix, which results in OOM (Out of Memory) errors on consumer GPUs. I wrote a custom SlidingWindowAttention from scratch that physically truncates old tokens before computing the QxK matrix, using chunking. This prevents the creation of massive NxN matrices. My benchmark on a Tesla T4 (15,000 tokens,…

热度 44.1 / 100；排名与评分保留该期记录。