Rotary GPU

Rotary GPU is a method for executing large Mixture-of-Experts (MoE) models on consumer-grade GPUs with limited VRAM. It works by dynamically swapping model layers between GPU and CPU memory during inference, effectively rotating which parts of the model reside on the GPU at any given time. This allows running models that would otherwise exceed available VRAM, such as 100B+ parameter MoE architectures, on a single 24GB GPU. The approach is described in a recent arXiv paper (2605.29135) and has generated interest on Hacker News for its potential to democratize access to large language models. While still a research concept, it addresses a key bottleneck for local LLM inference: the memory wall. The technique is particularly relevant for MoE models, which have sparse activation patterns that can be exploited for layer-wise swapping. Early community signals suggest it could enable practical local deployment of frontier-scale models without expensive multi-GPU setups.

An arXiv paper (2605.29135) posted in May 2025 gained traction on Hacker News, sparking discussion about local execution of large MoE models under VRAM constraints.