PowerInfer

library
Shanghai Jiao Tong University

IPADS's activation-sparsity LLM serving engine (9.6K★): hot neurons stay on the GPU, cold ones compute on the CPU, making consumer-GPU inference practical (SOSP '24). PowerInfer-2 (June 2024) extends the design to smartphones with neuron-cluster-granularity heterogeneous compute. The team spun out as Tiiny-AI (tiiny.ai), where the repo now lives.

PowerInfer Paper (arXiv)PowerInfer-2 Paper (arXiv)GitHub

Paper

arXiv HTML

Library

GitHub Repository →
efficiencyinferenceopen-source