SGLang

Fast LLM serving framework — efficient serving for LLMs and vision-language models

SGLang (Structured Generation Language) is an open-source framework for fast LLM serving and inference. It achieves 5–10× throughput improvements over naive inference through RadixAttention (efficient KV cache reuse), continuous batching, and structured generation with constrained decoding. Supports Llama, Mistral, Gemma, Qwen, and most open-source LLMs.

Key specs

7,000 GitHub stars source

as of 2026-03-27