SGLang

Overview

SGLang is an open-source inference framework for large language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts. It supports a wide range of hardware, including NVIDIA B200/B300/GB200/GB300 and H100/H200 GPUs, AMD Instinct MI300X and MI325X, Google TPU, Intel GPUs and Xeon CPUs, Apple Silicon, Huawei Ascend NPUs and Moore Threads GPUs. The repository includes SGLang Diffusion as a built-in image and video generation engine, and a surrounding ecosystem for speculative decoding, RL and post-training, deployment and orchestration. SGLang is hosted by the non-profit LMSYS organisation and released under the Apache 2.0 licence.

Use cases

• Serve LLMs and vision-language models at high throughput
• Run diffusion model image and video generation
• Optimize inference for agentic workloads
• Generate rollouts for reinforcement learning

Sources

• SGLang repository
• SGLang Apache 2.0 licence