Text Generation Inference (TGI)

Overview

Text Generation Inference is a Rust, Python and gRPC server for text generation inference, used in production at Hugging Face, and a toolkit for deploying and serving large language models. Its feature list includes continuous batching of incoming requests for increased total throughput, tensor parallelism for faster inference on multiple GPUs, token streaming with Server-Sent Events, Flash Attention and Paged Attention kernels, and quantization with bitsandbytes, GPT-Q, EETQ, AWQ, Marlin and FP8 to reduce the VRAM requirement, including NF4 and FP4 4-bit types. It also offers speculation for roughly 2x latency improvement, Prometheus metrics and OpenTelemetry tracing, and an OpenAI-compatible Messages API. The README states that the project is now in maintenance mode, with the Hugging Face team recommending vLLM, SGLang and local engines such as llama.cpp or MLX going forward.

Use cases

• Serve LLMs in production deployments
• Increase serving throughput with continuous batching
• Reduce VRAM use through 4-bit and 8-bit quantization
• Stream tokens to clients over an OpenAI-compatible API

Sources

• Text Generation Inference repository
• Text Generation Inference Apache 2.0 licence