Multi-backend inference server for deploying and scaling models in datacenters—supports TensorRT, ONNX, PyTorch, and more, not only LLMs.
Triton is aimed at teams that need a unified serving layer across vision, speech, recommender, and LLM workloads with dynamic batching, model ensembles, and GPU/CPU scheduling.
In LLM-heavy roadmaps
You might run TensorRT-LLM or other backends behind Triton while still using vLLM or TGI elsewhere—choose per service’s latency, throughput, and ops maturity.