LLM engines
Inference engines powering LLMs.
Generic
- Hyperflare : a local inference server for NVIDIA SM120/SM121 GPUs.
- Lucebox : LLM speculative inference server for heterogeneous hardware & consumer GPUs (esp. AMD GPUs).
- SGLang a high-performance serving framework. See also SGLang Cookbook
- TensorFold : LLM Inference Engine for Metal and CUDA.
- Tensorsharp : a native .NET LLM inference engine and agent runtime for GGUF models.
llama.cpp
llama.cpp : LLM inference in C/C++ supporting GGUF format.
- beellama.cpp :
llama.cppfork supporting the KVarN KV cache format. - ik_llama.cpp :
llama.cppfork with new quants and improved performance for MoE models. - KoboldCpp :
llama.cppfork with text, image, speech, music, and video generation.
- llama-optimize : Find good llama.cpp command-line parameters for a given GGUF model on your machine, automatically, using statistically-designed experiments.
vLLM
vllm : a high-throughput and memory-efficient inference and serving engine for LLMs.
- DeepInfer : High-performance, large-scale inference framework for DeepSeek models.
- Intel vllm a vLLM docker image for Intel GPUs.
- Paiton : optimized for AMD Radeon AI PRO R9700.
- spark-vllm-docker : a vLLM docker image for DGX sparks.
- vllm-radiance : a vLLM inference-server image for the AMD Radeon AI PRO R9700.
Mixture of Experts (MoE)
Tiered data access: hot experts on the GPU (VRAM), cold ones in the RAM, even data streaming from the SSD.
- Basalt : Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell.
- FreeToken : Local inference engine with Bandwidth-Adaptive Execution.
- PolyStrata : supports various MoE models.
- Project Maya : supports GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE), and MiniMax-M3.1-Flash (428B MoE).
- Strata : runs Qwen3.8-Flash-Next and DeepSeekV4.1-Flash-1017 from 12GB+ VRAM with smart expert caching.
Specialized
- TabbyAPI the official API server for exllamav3 backend, an inference library for the EXL3 quantization format (smaller bit per weight).
- ninfer, a C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090.
- ninfer-3090 : an
ninferfork for RTX 3090. - ninfer-4090 : an
ninfer-3090fork for RTX 4090. - MegaCapybara : a fast inference engine for Qwen3.8-27B on the NVIDIA RTX 5090.
Last updated on