Skip to content
LLM engines

LLM engines

Inference engines powering LLMs.

Generic

  • Hyperflare : a local inference server for NVIDIA SM120/SM121 GPUs.
  • Lucebox : LLM speculative inference server for heterogeneous hardware & consumer GPUs (esp. AMD GPUs).
  • SGLang a high-performance serving framework. See also SGLang Cookbook
  • TensorFold : LLM Inference Engine for Metal and CUDA.
  • Tensorsharp : a native .NET LLM inference engine and agent runtime for GGUF models.

llama.cpp

llama.cpp : LLM inference in C/C++ supporting GGUF format.

  • beellama.cpp : llama.cpp fork supporting the KVarN KV cache format.
  • ik_llama.cpp : llama.cpp fork with new quants and improved performance for MoE models.
  • KoboldCpp : llama.cpp fork with text, image, speech, music, and video generation.

  • llama-optimize : Find good llama.cpp command-line parameters for a given GGUF model on your machine, automatically, using statistically-designed experiments.

vLLM

vllm : a high-throughput and memory-efficient inference and serving engine for LLMs.

  • DeepInfer : High-performance, large-scale inference framework for DeepSeek models.
  • Intel vllm a vLLM docker image for Intel GPUs.
  • Paiton : optimized for AMD Radeon AI PRO R9700.
  • spark-vllm-docker : a vLLM docker image for DGX sparks.
  • vllm-radiance : a vLLM inference-server image for the AMD Radeon AI PRO R9700.

Mixture of Experts (MoE)

Tiered data access: hot experts on the GPU (VRAM), cold ones in the RAM, even data streaming from the SSD.

  • Basalt : Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell.
  • FreeToken : Local inference engine with Bandwidth-Adaptive Execution.
  • PolyStrata : supports various MoE models.
  • Project Maya : supports GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE), and MiniMax-M3.1-Flash (428B MoE).
  • Strata : runs Qwen3.8-Flash-Next and DeepSeekV4.1-Flash-1017 from 12GB+ VRAM with smart expert caching.

Specialized

  • TabbyAPI the official API server for exllamav3 backend, an inference library for the EXL3 quantization format (smaller bit per weight).
  • ninfer, a C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090.
  • ninfer-3090 : an ninfer fork for RTX 3090.
  • ninfer-4090 : an ninfer-3090 fork for RTX 4090.
  • MegaCapybara : a fast inference engine for Qwen3.8-27B on the NVIDIA RTX 5090.
Last updated on