llama.cpp

The open-source C/C++ engine for running LLMs and vision models locally, with a CLI, an OpenAI-compatible server with a built-in web UI, 1.5-bit to 8-bit quantization, and backends from CUDA and Metal to Vulkan and SYCL.

Visit Website 133k
Pricing Open Source
Category Infrastructure
Local LLM Open Source C++ GGUF Inference

Founder's Verdict

"The engine underneath much of local AI, LM Studio included. It's more hands-on than Ollama or LM Studio, and in return gives you the most direct control over hardware, quantization and performance."

llama.cpp

llama.cpp describes itself as “LLM inference in C/C++”. Its stated goal is to enable LLM and vision-model inference “with minimal setup and state-of-the-art performance”. It’s MIT-licensed and built on the ggml library.

Get started

# Install on macOS or Linux
curl -LsSf https://llama.app/install.sh | sh

# Install on Windows (PowerShell)
irm https://llama.app/install.ps1 | iex

# Download a model from Hugging Face and chat with it
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Serve it through an OpenAI-compatible API, with a web UI
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Docker images, prebuilt binaries and build-from-source instructions are in the repository.

Why people use it

  • Hardware coverage. It runs on plain CPUs (ARM NEON, AVX, AVX2, AVX-512 and AMX on x86, plus RISC-V) and on GPUs through CUDA, HIP, Metal, Vulkan, SYCL, MUSA, OpenCL and WebGPU, with more backends for NPUs. CPU+GPU hybrid inference lets you run models larger than your VRAM.
  • Quantization from 1.5-bit to 8-bit integers, for faster inference and lower memory use.
  • GGUF, the model format it uses, with a converter for Hugging Face models.
  • Vision models, not just text.

Who it’s for

  • Tinkerers and performance chasers who want every knob.
  • Developers embedding inference in their own apps.
  • Unusual hardware, from Intel GPUs to Ascend NPUs.

Alternatives

If you’d rather not manage models and flags yourself, Ollama gives you one-command setup and LM Studio gives you a desktop app. Compare them in Ollama vs LM Studio vs llama.cpp.

Compared with

Ollama vs LM Studio vs llama.cpp (2026): Best Way to Run LLMs Locally Start with Ollama if you want local models behind an API for your apps and coding agents: it's MIT-licensed and a one-line install on macOS, Windows and Linux. Pick LM Studio if you'd rather work in a desktop app, especially on an Apple Silicon Mac, where it also runs MLX models. Use llama.cpp directly when you want the engine itself, with the most control over hardware backends and quantization.

Related Tools

LM Studio

Desktop app for macOS, Windows and Linux that finds, downloads and runs open models locally (GGUF through llama.cpp, MLX on Apple Silicon), with chat, offline document chat, MCP tools and an OpenAI-compatible local server.

Local LLMDesktop App

Ollama

★185.5k

Run open models such as Gemma, Qwen, DeepSeek and gpt-oss on your own computer with one command, then use them through a local REST API, Python or JavaScript, or coding agents like Claude Code and Codex. Optional cloud models cover anything too big for your hardware.

Local LLMOpen Source