llama.cpp

On March 10, 2023, Georgi Gerganov pushed the first commit of llama.cpp to GitHub. The library, written in C and C++, aimed to do one thing: run Meta's LLaMA models on a laptop CPU without a GPU. Within weeks, it had thousands of stars and a growing community of developers pushing its capabilities further.

The core innovation is quantization. By compressing model weights from 32-bit floating point to 4-bit or 8-bit integers using the GGUF format, llama.cpp cuts memory requirements by 75 percent or more. A 7-billion-parameter model that would need 28 GB of VRAM on a GPU can run in under 6 GB of system RAM. The GGUF format also bundles the tokenizer, vocabulary, and chat template into a single file, making model distribution trivial.

llama.cpp supports an unusually broad range of hardware: x86 CPUs with AVX2, Apple Silicon via Metal, NVIDIA GPUs with CUDA, AMD GPUs with hipBLAS, Intel GPUs with SYCL, and even Vulkan. The project ships command-line tools for generation, benchmarking, and model conversion, along with a lightweight HTTP server. As of 2025, it has over 900 contributors and 69,000 GitHub stars. Tools like Ollama and LM Studio build directly on llama.cpp, making local LLM inference accessible to non-technical users. The library has become the de facto standard for running large language models on consumer hardware, enabling private, cost-effective AI experimentation without cloud dependencies.[edit]