llama.cpp and ggml are a pair of projects by Georgi Gerganov that made running large language models on ordinary hardware practical. ggml is the foundation: its repository describes it as a tensor library for machine learning, a low-level, cross-platform implementation written in C with no third-party dependencies, providing the building blocks for neural-network computation including integer quantization, automatic differentiation, and optimizers. llama.cpp, released in 2023, is the inference engine built on that library, and its stated goal is to enable LLM inference with minimal setup and state-of-the-art performance across a wide range of hardware, locally and in the cloud.
The technical choice that defines both projects is the plain C/C++ implementation with no external dependencies. Mainstream machine-learning frameworks pull in large runtime stacks and assume a heavyweight environment; ggml deliberately does the opposite, compiling to a small, portable artifact that runs close to the metal. This is what lets the engine target everything from Apple Silicon and x86 CPUs with their vector instruction sets to GPUs through custom kernels, without dragging a framework along.
The second defining idea is quantization. The llama.cpp documentation describes support for integer quantization from roughly 1.5-bit through 8-bit, which compresses a model’s weights to a fraction of their original size. Quantization is what bridges the gap between a model that nominally needs many gigabytes of high-end accelerator memory and a laptop with limited RAM: by trading a controlled amount of numeric precision for a large reduction in footprint, a model that previously required server-class hardware becomes runnable on a personal machine.
Together these choices changed who could run a large language model. Before llama.cpp, using such a model generally meant either renting GPU time or calling a hosted API. By combining a dependency-free engine with aggressive quantization and hybrid CPU-plus-GPU execution for models that exceed available video memory, the project made fully local, offline inference an everyday possibility. That shift seeded a large community of local-model tooling that builds on the engine.
The projects also drove a companion standard. The need to ship quantized weights as self-contained, portable files led to the GGUF format, the single-file container that llama.cpp consumes, defined in the ggml repository. The library, the engine, and the file format form a coherent stack: ggml supplies the math, llama.cpp supplies the runtime and quantization, and GGUF supplies the distribution format. As a piece of systems software, the work is a reminder that careful, low-level engineering — not just larger models — can decide where a technology can actually run.