llamacpp

llama.cpp

Licence: MIT | Language: C++ | Source: https://github.com/ggml-org/llama.cpp

Role in the stack

Direct GGUF inference engine

Why it's here

The lowest-level way to run quantised GGUF models. Use it when you want full control over context, quantisation and sampling.

Using it

See the project README in the fetched source:

./fetch-sources.sh ./sources
tar xzf ./sources/llama.cpp.tar.gz -C ./sources

This component is a library, not a service - install it into your own application environment.