# text-generation-inference

**Licence:** Apache-2.0  |  **Language:** Python  |  **Source:** https://github.com/huggingface/text-generation-inference

## Role in the stack

Production Hugging Face model serving

## Why it's here

Continuous batching, token streaming and a production HTTP API for open models.

## Using it

See the project README in the fetched source:

```bash
./fetch-sources.sh ./sources
tar xzf ./sources/text-generation-inference.tar.gz -C ./sources
```

## Running it

Started by `docker compose up -d` on port 8080.