docs(llama-cpp): deployment detail goes back to the setup page
The overview page carried `docker run` invocations (including a CUDA variant, on a fleet with no NVIDIA card left in it) and the `LLAMA_ARG_` environment rule, while `setup.md` — the page whose own intro promises "build instructions, Docker, and systemd deployment" — carried the Compose example and the same env form in use. Two places to look, one place to be wrong. `setup.md` now states the env-prefix rule where the flags live (and why units prefer it: one flag per line is auditable, a forty-flag command line is not), and the overview keeps only the API surface and the GGUF format, pointing at the sections that absorbed the rest. Refs #64
This commit is contained in:
1 parent
8b87b5ae1a
commit
ec19771b8b
2 files changed
+11
-26
No files matched your search
@@ -124,31 +124,8 @@ print(completion.choices[0].message.content)
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Docker
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker run -p 8080:8080 -v /models:/models \
|
|
||||||
ghcr.io/ggml-org/llama.cpp:server \
|
|
||||||
-m /models/model.gguf --host 0.0.0.0 --port 8080
|
|
||||||
|
|
||||||
# CUDA variant
|
|
||||||
docker run -p 8080:8080 --gpus all \
|
|
||||||
ghcr.io/ggml-org/llama.cpp:server-cuda \
|
|
||||||
-m /models/model.gguf --n-gpu-layers 99
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Environment Variables
|
|
||||||
|
|
||||||
All CLI options can be set via `LLAMA_ARG_` prefix:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192 llama-server
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Cross-References
|
## Cross-References
|
||||||
|
|
||||||
- See [Setup Guide](setup.md) for build instructions, GPU offloading, and deployment.
|
- See [Setup Guide](setup.md) for build instructions, GPU offloading, the `LLAMA_ARG_*`
|
||||||
|
environment form, the Docker Compose example, and running under systemd. Deployment lives
|
||||||
|
there; this page is the API surface and the GGUF format.
|
||||||
@@ -111,6 +111,14 @@ cmake --build build --config Release -t llama-server
|
|||||||
| `-lora FNAME` | LoRA adapter (repeatable) | — |
|
| `-lora FNAME` | LoRA adapter (repeatable) | — |
|
||||||
| `-ngl, --gpu-layers N` | Layers to offload to GPU | auto |
|
| `-ngl, --gpu-layers N` | Layers to offload to GPU | auto |
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
Every CLI option can be set from the environment instead: prefix the upper-snake form of the
|
||||||
|
long flag with `LLAMA_ARG_` — `LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192
|
||||||
|
llama-server`. This is how the container and systemd units pass configuration (see the Compose
|
||||||
|
example below), because a unit file with one flag per line is auditable and a shell command line
|
||||||
|
with forty is not.
|
||||||
|
|
||||||
### Context & Performance
|
### Context & Performance
|
||||||
|
|
||||||
| Flag | Description | Default |
|
| Flag | Description | Default |
|
||||||
|
|||||||
Reference in new issue
Block a user