68 lines
3.7 KiB
Markdown
68 lines
3.7 KiB
Markdown
---
|
|
name: halogen-flash-server-ops
|
|
description: "Operate the frank-large halogen-flash-server deployment on Dankclaw (model swaps, restarts, memory sizing, health and client verification, rollback) — use for anything touching frank-large after the 2026-09-11 runtime swap"
|
|
---
|
|
|
|
# halogen-flash-server ops (frank-large, Dankclaw)
|
|
|
|
frank-large is a **rootless Quadlet container** since 2026-09-11:
|
|
`dankclaw/frank-large/quadlet/frank-large.container` in the serverAppConfigs repo,
|
|
installed to `~/.config/containers/systemd/`, generated unit `frank-large.service`
|
|
in the **user** manager. Full runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
|
|
|
|
## Never use sudo systemctl on it
|
|
|
|
It is a user unit. `sudo systemctl stop frank-large` does nothing useful.
|
|
|
|
```bash
|
|
systemctl --user status frank-large.service
|
|
podman logs frank-large | head -40 # version banner + memory ledger
|
|
curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs
|
|
curl -s http://127.0.0.1:7770/v1/models # must list frank-large
|
|
```
|
|
|
|
## Deploy / change flags
|
|
|
|
1. Edit the `.container` file in the repo (all tunables are inline `Environment=` lines).
|
|
2. `./tests/run.sh` (the quadlet seam covers GPU devices, memlock, ipc/seccomp, the single published port, writable weights mount).
|
|
3. Commit, push, then `./scripts/deploy.sh dankclaw --service frank-large`.
|
|
|
|
The handler retires the old root unit (moving its symlink to the home trash), pulls and
|
|
re-tags the image to the `localhost/` alias, installs the unit, restarts it, and gates on
|
|
the download when the checkpoint is absent. Never `systemctl --user enable` a generated unit.
|
|
|
|
## Sizing a swap
|
|
|
|
Read the startup ledger before and after any change: it prints the locked weights, the KV
|
|
pool it chose, the working set, the BIOS iGPU carve-out, and the corrected free-memory
|
|
figure. `free`, MemAvailable and every monitor built on them overstate free memory by the
|
|
size of the model (~68 GiB) because the kernel counts the locked weights as reclaimable
|
|
page cache.
|
|
|
|
- KV is ~7.2 GiB per 262,144 positions; the prefill arena is 16.7 GiB at `HALOGEN_MAX_TOK=32768`.
|
|
- The device budget stops near 47 GiB *with the weights pinned*, and lower on a machine
|
|
carrying more besides this server. `HALOGEN_KV_POOL_FIT=1` lowers the pool to fit and
|
|
prints what it chose; a refused start needs a smaller `HALOGEN_KV_POOL_POSITIONS`
|
|
(`HALOGEN_KV_SLOTS` will not help — slots share one pool).
|
|
- Long prompts crawling with the disk busy = file cache, not memory: lower the pool or
|
|
raise `HALOGEN_HOST_RESERVE_GIB`.
|
|
|
|
## Client contract traps
|
|
|
|
- No server-side sampler defaults: a request without `temperature` decodes GREEDY.
|
|
- The per-request budget defaults to 8192 and **includes thinking**; a reasoning model
|
|
plus a small budget returns EMPTY content with `finish_reason: "length"`. Cap is 65536
|
|
(a 400 above it), and `max_tokens`/`max_completion_tokens`/`max_output_tokens` must agree.
|
|
- `response_format` is a 400 ("no constrained decoding") — as it is on FreeToken/frank-small.
|
|
Any caller that needs structured output or a tiny budget needs a different backend.
|
|
- A request reserves prompt + `max_tokens` positions against the pool; larger requests queue
|
|
and then fail with `503 engine_busy` after 3600 s.
|
|
|
|
## Verification after any change
|
|
|
|
Run a real completion (never just `/health`), a vision probe if the tower is enabled
|
|
(image content part with a `data:` URL; `http(s)` URLs are refused), then one call from the
|
|
client that consumes it. Rollback: the llama.cpp era is in `.AGENTTRASH/` (`.llama.cpp`),
|
|
the STRIX payload and ROCmFPX build stay on disk — but remove the Quadlet from
|
|
`~/.config/containers/systemd/` before restoring the root unit, or both claim port 7770.
|