119 lines
10 KiB
Markdown
119 lines
10 KiB
Markdown
## Runtime Stack
|
||
|
||
**llama.cpp** is the inference runtime for all `frank-zealot`-class endpoints. The `frank-zealot` service runs llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build, under a root systemd unit.
|
||
|
||
## Model Inventory
|
||
|
||
### frank-zealot (Dankclaw: 192.168.0.143:7771)
|
||
|
||
**Ternary-Bonsai-2-27B-PQ2_0.gguf** (prism-ml/Ternary-Bonsai-2-27B-gguf)
|
||
|
||
- Runtime: llama.cpp
|
||
- Format: PQ2_0 quantized GGUF
|
||
- 27B parameter class
|
||
- Vision: native BF16 mmproj projector for image input
|
||
- MTP: OFF (no `nextn` tensors in checkpoint)
|
||
- Context: 256k per-slot window (`--ctx-size 401408 --parallel 2` = 200,704 tokens effective)
|
||
- Batch: 4096 / ubatch 2048 (CUDA optimum)
|
||
- Log verbosity: 4
|
||
- Ports: 7771 on Dankclaw
|
||
|
||
### frank-dragoon (Dankclaw: 192.168.0.143:7770)
|
||
|
||
**Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated** (`.hgn` format)
|
||
|
||
- Runtime: halogen-flash-server 0.16.2 (digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`)
|
||
- Format: v2 `.hgn` (flash-server exclusive)
|
||
- MTP head: INSIDE checkpoint (`HALOGEN_MTP_HEAD` unset)
|
||
- Vision: ON (v2 tower loads beside checkpoint)
|
||
- Base: abliterated (27B-era was censored finetune)
|
||
- Slots: 2 × 262,144 ctx (KV pool 524,288 positions)
|
||
- Server sampler: 1.0/0.95/20
|
||
- Prompt cache: ON by default
|
||
|
||
### frank-archon (Dankserver: 192.168.0.235:7770)
|
||
|
||
**StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe** (`.rad` format)
|
||
|
||
- Runtime: radiance 1.0.13 (digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`)
|
||
- Format: fp8-KV `.rad` (121,969,901,568 B = 116,322 MiB)
|
||
- MTP head: MTP depth 2, MNBT 2048, Froggeric v22.5 template
|
||
- Vision: 27-layer Qwen-VL tower native
|
||
- KV: 3 × 200,000 fp8-KV (`--max-num-seqs 3`)
|
||
- 30,720 MiB of 30,969 MiB claimable
|
||
- 44 GiB/rank pinned host pool, PLE pinned to RAM
|
||
- Base-alignment: censored (abliterated family lives on frank-dragoon)
|
||
|
||
## Resolution Rules
|
||
|
||
Official model ids are StarCraft unit names, on the wire since 2026-10-09, and since 2026-10-10 they are also the SERVICE names: `frank-archon` → `192.168.0.235:7770`, `frank-dragoon` → `192.168.0.143:7770`, `frank-zealot` → `192.168.0.143:7771`.
|
||
|
||
The default model resolves as a `defaultProvider` + `defaultModel` pair (provider must serve the model); a subagent override uses a bare model `id`, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port".
|
||
|
||
When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-10-06 **all three endpoints accept images** (`input: [text, image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap, while `frank-archon` takes them natively and `frank-zealot` through its BF16 mmproj.
|
||
|
||
Reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — `models.yml` sets `reasoningContentField` per endpoint to match.
|
||
|
||
There is **no audio** input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01.
|
||
|
||
## Boot Persistence & Self-Healing (MANDATORY)
|
||
|
||
Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from `compose.yaml` or a manual `podman run` dies silently at reboot — rootless Podman does not auto-start bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. Never report such a service as "up".
|
||
|
||
Unit requirements:
|
||
|
||
- **Boot persistence** — `[Install] WantedBy=default.target`, installed to `~/.config/containers/systemd/<name>.container`, enabled in the user manager, with linger on for the deploy user (`loginctl show-user dank -p Linger` → `Linger=yes`).
|
||
- **Self-healing** — `Restart=always` (or `on-failure`) with a short `RestartSec`; quadlet health check where the service has one.
|
||
- **Repo provenance** — the `.container` file lives in this repo (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.
|
||
|
||
Before declaring a service deployed: `systemctl --user is-enabled <name>` → `enabled`, `is-active` → `active`, and the container answers its health check.
|
||
|
||
A running container with no unit is a gap: write the unit, test it, deploy it.
|
||
|
||
This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under `dankclaw/observability/quadlet/` and their compose definitions were removed — one owner per container name. The stack is **configured and delivering as of 2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`, loopback :9093) → host-side Signal relay (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN IP, the one bind a container can reach as `host.containers.internal`; iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
|
||
|
||
Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||
|
||
## Benchmarking Policy (MANDATORY)
|
||
|
||
**STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench requires asking first.
|
||
|
||
Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself.
|
||
|
||
A gufo-era law: **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
|
||
|
||
Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, replay **real** text through the live server and read the `Server-Timing` header — but note the unit rides in the **name** half (`ttft;dur=402.485`), so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently returns nothing. Generate **at least 128 tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise and the run-to-run spread swamps every real difference.
|
||
|
||
This covers **any GPU experiment, not just benchmarks** — including a bare 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP.
|
||
|
||
1. Stop and mask (self-healing would restart it):
|
||
- Dankclaw `frank-dragoon` (root unit, halogen podman on the iGPU):
|
||
`sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon`
|
||
- Dankclaw `frank-zealot` (root unit, llama.cpp on the 3090):
|
||
`sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot`
|
||
- Dankserver `frank-archon` (root unit, radiance on both R9700s, TP=2):
|
||
`sudo systemctl stop frank-archon && sudo systemctl mask frank-archon`
|
||
2. Bench.
|
||
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in `/etc/systemd/system/` is itself a symlink into the checkout — mask refuses to replace it). Stop is enough for the bench window; the deploy restores the unit, so re-deploy after benching instead of unmasking.
|
||
3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.
|
||
|
||
**CRITICAL:** never `systemctl disable` a system unit on these hosts — it deletes the unit file from `/etc/systemd/system/`, not just the symlink. `mask` symlinks to `/dev/null` and is safe/reversible.
|
||
|
||
## Subagent Lifecycle (MANDATORY)
|
||
|
||
A subagent that appears stuck must have an explicit recovery path defined before the subagent is considered deployed. "Stuck" includes: silent completion halt, repeated identical responses, or extended latency without token emission.
|
||
|
||
Recovery requirements:
|
||
|
||
- **Explicit timeout** — a hard timeout (≤30s) on the subagent's response channel, with a defined action (retry, fallback, or escalate).
|
||
- **State checkpoint** — the subagent maintains checkpoints at decision boundaries, allowing resumption from a known good state.
|
||
- **Health signal** — the subagent publishes a heartbeat (≤5s interval) that triggers the recovery path when missed.
|
||
|
||
A subagent without these three properties is a gap: write the timeout, add the checkpoint, wire the heartbeat, then deploy.
|
||
|
||
This rule exists because subagent hangs were observed in early deployments (2026-09-xx: silent completion halt; 2026-09-xx: repeated identical responses). The deployment script must verify the heartbeat channel is live before reporting "deployed".
|
||
|
||
Before declaring a subagent deployed: verify the heartbeat endpoint returns within the timeout window, and confirm the recovery path is wired (retry/fallback/escalation logic exists in the codepath).
|
||
|
||
A subagent without a heartbeat or recovery path is a gap: wire the heartbeat, add the recovery path, then deploy.
|