llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot

The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
Frank committed 2026-10-10 17:03:14 -07:00
1 parent 5ced474eb9
commit 55f87fcdce
195 files changed
+2085 -2064

No files matched your search

@@ -0,0 +1,200 @@
# frank-dragoon — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
frank-dragoon is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
pinned digest, launched by `scripts/serve-frank-dragoon.sh` under a root
systemd unit as a podman container — serving
**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`,
35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors) with the
**DFlash2** drafter at **one slot x 200,700 ctx**.
It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]),
which remains the rollback target with its payload on disk.
## Why swap
|Metric|ROCmFPX Q6_0_ROCMFPX_AGENT + MTP (retired)|halogen qwen3.8-27b + DFlash2 (active)|
|---|---|---|
|Prose decode|27.6–28.1 t/s measured (4.8k depth, acceptance 0.868)|31.7 t/s mean across eight text workloads (publisher card, bf16 reference hardware)|
|Copy-echo decode|31.4 t/s|publisher reports echo inside the same 31.7 mean|
|Context|1 x 262,144|1 x 200,700|
|Format|GGUF (fork-private tensor types 102/103)|`.hgn` — loads in NO other engine|
The swap was accepted with eyes open on two product regressions, both
confirmed by Frank before deploy:
1. **Text-only.** halogen loads the language tower only; README "honest
limits": *"The vision encoder is ignored; image inputs are not supported."*
The mmproj-era vision role moves wholly to frank-zealot.
2. **Censored base.** The checkpoint is a finetune of the censored
`Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
## Shape and why
`scripts/frank-dragoon.env` carries every tunable with its evidence; the load-
bearing numbers:
- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
a step batches when more than one conversation is running, so the drafter
stays off with more than one slot"). Slots>1 silently loses DFlash2; the
sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-dragoon
was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
prompt is a 400 naming the limit, never a truncation. The harness side is
wired to it: `models.yml` `contextWindow: 200700`, and OMP compacts at 70%
= 140,490, leaving 60,210 of the slot for the reply.
- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
pair (FLAGS.md table): a request at the cap must not outlive the timeout.
65536 > the harness's 57,344 `maxTokens` so an eager request never meets
the engine's 400.
- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
and **prints a boot ledger naming any clamp** — the post-deploy check reads
that ledger, not a formula.
- **`HALOGEN_CACHE_MB=16384` (pinned 2026-10-06)** — auto is OFF on this
box: the floor is 2 x a full-context entry (36.9 GB at 262k; one entry
is ~13.7 GB at our 200,700-slot x ~70 KB/token) and the ledger had
35.7 GB free, so the engine refused and every deep agent request
re-prefilled its whole 79-81k context — 84.9-179.8 s measured
(serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. The
README prices the cache at ~20x time-to-first-token on a warm turn;
entries evict LRU by bytes; the reserve (HALOGEN_CACHE_RESERVE_MB,
default 8192) is re-checked against live MemAvailable on every store.
- **No server-side sampler:** an absent temperature is GREEDY. The card's
thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
— on the wire only. The template defaults `reasoning_effort` to **xhigh**
when a request sends none, and **thinking tokens consume the
`max_tokens` budget**: a budget exhausted mid-thought returns
`finish_reason: "length"` with EMPTY `content` and the partial reasoning
in `reasoning_content` — clients must read both fields.
## Client contract deltas vs llama.cpp
|Behavior|halogen|
|---|---|
|Prompt over slot|400 naming the limit (no truncation)|
|`max_tokens` over cap|400 naming the limit (never silent truncation)|
|full-length request outliving `HALOGEN_QUEUE_TIMEOUT`|503s for everyone queued|
|`timings`|every response carries llama-server-shape `timings` (`draft_n`, `draft_n_accepted`) — the drafter check reads the RESPONSE, and it proves speculation fired|
|`response_format`|NOT supported (no constrained decoding)|
|thinking field|`reasoning_content` (same as before)|
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|image input|ignored/error — route to frank-zealot|
## Deploy mechanics
- **Image:** `ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea`
(published tag 0.1.4). The deployer gates the exact digest exists in podman
storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
was the Flash-Next w4b era, different image, retired.
- **Payload:** `scripts/download-frank-dragoon-halogen.sh` (fetch law: `curl -f`,
exact byte count, LFS sha256 where the Hub publishes one, `--verify-only`,
token never in argv) into `/home/dank/models/halogen-qwen3.8-27b/`. The
tokenizer is a FLAT DIRECTORY (`tokenizer/`, 5 files — tokenizer.json,
vocab.json, merges.txt, chat_template.jinja, tokenizer_config.json),
mounted whole at `/tokenizer`; the engine's `HALOGEN_TOKENIZER` default is
that directory, so the container gets no tokenizer flag (model-card
docker-run parity). The deployer and serve script refuse a missing file or
ANY symlink: an HF cache snapshot symlinks each entry into `blobs/` and
those links dangle inside the container mount.
- **Unit:** `scripts/frank-dragoon.service` (root, `User=dank`, relinked into
`/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
stop -t 120 frank-dragoon-halogen`, script runs `podman run` FOREGROUND
(2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured
only the container ID — the engine ledger existed nowhere on disk and
the post-deploy grep below read a one-line hex file; foreground run
returns the container's exit code so `Restart=always` is real and the
ledger lands in the log). Engine stdout goes to
`/home/dank/logs/frank_dragoon.log` (the boot ledger lives there, not in
the journal).
- **Cutover:** `deploy_llm_dragoon` in `scripts/deploy-remote.sh` sweeps the
legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
the legacy `frank-dragoon` container name by NAME (never `pkill -f
llama-server` — that hits frank-zealot), retires the old quadlet if the 2026-09
halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
- **Bench law:** stop+mask the unit before any GPU experiment on this box
(AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
is loading, not failure.
## Verification (post-deploy checklist)
1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_dragoon.log`
— ledger shows the pool at the asked shape, DFlash2 active, the
prompt cache line naming the pinned budget, NO clamp line. EXPECT the
WARNING "smaller than ONE full-context entry (18.4 GB at
kMaxCtx=262144)": it sizes against the NATIVE window, not our
200,700 slot — the deepest possible entry (~14 GB at ~70 KB/token)
fits the 16 GB pin, and the 262k entry it fears cannot exist (over
slot is a 400). Probe-proven 2026-10-06: the warm turn hit with
cache_n=16,384.
2. `curl -s http://192.168.0.143:7770/health` -> 200.
3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
`reasoning_content` populated; decode rate from the timings.
4. Spec decode fires: multi-token step behavior visible in rate; compare
against DRAFTER=0 arm if in doubt (never against gufo — bench law).
5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-dragoon`
incl. a tool call (V1TOOLS).
6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
## Tuning notes (2026-10-06)
Measured on real agent traffic (serve_api lines) the day after cutover:
- **Decode is depth-bound, and the KV path is the reason.** Shallow
prompts (< 1k tokens) decode 30-50 t/s at commit 4-6.3/round; 46k runs
5.5, 63-67k runs 4.1-6.5, 79-81k runs 3.3-3.8. The round-time slope is
~8 us/token of context: the KV pool reads at ~8 GB/s effective while
the weights stream ~180-240 GB/s (the engine's README puts batch-1
weight streaming at the 240 GB/s hardware wall). Commit also falls
from 4-6 to 2.5-3 accepted/round as the conversation grows.
- **No supported flag touches decode.** `docs/FLAGS.md`: HALOGEN_W4A4 /
W4A4_EXCL are prefill-only ("decode is unaffected either way"); the
undocumented kernel levers are declared unsupported and their defaults
are the measured winners. 0.1.4 is the latest release (CHANGELOG
head; kernels unchanged since 0.1.0). The publisher's tg bench never
measured depth — its tg rows run over the ten SHORT eval prompt
shapes — so the README's "decode barely degrades at depth" claim was
never exercised at 46-80k.
- **A/B'd the one FA lever with public precedent (2026-10-06 stop-window
probe): rejected.** Issue #12's `HALOGEN_FA_TQ=256` (the 64 GB-carve
crash workaround) vs the default tile — same 80k filler prompt,
128-token essays, the runtime's own `timings`: decode 6.35 vs 6.34 t/s
cold and 5.74 vs 5.76 warm (±0.3 % = noise; draft acceptance identical,
53/74 and 58/69), while prefill LOST 9.5 % on this topology (447.5 ->
404.8 t/s). The depth collapse is not an FA tile artifact — it sits
deeper than the knob. The engine binary advertises 40+ undocumented
`HALOGEN_*` kernel levers (FA/DN/MATMUL families, readable via
`grep -aoE "HALOGEN_[A-Z0-9_]+"` on the binary); `FA_TQ` is the only
one with public evidence, and it moved nothing that matters here.
- **Routing is the applied mitigation (2026-10-06):** the harness's
deep-conversation roles (`slow`, `advisor` in `~/.omp/agent/config.yml`)
moved to frank-archon (radiance on Dankserver: fp8 KV at
7.32 KiB/token = 8.7x less KV to read per round at depth, MTP depth 3,
200k window, prefix-cache host+disk tiers). frank-dragoon keeps OpenWebUI
and shallow bursts, where it runs 30-50 t/s. frank-archon's
depth-decode rate is designed-for but not yet measured on record.
- **The lever that existed was the prompt cache, and it was OFF** (see
the shape bullet). The pin turns the 85-180 s re-prefill per deep
turn into a resume; decode itself is expected UNCHANGED by the pin —
the cache resumes prefill, it does not touch the round loop.
Probe-proven (two synthetic 17.3k turns, 2026-10-06): cold prefill
27.0 s (640 t/s, cache_n 0) -> warm turn cache_n 16,384, prompt_n 946
(the extension + one 2048-aligned chunk), prefill 1.7 s — 16x on the
pair, answer correct warm.
- **Everything else stays put.** dflash2 over mtp (publisher 31.71 vs
26.88 mean; at depth both pay the same KV term and dflash2's higher
commit wins), KV_SLOTS=1 (batch XOR speculation), SLOT_CTX=200700
(Frank's window), W4A4 shipped (+9 % prefill, decode unharmed).
## Rollback
Revert the frank-dragoon swap commit and redeploy `dankclaw --service frank-dragoon`:
the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
disk. The halogen payload can stay: it costs nothing while the unit points
elsewhere.