Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
201 lines
12 KiB
Markdown
201 lines
12 KiB
Markdown
# frank-dragoon — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
|
||
|
||
frank-dragoon is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
|
||
2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
|
||
pinned digest, launched by `scripts/serve-frank-dragoon.sh` under a root
|
||
systemd unit as a podman container — serving
|
||
**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`,
|
||
35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors) with the
|
||
**DFlash2** drafter at **one slot x 200,700 ctx**.
|
||
|
||
It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]),
|
||
which remains the rollback target with its payload on disk.
|
||
|
||
## Why swap
|
||
|
||
|Metric|ROCmFPX Q6_0_ROCMFPX_AGENT + MTP (retired)|halogen qwen3.8-27b + DFlash2 (active)|
|
||
|---|---|---|
|
||
|Prose decode|27.6–28.1 t/s measured (4.8k depth, acceptance 0.868)|31.7 t/s mean across eight text workloads (publisher card, bf16 reference hardware)|
|
||
|Copy-echo decode|31.4 t/s|publisher reports echo inside the same 31.7 mean|
|
||
|Context|1 x 262,144|1 x 200,700|
|
||
|Format|GGUF (fork-private tensor types 102/103)|`.hgn` — loads in NO other engine|
|
||
|
||
The swap was accepted with eyes open on two product regressions, both
|
||
confirmed by Frank before deploy:
|
||
|
||
1. **Text-only.** halogen loads the language tower only; README "honest
|
||
limits": *"The vision encoder is ignored; image inputs are not supported."*
|
||
The mmproj-era vision role moves wholly to frank-zealot.
|
||
2. **Censored base.** The checkpoint is a finetune of the censored
|
||
`Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
|
||
ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
|
||
|
||
## Shape and why
|
||
|
||
`scripts/frank-dragoon.env` carries every tunable with its evidence; the load-
|
||
bearing numbers:
|
||
|
||
- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
|
||
speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
|
||
a step batches when more than one conversation is running, so the drafter
|
||
stays off with more than one slot"). Slots>1 silently loses DFlash2; the
|
||
sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-dragoon
|
||
was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
|
||
- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
|
||
prompt is a 400 naming the limit, never a truncation. The harness side is
|
||
wired to it: `models.yml` `contextWindow: 200700`, and OMP compacts at 70%
|
||
= 140,490, leaving 60,210 of the slot for the reply.
|
||
- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
|
||
pair (FLAGS.md table): a request at the cap must not outlive the timeout.
|
||
65536 > the harness's 57,344 `maxTokens` so an eager request never meets
|
||
the engine's 400.
|
||
- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
|
||
pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
|
||
set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
|
||
and **prints a boot ledger naming any clamp** — the post-deploy check reads
|
||
that ledger, not a formula.
|
||
- **`HALOGEN_CACHE_MB=16384` (pinned 2026-10-06)** — auto is OFF on this
|
||
box: the floor is 2 x a full-context entry (36.9 GB at 262k; one entry
|
||
is ~13.7 GB at our 200,700-slot x ~70 KB/token) and the ledger had
|
||
35.7 GB free, so the engine refused and every deep agent request
|
||
re-prefilled its whole 79-81k context — 84.9-179.8 s measured
|
||
(serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. The
|
||
README prices the cache at ~20x time-to-first-token on a warm turn;
|
||
entries evict LRU by bytes; the reserve (HALOGEN_CACHE_RESERVE_MB,
|
||
default 8192) is re-checked against live MemAvailable on every store.
|
||
- **No server-side sampler:** an absent temperature is GREEDY. The card's
|
||
thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
|
||
— on the wire only. The template defaults `reasoning_effort` to **xhigh**
|
||
when a request sends none, and **thinking tokens consume the
|
||
`max_tokens` budget**: a budget exhausted mid-thought returns
|
||
`finish_reason: "length"` with EMPTY `content` and the partial reasoning
|
||
in `reasoning_content` — clients must read both fields.
|
||
|
||
## Client contract deltas vs llama.cpp
|
||
|
||
|Behavior|halogen|
|
||
|---|---|
|
||
|Prompt over slot|400 naming the limit (no truncation)|
|
||
|`max_tokens` over cap|400 naming the limit (never silent truncation)|
|
||
|full-length request outliving `HALOGEN_QUEUE_TIMEOUT`|503s for everyone queued|
|
||
|`timings`|every response carries llama-server-shape `timings` (`draft_n`, `draft_n_accepted`) — the drafter check reads the RESPONSE, and it proves speculation fired|
|
||
|`response_format`|NOT supported (no constrained decoding)|
|
||
|thinking field|`reasoning_content` (same as before)|
|
||
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
|
||
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|
||
|image input|ignored/error — route to frank-zealot|
|
||
|
||
## Deploy mechanics
|
||
|
||
- **Image:** `ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea`
|
||
(published tag 0.1.4). The deployer gates the exact digest exists in podman
|
||
storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
|
||
was the Flash-Next w4b era, different image, retired.
|
||
- **Payload:** `scripts/download-frank-dragoon-halogen.sh` (fetch law: `curl -f`,
|
||
exact byte count, LFS sha256 where the Hub publishes one, `--verify-only`,
|
||
token never in argv) into `/home/dank/models/halogen-qwen3.8-27b/`. The
|
||
tokenizer is a FLAT DIRECTORY (`tokenizer/`, 5 files — tokenizer.json,
|
||
vocab.json, merges.txt, chat_template.jinja, tokenizer_config.json),
|
||
mounted whole at `/tokenizer`; the engine's `HALOGEN_TOKENIZER` default is
|
||
that directory, so the container gets no tokenizer flag (model-card
|
||
docker-run parity). The deployer and serve script refuse a missing file or
|
||
ANY symlink: an HF cache snapshot symlinks each entry into `blobs/` and
|
||
those links dangle inside the container mount.
|
||
- **Unit:** `scripts/frank-dragoon.service` (root, `User=dank`, relinked into
|
||
`/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
|
||
stop -t 120 frank-dragoon-halogen`, script runs `podman run` FOREGROUND
|
||
(2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured
|
||
only the container ID — the engine ledger existed nowhere on disk and
|
||
the post-deploy grep below read a one-line hex file; foreground run
|
||
returns the container's exit code so `Restart=always` is real and the
|
||
ledger lands in the log). Engine stdout goes to
|
||
`/home/dank/logs/frank_dragoon.log` (the boot ledger lives there, not in
|
||
the journal).
|
||
- **Cutover:** `deploy_llm_dragoon` in `scripts/deploy-remote.sh` sweeps the
|
||
legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
|
||
the legacy `frank-dragoon` container name by NAME (never `pkill -f
|
||
llama-server` — that hits frank-zealot), retires the old quadlet if the 2026-09
|
||
halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
|
||
- **Bench law:** stop+mask the unit before any GPU experiment on this box
|
||
(AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
|
||
is loading, not failure.
|
||
|
||
## Verification (post-deploy checklist)
|
||
|
||
1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_dragoon.log`
|
||
— ledger shows the pool at the asked shape, DFlash2 active, the
|
||
prompt cache line naming the pinned budget, NO clamp line. EXPECT the
|
||
WARNING "smaller than ONE full-context entry (18.4 GB at
|
||
kMaxCtx=262144)": it sizes against the NATIVE window, not our
|
||
200,700 slot — the deepest possible entry (~14 GB at ~70 KB/token)
|
||
fits the 16 GB pin, and the 262k entry it fears cannot exist (over
|
||
slot is a 400). Probe-proven 2026-10-06: the warm turn hit with
|
||
cache_n=16,384.
|
||
2. `curl -s http://192.168.0.143:7770/health` -> 200.
|
||
3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
|
||
`reasoning_content` populated; decode rate from the timings.
|
||
4. Spec decode fires: multi-token step behavior visible in rate; compare
|
||
against DRAFTER=0 arm if in doubt (never against gufo — bench law).
|
||
5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-dragoon`
|
||
incl. a tool call (V1TOOLS).
|
||
6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
|
||
|
||
## Tuning notes (2026-10-06)
|
||
|
||
Measured on real agent traffic (serve_api lines) the day after cutover:
|
||
|
||
- **Decode is depth-bound, and the KV path is the reason.** Shallow
|
||
prompts (< 1k tokens) decode 30-50 t/s at commit 4-6.3/round; 46k runs
|
||
5.5, 63-67k runs 4.1-6.5, 79-81k runs 3.3-3.8. The round-time slope is
|
||
~8 us/token of context: the KV pool reads at ~8 GB/s effective while
|
||
the weights stream ~180-240 GB/s (the engine's README puts batch-1
|
||
weight streaming at the 240 GB/s hardware wall). Commit also falls
|
||
from 4-6 to 2.5-3 accepted/round as the conversation grows.
|
||
- **No supported flag touches decode.** `docs/FLAGS.md`: HALOGEN_W4A4 /
|
||
W4A4_EXCL are prefill-only ("decode is unaffected either way"); the
|
||
undocumented kernel levers are declared unsupported and their defaults
|
||
are the measured winners. 0.1.4 is the latest release (CHANGELOG
|
||
head; kernels unchanged since 0.1.0). The publisher's tg bench never
|
||
measured depth — its tg rows run over the ten SHORT eval prompt
|
||
shapes — so the README's "decode barely degrades at depth" claim was
|
||
never exercised at 46-80k.
|
||
- **A/B'd the one FA lever with public precedent (2026-10-06 stop-window
|
||
probe): rejected.** Issue #12's `HALOGEN_FA_TQ=256` (the 64 GB-carve
|
||
crash workaround) vs the default tile — same 80k filler prompt,
|
||
128-token essays, the runtime's own `timings`: decode 6.35 vs 6.34 t/s
|
||
cold and 5.74 vs 5.76 warm (±0.3 % = noise; draft acceptance identical,
|
||
53/74 and 58/69), while prefill LOST 9.5 % on this topology (447.5 ->
|
||
404.8 t/s). The depth collapse is not an FA tile artifact — it sits
|
||
deeper than the knob. The engine binary advertises 40+ undocumented
|
||
`HALOGEN_*` kernel levers (FA/DN/MATMUL families, readable via
|
||
`grep -aoE "HALOGEN_[A-Z0-9_]+"` on the binary); `FA_TQ` is the only
|
||
one with public evidence, and it moved nothing that matters here.
|
||
- **Routing is the applied mitigation (2026-10-06):** the harness's
|
||
deep-conversation roles (`slow`, `advisor` in `~/.omp/agent/config.yml`)
|
||
moved to frank-archon (radiance on Dankserver: fp8 KV at
|
||
7.32 KiB/token = 8.7x less KV to read per round at depth, MTP depth 3,
|
||
200k window, prefix-cache host+disk tiers). frank-dragoon keeps OpenWebUI
|
||
and shallow bursts, where it runs 30-50 t/s. frank-archon's
|
||
depth-decode rate is designed-for but not yet measured on record.
|
||
- **The lever that existed was the prompt cache, and it was OFF** (see
|
||
the shape bullet). The pin turns the 85-180 s re-prefill per deep
|
||
turn into a resume; decode itself is expected UNCHANGED by the pin —
|
||
the cache resumes prefill, it does not touch the round loop.
|
||
Probe-proven (two synthetic 17.3k turns, 2026-10-06): cold prefill
|
||
27.0 s (640 t/s, cache_n 0) -> warm turn cache_n 16,384, prompt_n 946
|
||
(the extension + one 2048-aligned chunk), prefill 1.7 s — 16x on the
|
||
pair, answer correct warm.
|
||
- **Everything else stays put.** dflash2 over mtp (publisher 31.71 vs
|
||
26.88 mean; at depth both pay the same KV term and dflash2's higher
|
||
commit wins), KV_SLOTS=1 (batch XOR speculation), SLOT_CTX=200700
|
||
(Frank's window), W4A4 shipped (+9 % prefill, decode unharmed).
|
||
|
||
## Rollback
|
||
|
||
Revert the frank-dragoon swap commit and redeploy `dankclaw --service frank-dragoon`:
|
||
the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
|
||
the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
|
||
`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
|
||
disk. The halogen payload can stay: it costs nothing while the unit points
|
||
elsewhere.
|