docs: frank-large halogen runbook, AGENTS.md + indices for the swap
New runbook frank-large-halogen27b-dflash2.md; LLM + vault indices re-pointed, ROCmFPX q6agent runbook marked RETIRED (rollback target); AGENTS.md topology row, detail section, image-capability paragraph (two endpoints accept images now), bench-law prose and runbook index updated.
This commit is contained in:
Frank committed
1 parent
fece3ffdd0
commit
0900551317
5 files changed
+219
-102
No files matched your search
@@ -0,0 +1,119 @@
|
||||
# frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
|
||||
|
||||
frank-large is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
|
||||
2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
|
||||
pinned digest, launched by `scripts/serve-frank-large.sh` under a root
|
||||
systemd unit as a podman container — serving
|
||||
**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`, 35,880,329,102 B
|
||||
≈ 33.4 GiB) with the **DFlash2** drafter at **one slot x 200,700 ctx**.
|
||||
|
||||
It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-large-rocmfpx-q6agent-uncensored-mtp]]),
|
||||
which remains the rollback target with its payload on disk.
|
||||
|
||||
## Why swap
|
||||
|
||||
|Metric|ROCmFPX Q6_0_ROCMFPX_AGENT + MTP (retired)|halogen qwen3.8-27b + DFlash2 (active)|
|
||||
|---|---|---|
|
||||
|Prose decode|27.6–28.1 t/s measured (4.8k depth, acceptance 0.868)|31.7 t/s mean across eight text workloads (publisher card, bf16 reference hardware)|
|
||||
|Copy-echo decode|31.4 t/s|publisher reports echo inside the same 31.7 mean|
|
||||
|Context|1 x 262,144|1 x 200,700|
|
||||
|Format|GGUF (fork-private tensor types 102/103)|`.hgn` — loads in NO other engine|
|
||||
|
||||
The swap was accepted with eyes open on two product regressions, both
|
||||
confirmed by Frank before deploy:
|
||||
|
||||
1. **Text-only.** halogen loads the language tower only; README "honest
|
||||
limits": *"The vision encoder is ignored; image inputs are not supported."*
|
||||
The mmproj-era vision role moves wholly to frank-small.
|
||||
2. **Censored base.** The checkpoint is a finetune of the censored
|
||||
`Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
|
||||
ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
|
||||
|
||||
## Shape and why
|
||||
|
||||
`scripts/frank-large.env` carries every tunable with its evidence; the load-
|
||||
bearing numbers:
|
||||
|
||||
- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
|
||||
speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
|
||||
a step batches when more than one conversation is running, so the drafter
|
||||
stays off with more than one slot"). Slots>1 silently loses DFlash2; the
|
||||
sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-large
|
||||
was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
|
||||
- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
|
||||
prompt is a 400 naming the limit, never a truncation. The harness side is
|
||||
wired to it: `models.yml` `contextWindow: 200700`, and OMP compacts at 70%
|
||||
= 140,490, leaving 60,210 of the slot for the reply.
|
||||
- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
|
||||
pair (FLAGS.md table): a request at the cap must not outlive the timeout.
|
||||
65536 > the harness's 57,344 `maxTokens` so an eager request never meets
|
||||
the engine's 413.
|
||||
- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
|
||||
pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
|
||||
set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
|
||||
and **prints a boot ledger naming any clamp** — the post-deploy check reads
|
||||
that ledger, not a formula.
|
||||
- **No server-side sampler:** an absent temperature is GREEDY. The card's
|
||||
thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
|
||||
— on the wire only.
|
||||
|
||||
## Client contract deltas vs llama.cpp
|
||||
|
||||
|Behavior|halogen|
|
||||
|---|---|
|
||||
|Prompt over slot|400 naming the limit (no truncation)|
|
||||
|`max_tokens` over cap|413, max_tokens over queue timeout|503s for the queue|
|
||||
|`response_format`|NOT supported (no constrained decoding)|
|
||||
|thinking field|`reasoning_content` (same as before)|
|
||||
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays|
|
||||
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|
||||
|image input|ignored/error — route to frank-small|
|
||||
|
||||
## Deploy mechanics
|
||||
|
||||
- **Image:** `ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea`
|
||||
(published tag 0.1.4). The deployer gates the exact digest exists in podman
|
||||
storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
|
||||
was the Flash-Next w4b era, different image, retired.
|
||||
- **Payload:** `scripts/download-frank-large-halogen.sh` (fetch law: `curl -f`,
|
||||
exact byte count, LFS/hub sha256, `--verify-only`, token never in argv) into
|
||||
`/home/dank/models/halogen-qwen3.8-27b/`. The tokenizer bundle
|
||||
(`frank-large.tokenizer.gguf`) must be a FLAT regular file — the deployer
|
||||
and serve script both refuse a symlink or directory: an HF symlinked/snapshot
|
||||
layout passes `test -f` but fails the engine's open at request time.
|
||||
- **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into
|
||||
`/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
|
||||
stop -t 120 frank-large-halogen`, script `exec podman wait` so
|
||||
`Restart=always` is real. Engine stdout goes to
|
||||
`/home/dank/logs/frank_large.log` (the boot ledger lives there, not in the
|
||||
journal).
|
||||
- **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the
|
||||
legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
|
||||
the legacy `frank-large` container name by NAME (never `pkill -f
|
||||
llama-server` — that hits frank-small), retires the old quadlet if the 2026-09
|
||||
halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
|
||||
- **Bench law:** stop+mask the unit before any GPU experiment on this box
|
||||
(AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
|
||||
is loading, not failure.
|
||||
|
||||
## Verification (post-deploy checklist)
|
||||
|
||||
1. `grep -iE 'clamp|kv|pool|drafter|dflash' /home/dank/logs/frank_large.log`
|
||||
— ledger shows the pool at the asked shape, DFlash2 active, NO clamp line.
|
||||
2. `curl -s http://192.168.0.143:7770/health` -> 200.
|
||||
3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
|
||||
`reasoning_content` populated; decode rate from the timings.
|
||||
4. Spec decode fires: multi-token step behavior visible in rate; compare
|
||||
against DRAFTER=0 arm if in doubt (never against gufo — bench law).
|
||||
5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-large`
|
||||
incl. a tool call (V1TOOLS).
|
||||
6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
|
||||
|
||||
## Rollback
|
||||
|
||||
Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`:
|
||||
the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
|
||||
the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
|
||||
`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
|
||||
disk. The halogen payload can stay: it costs nothing while the unit points
|
||||
elsewhere.
|
||||
Reference in new issue
Block a user