From 0900551317308b4a812eea31965782539872c9d0 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 5 Oct 2026 16:29:11 -0700 Subject: [PATCH] docs: frank-large halogen runbook, AGENTS.md + indices for the swap New runbook frank-large-halogen27b-dflash2.md; LLM + vault indices re-pointed, ROCmFPX q6agent runbook marked RETIRED (rollback target); AGENTS.md topology row, detail section, image-capability paragraph (two endpoints accept images now), bench-law prose and runbook index updated. --- AGENTS.md | 139 ++++++++---------- .../LLM/frank-large-halogen27b-dflash2.md | 119 +++++++++++++++ ...nk-large-rocmfpx-q6agent-uncensored-mtp.md | 6 +- serverAppConfigs-docs/LLM/index.md | 45 +++--- serverAppConfigs-docs/index.md | 12 +- 5 files changed, 219 insertions(+), 102 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md diff --git a/AGENTS.md b/AGENTS.md index 9816b4a..cac2a5b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved). Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, - Alloy, Pi-hole) plus TWO LLM services: `frank-large` (ROCmFPX llama.cpp - fork on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on + Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on + the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside @@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser … -| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`** (23.15 GiB, ftype 114 ≈ 7.44 bpw: 353 F32 + 159 `Q8_0_ROCMFPX` + 339 `GGML_TYPE_Q6_0_ROCMFPX` tensors — only this fork can read them — with 64 protected output-head tensors), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-03: prose decode **27.6–28.1 t/s** at 4,813-token depth (draft acceptance 396/456 = 0.868, mean acceptance length ≈ 4.4), schema-echo 31.4 t/s at 100 %; `ngram-mod` chained onto MTP measured **−15 % on prose** and was rejected (the chain disables the hybrid's recurrent-state rollback). Cutover 2026-10-03 from the aligned Q8 file (17.0 t/s, acceptance 0.462 — rollback payload still on disk) | 7770 | +| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (33.4 GiB; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (the sibling engine measured 41 → 14 t/s), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot is a 400 naming the limit, never a truncation; `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` is a 413. Cutover 2026-10-05|7770| | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -223,7 +223,7 @@ Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. -(A gufo-era law; frank-large is ROCmFPX llama.cpp since 2026-10-02, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.** +(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of @@ -249,7 +249,7 @@ one: `ip neigh show ` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP. 1. Stop and mask (self-healing would restart it): - - Dankclaw `frank-large` (root unit, vLLM podman on the iGPU): + - Dankclaw `frank-large` (root unit, halogen podman on the iGPU): `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` @@ -285,9 +285,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As -of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`): -`frank-large` through the BF16 mmproj that ships inside its GGUF payload, -`frank-primary` through a BF16 mmproj loaded explicitly (this build never +of 2026-10-05 **two of the three endpoints accept images** (`input: [text, +image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine +ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while +`frank-primary` takes them through a BF16 mmproj loaded explicitly (this build never autoloads projectors), and `frank-small` through `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is @@ -304,84 +305,59 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (the MiMo tower is vision-only). -### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE) +### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE) -The **kingjones30/ROCmFPX llama.cpp fork** at commit -`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by -`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`, -as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`, -`Restart=always`) running a bare `llama-server` (no container) on port 7770 -from the Strix Halo iGPU. Boot to healthy: **~15 s**. +The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, +digest-pinned, published tag 0.1.4) as a podman container driven by +`scripts/serve-frank-large.sh` under a **root systemd unit** +(`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman +stop -t 120 frank-large-halogen`, the script `exec podman wait` so the restart +policy is real) on port 7770 from the Strix Halo iGPU. Engine stdout goes to +`/home/dank/logs/frank_large.log` — **the boot ledger (KV pool, drafter state, +any clamp) lives there, not in the journal**. Serving +`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,880,329,102 B +≈ 33.4 GiB, p1 weights / w4 activations / d2 KV) + `frank-large.tokenizer.gguf` +(1,293,343 B). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork +([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose), +chosen for the publisher's 31.7 t/s DFlash2 mean. -**The runtime is not interchangeable — it is chosen by the FILE.** -`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF`'s protected-heads -rung, `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` (24,856,607,008 B, -sha256 `e966deb3…`, ftype 114 ≈ 7.44 bpw), stores **339 tensors of type 102, -`GGML_TYPE_Q6_0_ROCMFPX`, plus 159 type-103 `Q8_0_ROCMFPX`** — quantizations -that exist in no other loader: stock llama.cpp and gufo both refuse the file at -load (`unknown tensor type`). The AGENT construction keeps 64 output-head -tensors at plain precision (the protected-heads design of the aligned repo's -Q8 file, one quant rung down), and the uncensored tuning is what Frank swapped -to on 2026-10-03. Vision comes from the BF16 `mmproj-Qwen3.8-27B-BF16.gguf` -(931,145,888 B — hardlinked from the aligned payload dir, byte-identical). +**The format is the lock.** `.hgn` loads in nothing else — the runtime is +chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the +2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the +halogen-server repo restarted versioning at 0.1.x. -**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in -`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh` -checks the commit out by SHA, so a fork-side force-push cannot silently -re-break the endpoint. +**Two product regressions, both accepted by Frank before deploy:** the +endpoint is **text-only** (README honest limits: the vision encoder is +ignored — frank-small owns images now) and the checkpoint is a finetune of the +**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one +commit-revert away with the AGENT payload still on disk). -**Spec decode is the performance feature, and it is measured.** The draft is -the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…`, -`nextn_predict_layers = 1`), pinned to the same card: -`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0 ---spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`. -On the ALIGNED Q8 file (2026-10-02, bench windows, 2,202-token prompt, temp 0): -**MTP 17.00–17.14 t/s vs 7.60–7.66 with the draft stripped — +123 %**, -acceptance 0.462. On the UNCENSORED Q6 file (2026-10-03, production service, -prose at 4,813-token depth): **27.57–28.10 t/s decode, draft acceptance -396/456 = 0.868** (mean acceptance length ≈ 4.4 — drafting at nearly the full -n-max 4), schema-echo traffic 31.44 t/s at 639/639 = 1.000. The drafter is -materially better on this file; treat the card's 0.95 as a file property, not a -throughput promise. **`ngram-mod` chained onto MTP was measured and REJECTED -(2026-10-03):** `--spec-type draft-mtp,ngram-mod` cost −15 % on prose -(23.40 t/s) and −7 % on verbatim block-copy even with the matcher demonstrably -firing — the boot log says why: `recurrent state rollback is not compatible -with 'ngram-mod' - disabling rollback support`, and the hybrid's MTP -verification needs that rollback. Keep `draft-mtp` alone. +**Engine laws that shape the config** (all in `scripts/frank-large.env` with +evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the +drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD +ceiling (400 naming the limit, never truncation); `models.yml` +`contextWindow: 200700` must match or the harness discovers the wall as a 400. +`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled +pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 413). +`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent +temperature is greedy, so the thinking recipe exists only in `models.yml` +`extraBody`. `response_format` is unsupported (no constrained decoding). +Thinking lands in **`reasoning_content`** (harness field unchanged); the same +Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the +`models.yml` fold stays mandatory. -**Shape and budget:** ONE session at the native 262,144 window, unquantized -f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT -aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16 ---cache-ram 8192 --cache-idle-slots`. Measured GTT ~61 GiB under live load -(host baseline 16.7 GiB). Prompt cache verified (Q8 era): a repeat pass over -the same 6k prompt replays in 0.5 s. +Fetch: `scripts/download-frank-large-halogen.sh` into +`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + sha256 + +`--verify-only`, no token on the command line). The deployer and serve script +both refuse a **symlinked or directory tokenizer** — an HF snapshot layout +passes `test -f` but fails the engine's open at request time. -**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on -(without it `chat_template_kwargs` are silently ignored), tools are native -(`get_weather` → `{"city":"Seattle"}`), thinking lands in -**`reasoning_content`**, `reasoning_effort` accepts ONLY -`xhigh|medium|low` (anything else is HTTP 500 — `models.yml` folds `high` and -`max` to `xhigh`), and thinking is enabled server-side with the card's -recipe (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0) plus -`--reasoning-format deepseek --reasoning-budget -1`. Two traps: a request -whose whole budget is consumed by reasoning returns `content: null` (bench -clients must fall back to `reasoning_content` or they crash *after* -measuring), and the fork logs -`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft -resets — self-recovering, not a fault. - -Fetch: `scripts/download-frank-large-kingjones-uncensored.sh` (hardlinks the -byte-identical MTP head and projector from the aligned payload dir when -present, otherwise fetches; `-f` + byte count + LFS sha256 verified, -`--verify-only`, no token on the command line). - -Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md`. -Rollback: the ALIGNED Q8 file (the previous shape) remains on disk at -`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the -2026-10-03 swap commit (unit/env/serve trio) and redeploy. The older gufo tag -`frank-large-gufo-q4kxl-20261001` is now a **re-download** rollback: its -Q4_K_XL payload and DFlash2 drafter were removed from disk 2026-10-03 -(`scripts/build-gufo.sh` stays for the rebuild). +Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`. +Rollback: revert the swap commit and redeploy — the ROCmFPX payload +(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build +and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested. +The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download** +rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays). ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) @@ -1005,7 +981,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md` | +| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` | | frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` | | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | @@ -1015,6 +991,7 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| +| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` | | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` | | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | diff --git a/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md b/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md new file mode 100644 index 0000000..4d04867 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md @@ -0,0 +1,119 @@ +# frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05) + +frank-large is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since +2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a +pinned digest, launched by `scripts/serve-frank-large.sh` under a root +systemd unit as a podman container — serving +**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`, 35,880,329,102 B +≈ 33.4 GiB) with the **DFlash2** drafter at **one slot x 200,700 ctx**. + +It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-large-rocmfpx-q6agent-uncensored-mtp]]), +which remains the rollback target with its payload on disk. + +## Why swap + +|Metric|ROCmFPX Q6_0_ROCMFPX_AGENT + MTP (retired)|halogen qwen3.8-27b + DFlash2 (active)| +|---|---|---| +|Prose decode|27.6–28.1 t/s measured (4.8k depth, acceptance 0.868)|31.7 t/s mean across eight text workloads (publisher card, bf16 reference hardware)| +|Copy-echo decode|31.4 t/s|publisher reports echo inside the same 31.7 mean| +|Context|1 x 262,144|1 x 200,700| +|Format|GGUF (fork-private tensor types 102/103)|`.hgn` — loads in NO other engine| + +The swap was accepted with eyes open on two product regressions, both +confirmed by Frank before deploy: + +1. **Text-only.** halogen loads the language tower only; README "honest + limits": *"The vision encoder is ignored; image inputs are not supported."* + The mmproj-era vision role moves wholly to frank-small. +2. **Censored base.** The checkpoint is a finetune of the censored + `Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The + ROCmFPX AGENT file was an *Uncensored* build; refusals can return. + +## Shape and why + +`scripts/frank-large.env` carries every tunable with its evidence; the load- +bearing numbers: + +- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR + speculation, never both (FLAGS.md: "Batching and speculation are exclusive: + a step batches when more than one conversation is running, so the drafter + stays off with more than one slot"). Slots>1 silently loses DFlash2; the + sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-large + was a one-conversation endpoint before (PARALLEL=1), so the contract holds. +- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer + prompt is a 400 naming the limit, never a truncation. The harness side is + wired to it: `models.yml` `contextWindow: 200700`, and OMP compacts at 70% + = 140,490, leaving 60,210 of the slot for the reply. +- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled + pair (FLAGS.md table): a request at the cap must not outlive the timeout. + 65536 > the harness's 57,344 `maxTokens` so an eager request never meets + the engine's 413. +- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB + pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working + set + host baseline on the 121.2 GiB box. The engine sizes the pool itself + and **prints a boot ledger naming any clamp** — the post-deploy check reads + that ledger, not a formula. +- **No server-side sampler:** an absent temperature is GREEDY. The card's + thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody` + — on the wire only. + +## Client contract deltas vs llama.cpp + +|Behavior|halogen| +|---|---| +|Prompt over slot|400 naming the limit (no truncation)| +|`max_tokens` over cap|413, max_tokens over queue timeout|503s for the queue| +|`response_format`|NOT supported (no constrained decoding)| +|thinking field|`reasoning_content` (same as before)| +|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays| +|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy| +|image input|ignored/error — route to frank-small| + +## Deploy mechanics + +- **Image:** `ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea` + (published tag 0.1.4). The deployer gates the exact digest exists in podman + storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that + was the Flash-Next w4b era, different image, retired. +- **Payload:** `scripts/download-frank-large-halogen.sh` (fetch law: `curl -f`, + exact byte count, LFS/hub sha256, `--verify-only`, token never in argv) into + `/home/dank/models/halogen-qwen3.8-27b/`. The tokenizer bundle + (`frank-large.tokenizer.gguf`) must be a FLAT regular file — the deployer + and serve script both refuse a symlink or directory: an HF symlinked/snapshot + layout passes `test -f` but fails the engine's open at request time. +- **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into + `/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman + stop -t 120 frank-large-halogen`, script `exec podman wait` so + `Restart=always` is real. Engine stdout goes to + `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in the + journal). +- **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the + legacy bare `llama-server --model /home/dank/models/kingjones-*` process and + the legacy `frank-large` container name by NAME (never `pkill -f + llama-server` — that hits frank-small), retires the old quadlet if the 2026-09 + halogen era's unit lingers, waits port 7770 free, relinks, health-gates. +- **Bench law:** stop+mask the unit before any GPU experiment on this box + (AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that + is loading, not failure. + +## Verification (post-deploy checklist) + +1. `grep -iE 'clamp|kv|pool|drafter|dflash' /home/dank/logs/frank_large.log` + — ledger shows the pool at the asked shape, DFlash2 active, NO clamp line. +2. `curl -s http://192.168.0.143:7770/health` -> 200. +3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe; + `reasoning_content` populated; decode rate from the timings. +4. Spec decode fires: multi-token step behavior visible in rate; compare + against DRAFTER=0 arm if in doubt (never against gufo — bench law). +5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-large` + incl. a tool call (V1TOOLS). +6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test). + +## Rollback + +Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`: +the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), +the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its +`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on +disk. The halogen payload can stay: it costs nothing while the unit points +elsewhere. diff --git a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md index 5f2cfcb..d09bd07 100644 --- a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md +++ b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md @@ -17,7 +17,11 @@ timestamp: 2026-10-03 # frank-large — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151) -> **ACTIVE since 2026-10-03.** Cut over from +> **RETIRED 2026-10-03 → 2026-10-05 (rollback target; payload, fork build and +> build script all on disk).** Replaced by [[frank-large-halogen27b-dflash2]] +> (DFlash2, publisher 31.7 t/s mean vs the 27.6–28.1 measured here). This page +> remains the contract record for the rollback. Original banner follows. +> ACTIVE since 2026-10-03. Cut over from > [[frank-large-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s) > on Frank's request: the *uncensored* variant of the same protected-heads > family, one quant rung down. The swap turned out to be a strict upgrade on diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 682770a..9bd3814 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -14,11 +14,11 @@ mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every `torch.compile took 181.20 s`). `--max-num-seqs` must stay 3: SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001), and Dankclaw -runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770: -kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF -Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, a ftype-114 file only that fork can -read, with the repo's MTP head drafting 4 tokens at 27.6-28.1 t/s decode, ONE -session at 262,144, f16 KV, vision via the BF16 projector)** and +runs two endpoints — **halogen container (frank-large, port 7770: +ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai +/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other +engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision +lives on frank-small)** and **llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**. @@ -45,10 +45,10 @@ expert-precise-memory arena 7,522 slots, /root/.cache mounted] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[RTX 3090 (PCIe x4 dock) + gfx1151 iGPU 114 GiB GTT] - E[ROCmFPX llama.cpp gfx1151, pinned fork build + E[halogen container gfx1151, digest-pinned image frank-large, port 7770] - F[Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP-Q4_0 head -1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision] + F[halogen-qwen3.8-27b .hgn p1w4d-d2 + DFlash2 drafter +1 x 200700 ctx, d2 KV, text-only] G[llama.cpp master CUDA sm_86 frank-small, port 7771] H[MiMo-V2.6 Q5_K_M GGUF @@ -71,16 +71,17 @@ frank-small, port 7771] ## Active Deployment -- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **ACTIVE since 2026-10-03** - (root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving - the uncensored protected-heads rung `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` - (23.15 GiB, ftype 114 = 159 type-103 + 339 GGML_TYPE_Q6_0_ROCMFPX tensors, - 64 protected output heads) with the repo's `mtp-Qwen3.8-27B-Q4_0.gguf` head - drafting 4 tokens, one 262,144 window, f16 KV, the BF16 projector for - vision. Measured 27.6-28.1 t/s prose decode at 86.8 % draft acceptance; the - ngram-mod chain onto MTP was measured and REJECTED (-15 %: it disables the - hybrid's recurrent state rollback). Rollback: the aligned Q8 file, still on - disk +- [[frank-large-halogen27b-dflash2]] — **ACTIVE since 2026-10-05** + (root systemd unit + podman, port 7770): the digest-pinned + `ghcr.io/peonist-ai/halogen` image serving `peonist-ai/halogen-qwen3.8-27b` + (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB) with the DFlash2 drafter on ONE slot + of 200,700 ctx (slots>1 disables speculation — engine law). Publisher card + 31.7 t/s mean across eight text workloads. Accepted tradeoffs: TEXT-ONLY + (the engine ignores the vision encoder; frank-small owns images) and the + CENSORED base model (the ROCmFPX AGENT file was Uncensored by construction). + No server-side sampler — the thinking recipe rides the wire in models.yml. + Rollback: revert the swap commit; the ROCmFPX payload + fork build + + `build-rocmfpx.sh` pin all remain on disk - [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO @@ -202,6 +203,14 @@ frank-small, port 7771] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change +- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 → + 10-05 (rollback target; payload, fork build and build script on disk)**: + the pinned ROCmFPX fork serving the uncensored protected-heads + Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens, + one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 % + acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP + rejection (-15 %, recurrent-state rollback) stays a live lesson for this + architecture. - [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the rollback target; payload still on disk)**: the same pinned ROCmFPX fork serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 2d55e5c..0bf2cfd 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -213,8 +213,16 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — the ACTIVE frank-large - runbook since 2026-10-03 (root systemd unit, port 7770): the pinned +- [[frank-large-halogen27b-dflash2]] — the ACTIVE frank-large runbook since + 2026-10-05 (root systemd unit + podman, port 7770): the digest-pinned + ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b + (qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other + engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1 + disables speculation — engine law); TEXT-ONLY and censored base, both + accepted tradeoffs; no server-side sampler, so the thinking recipe rides + the wire +- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — RETIRED 2026-10-03 → 10-05 + (rollback target; payload, fork build and build script on disk): the pinned kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/ Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF **Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT** — ftype 114, 339 tensors of