dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-large flash-server v2 cutover — runbook, index, AGENTS.md

Browse Source 
- New runbook: frank-large-flash-server-v2-swift15.md (engine, shape,
  memory model, engine laws, boot ledger, probes, rollback)
- LLM/index.md: active entry + retired halogen 27B entry
- Vault index: link the new runbook from the LLM Fleet section
- AGENTS.md: fleet topology table, Model Config section,
  Runbook Index, retired table (halogen 27B → 10-06)
This commit is contained in:
Frank
2026-10-06 03:35:27 -07:00
parent 20d7182f4b
commit 2d89bf97cc
3 changed files with 312 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+94 -77
View File
@@ -12,8 +12,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on
the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
container on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the
RTX 3090). Caddy on
Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -25,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tightest fit in the endpoint's history — **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **63.3 t/s recipe decode** (MTP acceptance 49.8 %; 48.0 no-spec). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771|
Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -234,7 +235,10 @@ Scratch ports are not used on LLM hosts — a
test process cannot share the single GPU anyway; validate against the service
itself.
(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
settings, and will invert the ranking.**
Measured 2026-09-28 on frank-large at depth 32768: its corpus
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
draft widths, i.e. the same highly-predictable text every run, and 93 % of
@@ -296,9 +300,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a provider the file does not define fails at call time. As
of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
of 2026-10-06 **all three endpoints accept images** (`input: [text,
image]`) — `frank-large` returned to vision with the flash-server v2 swap
(the v2 vision tower loads beside the checkpoint; the text-only halogen
27B window closed 2026-10-06) — while
`frank-primary` takes them natively (radiance runs the checkpoint's
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
are the engine core; `--mm-max-patches` available),
@@ -318,82 +323,93 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
(the MiMo tower is vision-only).
### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE)
### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`,
digest-pinned, published tag 0.1.4) as a podman container driven by
`scripts/serve-frank-large.sh` under a **root systemd unit**
The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
`ExecStop=podman
stop -t 120 frank-large-halogen`, the script runs `podman run` FOREGROUND —
the container's exit code is the service's exit so `Restart=always` stays
real, and the engine ledger lands on disk) on port 7770 from the Strix Halo
iGPU. Engine stdout goes to `/home/dank/logs/frank_large.log` — **the boot
ledger (KV pool, prompt cache, drafter state, any clamp) lives there, not in
the journal** (the 2026-10-05 `-d` + `podman wait` shape captured only the
container ID — fixed 2026-10-06).
Serving
`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B
≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations /
d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork
([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose),
chosen for the publisher's 31.7 t/s DFlash2 mean.
`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
run` FOREGROUND — the container's exit code is the service's exit so
`Restart=always` stays real, and the engine ledger lands on disk) on port
7770 from the Strix Halo iGPU. Engine stdout goes to
`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
Cutover 2026-10-06 from the halogen 27B-dense era
([[frank-large-halogen27b-dflash2]], retired the same window).
**The format is the lock.** `.hgn` loads in nothing else — the runtime is
chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the
2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the
halogen-server repo restarted versioning at 0.1.x.
Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B
append-only original; the abliteration workflow appends and the repo's
tooling drops dead padding, live tensors byte-identical) + the **REQUIRED**
n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B,
peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn`
(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files,
fetched fresh from peonist). Cutover 2026-10-06 from the halogen
27B-dense shape, chosen by Frank for the ABLITERATED base.
**Two product regressions, both accepted by Frank before deploy:** the
endpoint is **text-only** (README honest limits: the vision encoder is
ignored — frank-small owns images now) and the checkpoint is a finetune of the
**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one
commit-revert away with the AGENT payload still on disk).
**The format is the lock.** The v2 `.hgn` loads ONLY in the
flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the
engine moved with the model. The MTP head is INSIDE the checkpoint
(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn`
sidecar is deliberately not fetched).
**Engine laws that shape the config** (all in `scripts/frank-large.env` with
evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the
drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD
ceiling (400 naming the limit, never truncation); `models.yml`
`contextWindow: 200700` must match or the harness discovers the wall as a 400.
`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 400
naming the limit — README, never a 413). Thinking tokens COUNT against the
budget: exhausted mid-thought is `finish_reason: "length"` with empty
`content` and the reasoning in `reasoning_content`.
`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent
temperature is greedy, so the thinking recipe exists only in `models.yml`
`extraBody`. `response_format` is unsupported (no constrained decoding).
Thinking lands in **`reasoning_content`** (harness field unchanged); the same
Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the
`models.yml` fold stays mandatory.
`HALOGEN_CACHE_MB=16384` pins the prompt cache (2026-10-06): auto is OFF
on this box — the floor is 2 full-context entries (36.9 GB) and the boot
ledger had 35.7 GB free — so every deep request re-prefilled its whole
79-81k context (84.9-179.8 s per request for ~500-650 new tokens per
turn) until the pin. Probe-verified: a 17.3k cold prefill of 27.0 s
resumes in 1.7 s on the next turn (cache_n 16,384). Decode-at-depth is
structural: measured 5.5 t/s at 46k → 3.3-3.8 at 80k vs 30-50 shallow
(the KV pool reads at ~8 GB/s effective vs ~180-240 for the weights);
`docs/FLAGS.md` documents no decode-affecting lever and 0.1.4 is the
latest engine. The boot WARNING "smaller than ONE full-context entry
(18.4 GB at kMaxCtx=262144)" is a false positive for our slot: the
deepest possible entry is 200,700 × ~70 KB ≈ 14 GB, inside the pin, and
the 262k entry it fears cannot exist (over-slot is a 400).
**Product facts vs the 27B era, both regressions FIXED:** the base is
**abliterated** (the 27B was a censored-base finetune; refusals were an
accepted trade there) and **vision is ON** (the 27B engine silently
ignored its encoder — frank-small owned images for exactly one day at this
endpoint).
Fetch: `scripts/download-frank-large-halogen.sh` into
`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 +
`--verify-only`, no token on the command line; resume goes against the
RESOLVED CDN URL because curl drops `Range` across HF's 302). The deployer
and serve script both refuse a missing tokenizer file or **any symlink** —
an HF cache snapshot symlinks each entry into `blobs/` and those links
dangle inside the container mount.
**Engine laws that shape the config** (all in `scripts/frank-large.env`
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
policy disables the MTP head for whole requests after low acceptance and
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
for it (accepted). `HALOGEN_CTX=200192` is the PER-SLOT hard ceiling (400
naming the limit, never truncation); `models.yml` `contextWindow: 200192`
must match, and compaction at 70 % = 140,134 + 57,344 budget = 197,478 fits.
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
law does NOT carry over; batching and speculation coexist (the card's
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
`HALOGEN_KV_POOL_POSITIONS=400384` (explicit 2 × ctx; ~28 KiB/position ⇒
~10.7 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
harness always sends 57,344); `HALOGEN_MAX_TOKENS_CAP=65536` +
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
longer greedy; `models.yml` extraBody carries the same values, consistent
not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm
100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual
`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported.
Thinking lands in **`reasoning_content`**; the v2 chat template rejects
`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold
stays mandatory. Thinking tokens COUNT against the budget (empty content +
`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI
`image_url` content parts only; ≥256 px probe images or the model answers
"video".
Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`.
Rollback: revert the swap commit and redeploy — the ROCmFPX payload
(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build
and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested.
The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download**
rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
weights ~62 + KV ~10.7 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
~2 ≈ 82.7 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~18 GiB
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
page cache beside the checkpoint — the card's quickstart arithmetic does
not bill it as resident, but READ the boot ledger and re-size if the
engine prints a clamp.
Fetch: `scripts/download-frank-large-flashnext.sh` into
`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
sha256 + `--verify-only`, no token on the command line; resume goes
against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
The serve gates mirror the deployer EXACTLY (byte-exact checkpoint +
n-gram + vision tower, flat tokenizer with all six files as regular
files) — the 2026-10-01 crash-loop lesson.
Runbook:
`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
Rollback: revert the swap commit and redeploy — the 27B payload
(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
@@ -997,7 +1013,7 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
| frank-small (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×131,072 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md` |
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
@@ -1007,6 +1023,7 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| halogen 0.1.4 Qwen3.8-27B p1w4d-d2 + DFlash2 (`frank-large`) | Dankclaw | 2026-10-05 → 10-06 (rollback target — payload + fetch script + image digest on disk; owns the prompt-cache pin and FA-tile lessons) | `frank-large-halogen27b-dflash2.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |