From 2d89bf97cc0cca5fbbe6c8e214c1b6acefb7f832 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 6 Oct 2026 03:35:27 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20frank-large=20flash-server=20v2=20cutov?= =?UTF-8?q?er=20=E2=80=94=20runbook,=20index,=20AGENTS.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - New runbook: frank-large-flash-server-v2-swift15.md (engine, shape, memory model, engine laws, boot ledger, probes, rollback) - LLM/index.md: active entry + retired halogen 27B entry - Vault index: link the new runbook from the LLM Fleet section - AGENTS.md: fleet topology table, Model Config section, Runbook Index, retired table (halogen 27B → 10-06) --- AGENTS.md | 171 +++++++++-------- .../frank-large-flash-server-v2-swift15.md | 178 ++++++++++++++++++ serverAppConfigs-docs/LLM/index.md | 72 +++---- 3 files changed, 312 insertions(+), 109 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md diff --git a/AGENTS.md b/AGENTS.md index e1a078b..14af787 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,8 +12,9 @@ merge absorbed the former `vllmScripts` repo, history preserved). Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, - Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on - the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on + Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server + container on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the + RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside @@ -25,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *… -| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770| +| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 | | Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tightest fit in the endpoint's history — **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **63.3 t/s recipe decode** (MTP acceptance 49.8 %; 48.0 no-spec). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771| Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -234,7 +235,10 @@ Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. -(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.** +(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the +halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the +rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode +settings, and will invert the ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of @@ -296,9 +300,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As -of 2026-10-05 **two of the three endpoints accept images** (`input: [text, -image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine -ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while +of 2026-10-06 **all three endpoints accept images** (`input: [text, +image]`) — `frank-large` returned to vision with the flash-server v2 swap +(the v2 vision tower loads beside the checkpoint; the text-only halogen +27B window closed 2026-10-06) — while `frank-primary` takes them natively (radiance runs the checkpoint's 27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary are the engine core; `--mm-max-patches` available), @@ -318,82 +323,93 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (the MiMo tower is vision-only). -### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE) +### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE) -The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, -digest-pinned, published tag 0.1.4) as a podman container driven by -`scripts/serve-frank-large.sh` under a **root systemd unit** +The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, +digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container +driven by `scripts/serve-frank-large.sh` under a **root systemd unit** (`scripts/frank-large.service`, `User=dank`, `Restart=always`, -`ExecStop=podman -stop -t 120 frank-large-halogen`, the script runs `podman run` FOREGROUND — -the container's exit code is the service's exit so `Restart=always` stays -real, and the engine ledger lands on disk) on port 7770 from the Strix Halo -iGPU. Engine stdout goes to `/home/dank/logs/frank_large.log` — **the boot -ledger (KV pool, prompt cache, drafter state, any clamp) lives there, not in -the journal** (the 2026-10-05 `-d` + `podman wait` shape captured only the -container ID — fixed 2026-10-06). -Serving -`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B -≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations / -d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork -([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose), -chosen for the publisher's 31.7 t/s DFlash2 mean. +`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman +run` FOREGROUND — the container's exit code is the service's exit so +`Restart=always` stays real, and the engine ledger lands on disk) on port +7770 from the Strix Halo iGPU. Engine stdout goes to +`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV +pool, MTP head, prompt cache, any clamp) lives there, not in the journal**. +Cutover 2026-10-06 from the halogen 27B-dense era +([[frank-large-halogen27b-dflash2]], retired the same window). -**The format is the lock.** `.hgn` loads in nothing else — the runtime is -chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the -2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the -halogen-server repo restarted versioning at 0.1.x. +Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`: +`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1 +GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B +append-only original; the abliteration workflow appends and the repo's +tooling drops dead padding, live tensors byte-identical) + the **REQUIRED** +n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B, +peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn` +(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files, +fetched fresh from peonist). Cutover 2026-10-06 from the halogen +27B-dense shape, chosen by Frank for the ABLITERATED base. -**Two product regressions, both accepted by Frank before deploy:** the -endpoint is **text-only** (README honest limits: the vision encoder is -ignored — frank-small owns images now) and the checkpoint is a finetune of the -**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one -commit-revert away with the AGENT payload still on disk). +**The format is the lock.** The v2 `.hgn` loads ONLY in the +flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the +engine moved with the model. The MTP head is INSIDE the checkpoint +(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn` +sidecar is deliberately not fetched). -**Engine laws that shape the config** (all in `scripts/frank-large.env` with -evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the -drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD -ceiling (400 naming the limit, never truncation); `models.yml` -`contextWindow: 200700` must match or the harness discovers the wall as a 400. -`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled -pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 400 -naming the limit — README, never a 413). Thinking tokens COUNT against the -budget: exhausted mid-thought is `finish_reason: "length"` with empty -`content` and the reasoning in `reasoning_content`. -`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent -temperature is greedy, so the thinking recipe exists only in `models.yml` -`extraBody`. `response_format` is unsupported (no constrained decoding). -Thinking lands in **`reasoning_content`** (harness field unchanged); the same -Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the -`models.yml` fold stays mandatory. -`HALOGEN_CACHE_MB=16384` pins the prompt cache (2026-10-06): auto is OFF -on this box — the floor is 2 full-context entries (36.9 GB) and the boot -ledger had 35.7 GB free — so every deep request re-prefilled its whole -79-81k context (84.9-179.8 s per request for ~500-650 new tokens per -turn) until the pin. Probe-verified: a 17.3k cold prefill of 27.0 s -resumes in 1.7 s on the next turn (cache_n 16,384). Decode-at-depth is -structural: measured 5.5 t/s at 46k → 3.3-3.8 at 80k vs 30-50 shallow -(the KV pool reads at ~8 GB/s effective vs ~180-240 for the weights); -`docs/FLAGS.md` documents no decode-affecting lever and 0.1.4 is the -latest engine. The boot WARNING "smaller than ONE full-context entry -(18.4 GB at kMaxCtx=262144)" is a false positive for our slot: the -deepest possible entry is 200,700 × ~70 KB ≈ 14 GB, inside the pin, and -the 262k entry it fears cannot exist (over-slot is a 400). +**Product facts vs the 27B era, both regressions FIXED:** the base is +**abliterated** (the 27B was a censored-base finetune; refusals were an +accepted trade there) and **vision is ON** (the 27B engine silently +ignored its encoder — frank-small owned images for exactly one day at this +endpoint). -Fetch: `scripts/download-frank-large-halogen.sh` into -`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 + -`--verify-only`, no token on the command line; resume goes against the -RESOLVED CDN URL because curl drops `Range` across HF's 302). The deployer -and serve script both refuse a missing tokenizer file or **any symlink** — -an HF cache snapshot symlinks each entry into `blobs/` and those links -dangle inside the container mount. +**Engine laws that shape the config** (all in `scripts/frank-large.env` +with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive +policy disables the MTP head for whole requests after low acceptance and +HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance +for it (accepted). `HALOGEN_CTX=200192` is the PER-SLOT hard ceiling (400 +naming the limit, never truncation); `models.yml` `contextWindow: 200192` +must match, and compaction at 70 % = 140,134 + 57,344 budget = 197,478 fits. +`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter" +law does NOT carry over; batching and speculation coexist (the card's +quickstart ships 2 slots with the head), so 2 is concurrent capacity. +`HALOGEN_KV_POOL_POSITIONS=400384` (explicit 2 × ctx; ~28 KiB/position ⇒ +~10.7 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the +harness always sends 57,344); `HALOGEN_MAX_TOKENS_CAP=65536` + +`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table). +**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` = +1.0/0.95/20, the card's thinking recipe) — an absent temperature is no +longer greedy; `models.yml` extraBody carries the same values, consistent +not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm +100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual +`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported. +Thinking lands in **`reasoning_content`**; the v2 chat template rejects +`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold +stays mandatory. Thinking tokens COUNT against the budget (empty content + +`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI +`image_url` content parts only; ≥256 px probe images or the model answers +"video". -Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`. -Rollback: revert the swap commit and redeploy — the ROCmFPX payload -(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build -and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested. -The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download** -rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays). +**Memory ledger** (121.2 GiB unified; the referee is the boot ledger): +weights ~62 + KV ~10.7 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision +~2 ≈ 82.7 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~18 GiB +margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd +page cache beside the checkpoint — the card's quickstart arithmetic does +not bill it as resident, but READ the boot ledger and re-size if the +engine prints a clamp. + +Fetch: `scripts/download-frank-large-flashnext.sh` into +`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS +sha256 + `--verify-only`, no token on the command line; resume goes +against the RESOLVED CDN URL because curl drops `Range` across HF's 302). +The serve gates mirror the deployer EXACTLY (byte-exact checkpoint + +n-gram + vision tower, flat tokenizer with all six files as regular +files) — the 2026-10-01 crash-loop lesson. + +Runbook: +`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`. +Rollback: revert the swap commit and redeploy — the 27B payload +(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the +halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind +that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched. ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) @@ -997,7 +1013,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` | +| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` | | frank-small (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×131,072 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md` | | frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | @@ -1007,6 +1023,7 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| +| halogen 0.1.4 Qwen3.8-27B p1w4d-d2 + DFlash2 (`frank-large`) | Dankclaw | 2026-10-05 → 10-06 (rollback target — payload + fetch script + image digest on disk; owns the prompt-cache pin and FA-tile lessons) | `frank-large-halogen27b-dflash2.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` | | tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` | | FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` | diff --git a/serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md b/serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md new file mode 100644 index 0000000..16ea61b --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md @@ -0,0 +1,178 @@ +# frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU) + +**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio +(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`, +`scripts/frank-large.service`); the retired 27B payload stays on disk at +`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script +(`scripts/download-frank-large-halogen.sh`) stays in the repo. + +## Why this shape exists + +Frank's ask: replace the frank-large model with +`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`. +That payload is an **abliterated Swift1.5 finetune of peonist's +Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that +loads ONLY in the **halogen-flash-server** engine line, NOT in the halogen +0.1.4 container that served the 27B dense checkpoint. The model move is +therefore a **full engine migration**, and the accepted-product facts flip +twice in the fleet's favour: + +- the base is **abliterated** (the 27B halogen endpoint was a finetune of + the censored `Qwen/Qwen3.8-27B`; refusals were an accepted regression there), +- **vision is back** — the v2 vision tower loads beside the checkpoint and + the engine runs it (the 27B halogen engine silently ignored its encoder), +- **MTP speculative decoding returns** with the head inside the checkpoint + itself (the 27B shipped a DFlash2 drafter instead; comparable purpose, + different file plumbing), +- the card claims **53–57 t/s greedy decode, ~1,640 t/s prefill at 34k, ~72 % + fewer thinking tokens** than the censored 27B (905 mean), and **262,144 + native context**. Publisher-card numbers — verify against live traffic + before quoting further (see Verification). + +The engine is **closed-source**, ships as `ghcr.io/peonist-ai/ +halogen-flash-server:0.16.2` (the version the model card's quickstart pins), +digest-pinned: + +``` +ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a +``` + +(3.6 GB image; RepoDigest captured at pull 2026-10-06. Card-tested fallback +if a future pin misbehaves: 0.15.1.) + +## The payload — a triangle of files, all sha256-verified at fetch + +| File | Bytes | Source repo @ revision | sha256 (LFS) | +|---|---|---|---| +| `qwen38-flash-next-v2-swift15-abliterated.hgn` | 66,687,678,464 | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated @ main (`69a85db`) | `9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037` | +| `qwen38-flash-next-ngram.hgn` (REQUIRED) | 51,200,246,144 | peonist-ai/halogen-qwen3.8-flash-next @ `5cc17cea` | `3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe` | +| `qwen38-flash-next-vision.hgn` (enabled) | 897,916,416 | peonist-ai/halogen-qwen3.8-flash-next @ `e053f488` | `d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10` | +| `tokenizer/` (6 files, flat dir) | 22.4 MB total | peonist @ `e053f488` | `tokenizer.json` is LFS `0997f410…` (byte-identical to the 27B-era copy); the rest are plain git blobs, size-checked | + +Payload notes worth keeping: + +- The deployed checkpoint is a **compacted r2** of the card's append-only + original (89,775,868,032 B → 66,687,678,464 B): the abliteration workflow + appends and the repo's `tooling/` compact drops dead padding; live tensors + are byte-identical, the LFS sha256 is the contract, and the 24 GiB saving + is why it fits. +- **`HALOGEN_MTP_HEAD` stays UNSET**: the MTP head is INSIDE the v2 + checkpoint (the separate `qwen38-flash-next-mtp.hgn` sidecar is for a + different assembly and is deliberately not fetched). +- The **n-gram lookup table is REQUIRED**. The engine reads it beside the + checkpoint (documented fallback); the card's quickstart also names it + explicitly via `HALOGEN_NGRAM_TABLE`, and the serve script passes both — + the env var is belt-and-braces over the documented beside-the-checkpoint + default, and both mechanisms resolve to the same file. +- The tokenizer is fetched fresh from peonist @ `e053f488` into + `/home/dank/models/halogen-flashnext-v2/tokenizer/` — same flat-directory, + no-symlink law as every other payload (an HF cache snapshot's symlinks + dangle inside the container mount; both the serve gate and the deployer + refuse them). + +Fetch: `scripts/download-frank-large-flashnext.sh` (`--verify-only` to +re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -` +resume law throughout; no token — the repos are public). + +## Shape — the knobs that are not free + +`scripts/frank-large.env` carries the full rationale inline; the table: + +| Env | Value | Why | +|---|---|---| +| `HALOGEN_CTX` | 200192 | per-slot HARD ceiling (a longer prompt is a 400 naming the limit, never truncation). Harness compaction: 70 % = 140,134; 140,134 + 57,344 = 197,478 ≤ 200,192 | +| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity | +| `HALOGEN_KV_POOL_POSITIONS` | 400384 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ ~10.7 GiB KV | +| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) | +| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade | +| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness always sends 57,344); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) | +| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) | +| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) | +| `HALOGEN_MODEL_ID` / publish | frank-large / 7770:8731 | unchanged aliases; engine port 8730 never published | +| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose | + +**Memory ledger** (121.2 GiB unified, the referee is the boot ledger): +weights ~62 GiB resident + KV ~10.7 + arena ~8 (`3.8 + 8.4 × 8192/16384`) ++ vision ~2 ≈ **82.7 GiB engine-side**, + 20 host reserve = ~103 with +~18 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram +table rides as mmap'd page cache beside the checkpoint — the card's own +quickstart arithmetic does not bill it as resident, but WATCH the ledger +at boot and re-size if the engine prints a clamp. Prompt cache is ON by +default (`PROMPT_CACHE=2`): a follow-up turn at 100k ctx costs ~2 s +instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual +`HALOGEN_CACHE_MB` pin for is built in here. + +## Engine contract facts (FLAGS.md of the flash-server repo) + +- OpenAI chat **and** `/v1/responses` **and** `/v1/messages` **and** + `/v1/completions`; tool calls verified on the card; `response_format` + still unsupported (no constrained decoding). +- Thinking → **`reasoning_content`** (same field as every engine this + endpoint has run); the v2 chat template rejects `reasoning_effort` + outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays + mandatory. +- Vision: OpenAI `image_url` content parts only (the fleet's standard + contract; the legacy top-level `images` field is the frank-small trap). + ≥256 px probe images or the model answers "video". +- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a + truncation. `max_tokens` INCLUDES thinking. +- All flags read once at startup; every flag is BITWISE except + `HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`, + `HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`. Unset + `HALOGEN_CHECKPOINT` defaults to `/models/qwen38-flash-next-v2.hgn` + with the lookup table beside it. +- Prompt-lookup drafting (3,3 window) rides beside the MTP head by + default and is byte-identical to greedy verification where it fires. +- Watchdog: `HALOGEN_ENGINE_WATCHDOG_S=180` with defer-logic for memory + stalls — on a shared host the defaults are the safe choice; the + restart policy stays real through the root unit's `Restart=always`. + +Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it); `amdgpu.noretry=-1` +(auto) — for this IP auto resolves to retry-on, the same effective state +the 27B engine ran on. Only revisit if the ledger shows SVM complaints. + +## Mechanics + +- **Root systemd unit** (`scripts/frank-large.service`, `User=dank`, + `LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60 + frank-large-halogen`), driven by `scripts/serve-frank-large.sh` — the + fleet's `podman run`-direct pattern. The script runs the container + FOREGROUND so the boot ledger (weights resident, pool fit, MTP head, + prompt cache, any clamp) lands in `/home/dank/logs/frank_large.log`. +- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law): + image digest in podman storage, byte-exact checkpoint + n-gram table + + vision tower, flat tokenizer with all six files as regular files, + GPU nodes present. +- Engine stdout goes to the log, NOT the journal. The FIRST request after + a (re)start can sit silent ~45–60 s on this engine class (health + `engine_unresponsive`, 503s) and self-recovers — do not restart over it. + +## Verification checklist (run after any redeploy) + +1. Boot ledger greps in `/home/dank/logs/frank_large.log`: resident + weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line, + **no clamp named**. +2. `curl http://192.168.0.143:7770/health` → 200. +3. Text completion probe: reasoning lands in `reasoning_content`, + `timings` honest (generate ≥128 tokens; at 8 the inter_token number is + first-token noise). +4. Vision probe: a 512×512 solid-red PNG through an `image_url` part → + "Red" (≥256 px, one word expected). +5. Spec decode: `draft_n`/`draft_n_accepted` present in timings on + real-shaped traffic (SPEC_ADAPT=0 means the head is always engaged). + +## Client contract (harness) + +`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @ +`http://192.168.0.143:7770/v1`: `contextWindow: 200192`, +`maxTokens: 57344`, `input: [text, image]`, +`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort +fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides +frank-small (the `compactionModel` steering is unchanged). + +## Rollback + +Revert the swap commit and redeploy — the 27B trio is intact on disk +(payload at `~/models/halogen-qwen3.8-27b/`, fetch script, and the halogen +0.1.4 image digest still in podman storage). The older ROCmFPX rollback +door (`build-rocmfpx.sh` pin) is untouched. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 50afb99..51153f3 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -10,11 +10,12 @@ experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the 47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3, 2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005 and frank-primary-vllm-shapeG-20261003), -and Dankclaw runs two endpoints — **halogen container (frank-large, port 7770: -ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai -/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other -engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision -lives on frank-small)** and +and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2 +(frank-large, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned +digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of +Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine, +MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192 +ctx, vision ON)** and **llama.cpp CUDA (frank-small, port 7771: ajgazin Swift-1.5-Qwen3.8-27B Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's @@ -45,10 +46,10 @@ PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3, subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[RTX 3090 (PCIe x4 dock) + gfx1151 iGPU 114 GiB GTT] - E[halogen container gfx1151, digest-pinned image + E[flash-server container gfx1151, digest-pinned image frank-large, port 7770] - F[halogen-qwen3.8-27b .hgn p1w4d-d2 + DFlash2 drafter -1 x 200700 ctx, d2 KV, text-only] + F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower +2 x 200192 ctx, MTP always-on, image input] G[llama.cpp master CUDA sm_86 frank-small, port 7771] H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF @@ -71,30 +72,37 @@ frank-small, port 7771] ## Active Deployment -- [[frank-large-halogen27b-dflash2]] — **ACTIVE since 2026-10-05** +- [[frank-large-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06** (root systemd unit + podman, port 7770): the digest-pinned - `ghcr.io/peonist-ai/halogen` image serving `peonist-ai/halogen-qwen3.8-27b` - (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB) with the DFlash2 drafter on ONE slot - of 200,700 ctx (slots>1 disables speculation — engine law). Publisher card - 31.7 t/s mean across eight text workloads. Accepted tradeoffs: TEXT-ONLY - (the engine ignores the vision encoder; frank-small owns images) and the - CENSORED base model (the ROCmFPX AGENT file was Uncensored by construction). - No server-side sampler — the thinking recipe rides the wire in models.yml. - Rollback: revert the swap commit; the ROCmFPX payload + fork build + - `build-rocmfpx.sh` pin all remain on disk - Tuned 2026-10-06: `HALOGEN_CACHE_MB=16384` pins the prompt cache — - auto was OFF at the 36.9 GB two-entry floor, so every deep request - re-prefilled its 79-81k context (85-180 s each); a 17.3k cold prefill - of 27.0 s now resumes in 1.7 s on the warm turn (cache_n 16,384). The - serve script captures the engine ledger on disk (foreground `podman - run`; the `-d` era wrote only the container ID). Decode-at-depth is - structural: 3.3-3.8 t/s at 80k vs 30-50 shallow; no supported flag - touches it. - The FA tile lever from upstream issue #12 (`HALOGEN_FA_TQ=256`) was - A/B'd at 80k on 2026-10-06: decode ±0.3 % (rejected), prefill −9.5 % - on this topology — the default tile stands. The deep harness roles - (`slow`, `advisor`) rerouted to frank-primary; frank-large keeps - OpenWebUI and shallow callers. + `ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the + Quat3rnion ABLITERATED Swift1.5 finetune of peonist's + Qwen3.8-Flash-Next v2 checkpoint (`qwen38-flash-next-v2-swift15-abliterated.hgn`, + 66.7 GB compacted r2 of the card's 89.8 GB append-only original, + sha256-verified) with the REQUIRED n-gram lookup table (51.2 GB, + @peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488) beside + it. TWO slots of 200,192 ctx, MTP head ALWAYS ON (`HALOGEN_SPEC_ADAPT=0` + — the adaptive default turns the head off per request and halves + decode), server-side sampler at the card's 1.0/0.95/20, coupled + cap/timeout 65536/7200, prompt cache ON by default (the 27B engine's + manual `HALOGEN_CACHE_MB` pin era is over — warm 100k turn ≈ 2 s vs + ≈ 88 s cold). Vision ON (image_url parts only). Card: 53-57 t/s + greedy, ~1,640 t/s prefill at 34k, ~72 % fewer thinking tokens. + Rollback: revert the swap commit; the 27B payload + fetch script stay + on disk +- [[frank-large-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06 + (rollback target; payload + fetch script + image on disk)**: the + digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving + `peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB) + — DFlash2 drafter, ONE 200,700 slot, text-only (the engine ignored the + vision encoder), censored base. Owns the measured lessons: the prompt + cache pin (`HALOGEN_CACHE_MB=16384` — auto was OFF at the 36.9 GB + two-entry floor; a 17.3k cold prefill of 27.0 s resumed in 1.7 s warm), + the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode + ±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The + foreground-`podman run` ledger capture and the deep-role rerouting + (`slow`/`advisor` → frank-primary) carried forward. Superseded by the + MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the + engine moved with the model - [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since 2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since 2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master @@ -400,7 +408,7 @@ frank-small, port 7771] (deployed and reverted 2026-08-22) - [[Dankserver System Specs]] — the 2x R9700 box (frank-primary) - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on - the unsloth b11030 runtime since 2026-09-21, frank-small on the RTX 3090) + the flash-server v2 runtime since 2026-10-06, frank-small on the RTX 3090) - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as the retirement record — the IOMMU/SVA err=-19 root cause and where