From 8500600d09d0ac393d65abe9af9ac87f3e9d6999 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 26 Sep 2026 19:45:28 -0700 Subject: [PATCH] frank-primary: swap Dankserver to llama.cpp Qwen3.8-Flash-Next Q8_0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour: Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256- verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the frank-large shape: no container, no generated unit). The deployed contract: - Placement law: layer split -ts 1,1 with -ngl 999; the routed experts and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via -ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming over PCIe loses at 3.7 t/s, computing in RAM does not care). per_layer_token_embd.weight=CPU is required by the shared MTP head's borrow path (a GPU-resident borrow crashes the draft load). mmap stays on; --cache-ram 16384. - Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head (mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the draftless ngram-mod matcher chained behind it — the only legal way to layer two spec implementations in one process. - Vision explicit: --mmproj names the BF16 projector (this build never autoloads; a server without it comes up healthy and text-only). - Reasoning: --reasoning-format deepseek → reasoning_content; the Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded by reasoningEffortMap in models.yml. - One stream, the model's full 262,144 context; greedy server default with Qwen-recommended sampling riding compat.extraBody. The deployer's deploy_llm_primary is rewritten for the root unit: it gates every artifact at its declared byte count (stat -c %s, never wc -c — which read the whole 377 GiB lucebox file for minutes on 2026-09-26), retires the lucebox user quadlet to dated home trash, waits out port 7770, relinks/enables/restarts the root unit and waits up to 1800 s for /health. The quadlet-era tree under dankserver/frank-primary/ is archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this box; git history everywhere else) and its planner mapping replaced by the scripts/ trio. Observability flips back to the llamacpp namespace: a frank-primary Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark published-port quirk that forces host.containers.internal for the two Dankclaw endpoints is same-host only), the LlmEndpointDown runbook text, and the dashboard description for three llama.cpp endpoints. Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md), RETIRED banners on the lucebox and radiance runbooks and their index entries, the AGENTS.md fleet section rewritten for the root-unit shape, and the homepage frank-medium card renamed to frank-primary with the Q8_0 Flash-Next spec (docs.html #models and the modality note updated: images on frank-primary and frank-small, audio on frank-small only). models.yml: the dankserver provider is "llamacpp (dankserver)" serving frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap law: 183,501 + 65,536 <= 262,144), input [text, image]. Suite: 346 green (frank_primary_unit.bats rewritten for the root unit — 16 tests pinning the placement law, the shared-head requirement, the explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and the deployer's byte-count gates; the quadlet-era planner and changed-tests mappings rewritten for the trio; the observability scrape test asserts three llamacpp jobs again). Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the 378 GiB payload stay on the host; the Qwen payload stays fetched, so a re-swap in either direction is a redeploy, not a refetch. --- AGENTS.md | 156 +++++++------ .../frank-primary/frank-primary-entrypoint.sh | 90 -------- .../frank-primary-healthcheck.py | 14 -- .../frank-primary/lucebox-gfx1201.patch | 40 ---- .../quadlet/frank-primary.container | 90 -------- scripts/deploy-plan.py | 19 +- scripts/deploy-remote.sh | 128 +++++++---- scripts/frank-primary.env | 78 +++++++ scripts/frank-primary.service | 21 ++ scripts/serve-frank-primary.sh | 134 +++++++++++ scripts/test-changed.sh | 16 +- ...k-primary-llamacpp-qwen38-flash-next-q8.md | 135 +++++++++++ .../LLM/frank-primary-lucebox-ds4.md | 18 +- serverAppConfigs-docs/LLM/index.md | 67 +++--- serverAppConfigs-docs/index.md | 18 +- .../provisioning/dashboards/llm-fleet.json | 10 +- shared/homepage/docs.html | 14 +- shared/homepage/services.json | 4 +- shared/prometheus/prometheus.yml | 37 +-- shared/prometheus/rules/llm-fleet.yml | 18 +- tests/changed_tests.bats | 23 +- tests/deployment_scoping.bats | 18 +- tests/frank_primary_unit.bats | 212 ++++++++++++------ tests/homepage_config.bats | 15 +- tests/observability_quadlet.bats | 25 ++- 25 files changed, 856 insertions(+), 544 deletions(-) delete mode 100755 dankserver/frank-primary/frank-primary-entrypoint.sh delete mode 100755 dankserver/frank-primary/frank-primary-healthcheck.py delete mode 100644 dankserver/frank-primary/lucebox-gfx1201.patch delete mode 100644 dankserver/frank-primary/quadlet/frank-primary.container create mode 100644 scripts/frank-primary.env create mode 100644 scripts/frank-primary.service create mode 100755 scripts/serve-frank-primary.sh create mode 100644 serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md diff --git a/AGENTS.md b/AGENTS.md index 84544d3..3e0a6a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,7 +9,7 @@ services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22 merge absorbed the former `vllmScripts` repo, history preserved). - **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty, - Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service. + Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 | +| Dankserver | `frank-primary` | unsloth llama.cpp b11030-mix prebuilt (ROCm gfx120X, bundled runtime, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q_8 (175.30 GiB, 6 split parts) with the shared Q_8 MTP head (depth 2) chained behind `ngram-mod`, BF16 vision (mmproj), layer split `-ts 1,1` across both R9700s with the routed experts + 51B-param n-gram table computed in host RAM (`-ot exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU` — the shared-head borrow path needs the last), 262144 ctx single stream (1 slot), mmap on, `--cache-ram 16384` | 7770 | | Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | @@ -61,16 +61,15 @@ All edits flow through git. Never edit files on a remote machine directly. ./scripts/deploy.sh dankclaw ./scripts/deploy.sh dankserver ./scripts/deploy.sh dankclaw --service searxng -./scripts/deploy.sh dankserver --service frank-medium +./scripts/deploy.sh dankserver --service frank-primary ./scripts/deploy.sh dankclaw --dry-run ``` The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by -their own files — `frank-large`/`frank-small` by their unit/env/serve trio in -`scripts/`, `frank-medium` by its Quadlet unit, entrypoint and healthcheck under -`dankserver/frank-medium/` — so +their own files — every endpoint (`frank-large`/`frank-small`/`frank-primary`) +by its unit/env/serve trio in `scripts/` — so unrelated container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo; the value never enters the repo. @@ -232,9 +231,8 @@ merely wedged kernel usually still answers ARP. `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - - Dankserver `frank-medium` (**rootless Quadlet since 2026-09-20** — the - system unit is retired, so `sudo systemctl` sees nothing): - `systemctl --user stop frank-medium && systemctl --user mask frank-medium` + - Dankserver `frank-primary` (root unit, llama.cpp on the 2x R9700): + `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary` 2. Bench. 3. Unmask and start; verify `curl http://192.168.0.:7770/health`. @@ -247,9 +245,8 @@ deletes the unit file from `/etc/systemd/system/`, not just the symlink. The harness reaches the fleet through `.omp/agent/models.yml` in the operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is where the harness runs; there is no copy on the servers. Each provider is one -HOST:PORT; each model `id` is the served alias: `frank-medium` → +HOST:PORT; each model `id` is the served alias: `frank-primary` → `192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` → -`192.168.0.143:7771`. Per-role routing lives beside it in `config.yml` under `modelRoles`. Resolution rules: the default model resolves as a `defaultProvider` + @@ -262,16 +259,19 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As -of 2026-09-20 both `frank-small` and `frank-medium` accept images -(`input: [text, image]`) — frank-medium's checkpoint serves vision through the -Gemma4 merge path with no separate mmproj, verified on held-out synthetic -images; `frank-large` remains text-only because its stock llama.cpp deployment loads no +of 2026-09-26 both `frank-small` and `frank-primary` accept images +(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly +(this build never autoloads projectors); `frank-large` remains text-only +because its stock llama.cpp deployment loads no vision projector (the mmproj is not loaded). Note also that the reasoning channel differs by runtime: llama.cpp endpoints emit -`reasoning_content`, vLLM emits `reasoning`. +`reasoning_content`, vLLM emits `reasoning` — academic since the +2026-09-26 swap, as every endpoint in the fleet is llama.cpp now. There is **no audio** -input anywhere in the fleet: that capability left with the FLM NPU endpoint on -2026-09-11. +input anywhere in the fleet except frank-small: that capability left with the +FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4 +projector's audio encoder (frank-small transcribes WAV verbatim even though +`audio` is omitted from its `input` list — OMP's schema rejects it). ### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE) @@ -341,73 +341,66 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-medium — radiance TP=2, DFlash2 drafter (Dankserver, ACTIVE) +### frank-primary — llama.cpp Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE) -vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet** -`dankserver/frank-medium/quadlet/frank-medium.container`) serving -`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4/quark W4A8, 19.37 -GiB) with the **z-lab DFlash2 external drafter** (`num_speculative_tokens: 7`, -live mean accept 4.24/7) in **one tensor-parallel process across both -R9700s** (`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk -8192, **unquantized bf16 KV** (`KV_DTYPE=auto`, cut 2026-09-21 from the -FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output -quality), prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool -auto-sized to 486,464 tokens (1.86× the full window). DFlash2 was A/B tested -against MTP self-speculation on this checkpoint and wins on decode (100.0 vs -85.9 t/s) — see the runbook. Tools via `qwen3_coder`, reasoning via `qwen3` -(the field is `reasoning`, not llama.cpp's `reasoning_content`), vision through -the Gemma4 merge path, chat template Qwen-Sharp v22.4.0 (froggeric, -repo-tracked at `dankserver/frank-medium/`, terse system prompt by default). +unsloth llama.cpp **b11030-mix prebuilt** (ROCm gfx120X, commit `6ba30d05b`, +bundled ROCm runtime at `/home/dank/GitPackages/unsloth-llama-b11030/`, **root +systemd unit** `scripts/frank-primary.service`, `User=dank`) serving +`unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8** — 175.30 GiB across six split +parts under `/home/dank/models/Qwen3.8-Flash-Next-GGUF/Q8_0/`, fetched by +`scripts/download-frank-primary.sh` (per-file sha256 + byte-count verified) — +on **both** Radeon AI PRO R9700s. Replaced the Lucebox DS4.1 quadlet (hours +old) on 2026-09-26, which had replaced radiance frank-medium the same day. -**The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads -`quantization_config.quant_method` from the checkpoint's `config.json` and -serves the plain FP8 path (no MXFP4 patches, `--quantization fp8`, -`ROCM_AITER_UNIFIED_ATTN`, the DFlash2 drafter) or the MXFP4 patch-and-serve -path (the DFlash2 external drafter). A checkpoint swap is therefore four quadlet -env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`) — the deploy handler reads -the serving shape from the quadlet and gates on the right assets. The MXFP4 -checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` + the z-lab -DFlash2-FP8 drafter) remains on disk as the rollback target. Full swap -process + toolchain: [[frank-medium-model-swap]]. +**Placement: compute the experts in RAM, never stream them.** Layer split +`-ts 1,1` with `-ngl 999`; the routed experts and the 51B-param n-gram +(PLE) table are overridden to CPU (`-ot +exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU`) and computed +from the mmap page cache — the x4-pair law (streaming over PCIe loses at +3.7 t/s; computing in RAM does not care). `per_layer_token_embd.weight=CPU` +is REQUIRED by the shared MTP head's borrow path (a GPU-resident borrow +crashes the draft load with `invalid vector subscript`). Dense weights, +attention, the BF16 projector and the MTP head sit on the cards (~29 GiB); +mmap must stay on (`--no-mmap` forces the table into anonymous memory → +silent cgroup kill) and the PLE tensor must never carry a GPU target +(23 → 13.4 t/s when tried). `--cache-ram 16384`; KV is the hybrid +DeltaNet/QSA pattern, a fraction of the dense-model cost. -**TP=2 runs because the board changed.** The predecessor board (TUF GAMING -B650-E) put card #2 behind chipset bridges advertising `AtomicOpsCap: Routing-` -and no collective could be dispatched there. The ProArt B850-Creator WiFi NEO -routes both cards to CPU GPP root ports (`00:01.1 → 03:00.0`, -`00:01.3 → 07:00.0`, Gen5 x8, every hop `Routing+`, both endpoints `ReqEn+`). -The collective in use is vLLM's **R4D one-shot allreduce**, which never touches -RCCL (TheRock RCCL has no gfx1201 device code; the Ubuntu RCCL 2.27 build does, -and that is what `scripts/build-llama-rdna4.sh GGML_HIP_RCCL=ON` links). +**Spec chain `draft-mtp,ngram-mod`, depth 2.** The shared Q_8 head +(`MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`, 2,786,568,256 B — unsloth's +recommended pairing) loads via `--model-draft` and borrows the target's +embedding tensors; `ngram-mod` chains behind it (draftless, so legal). +The dankclaw UD-IQ4_XS deployment of the same model family ran this head +at depth 2 with 0.83–0.92 acceptance. ngram-mod prints no startup banner — +identify it behaviourally (`draft_n` > 0 where text repeats). -**Measured, not assumed** (`scripts/bench-conversation.py`, 10 real prompts, -max_tokens 512, greedy, salted per arm, service stopped between arms): -radiance TP=2 95.8 t/s decode geomean / 0.21 s TTFT / 259.4 t/s burst-4, vs the -llama.cpp layer split's 45.7 / 0.72 / 71.7 and llama.cpp `--split-mode tensor`'s -30.5 / 0.71 / 54.3. Honest caveat: the arms also differ in quantization (MXFP4 -vs Q8_0), so part of the margin is the quant, not the parallelism. Re-measured -against production after deploy: 92.4 t/s, 0.22 s, 248.9. +**Vision explicit, reasoning deepseek-format.** `--mmproj` names the BF16 +projector (this build never autoloads one — a server without the flag comes +up healthy and text-only and answers image requests 500). `--jinja` with the +GGUF's embedded template; `--reasoning-format deepseek` fills +`reasoning_content`. The Qwen3.8 template gates `reasoning_effort` to +`{xhigh, medium, low}` — folded by `reasoningEffortMap` in `models.yml`. +One stream, full 262,144 ctx (`PARALLEL_SLOTS=1`); greedy server default, +Qwen-recommended sampling rides `compat.extraBody` per request. `--flash-attn +on`, batch 8192 / ubatch 512 (the measured ROCm optimum class), port 7770, +alias `frank-primary`. -Files: `dankserver/frank-medium/quadlet/frank-medium.container`, -`dankserver/frank-medium/radiance-entrypoint.sh` (quantization-aware), -`dankserver/frank-medium/frank-medium-healthcheck.py`. Swap toolchain: -`scripts/hf-manifest.py` (generate a ` ` manifest -from the live repo tree), `scripts/hf-fetch.sh` (resumable, byte + sha256 -verified, idempotent fetch), `scripts/swap-frank-medium.sh` (generate -manifest → download → detect quantization → edit the quadlet → deploy). The -llama.cpp layer-split trio (`scripts/frank-medium.service`, `.env`, -`serve-frank-medium.sh`) is archived in `scripts/.AGENTTRASH/ -20260920-llama-layersplit/`; `deploy-remote.sh`'s `retire_llama_medium_unit` -moved the installed unit symlink there rather than `systemctl disable`, which -would delete it. The deploy handler reads `SNAP`/`SPEC_METHOD` from the -quadlet, gates on the image tag + the checkpoint's `config.json` (drafter only -for `dflash`, `libr4d` only for `mxfp4`/`quark`), then waits for :7770 and -gates on `/health`. +Files: `scripts/frank-primary.service` (root unit, relinked to +`/etc/systemd/system/`), `scripts/serve-frank-primary.sh` (validates binary ++ all 6 parts + drafter + mmproj, then `exec`s llama-server), +`scripts/frank-primary.env` (tunables; the bind address is `BIND_HOST` — +never a bare `HOST`, HIP reads it). The deployer's `deploy_llm_primary` +gates every artifact at its declared byte count (`stat -c %s`, never `wc -c`), +retires the lucebox quadlet to `~/.AGENTTRASH/20260926-llamacpp-cutover/`, +waits out port 7770, relinks/enables/restarts the root unit and waits up to +1800 s for `/health` (cold start page-ins ~175 GiB). -Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` (serving -details); swap process + toolchain: `serverAppConfigs-docs/LLM/ -frank-medium-model-swap.md`. -Rollback: the MXFP4 checkpoint on disk (`SPEC_METHOD=dflash`, `SPEC=7`); -before that, [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20). +Runbook: `serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md`. +Rollback: the lucebox DS4.1 material on host (image +`localhost/lucebox-rocm:gfx1201` + the 377 GiB payload + the archived +quadlet files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`) — see +[[frank-primary-lucebox-ds4]]; the Qwen payload stays fetched, so a +forward re-swap is a redeploy. ## Key Lessons Learned Durable operational lessons from the fleet's history. Full detail in the @@ -756,8 +749,7 @@ Active: |---------|---------| | frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | -| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | -| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` | +| frank-primary (llama.cpp Qwen3.8-Flash-Next Q8_0 + shared MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -767,6 +759,8 @@ Retired (rollback targets and history): |---------|------|--------|---------| | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | +| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` | +| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-primary`) | Dankserver | 2026-09-26 (same-day swap) | `frank-primary-lucebox-ds4.md` | | halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` | | CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` | | unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` | diff --git a/dankserver/frank-primary/frank-primary-entrypoint.sh b/dankserver/frank-primary/frank-primary-entrypoint.sh deleted file mode 100755 index 1f1547e..0000000 --- a/dankserver/frank-primary/frank-primary-entrypoint.sh +++ /dev/null @@ -1,90 +0,0 @@ -#!/usr/bin/env bash -# frank-primary entrypoint: exec the native luce_server with the DeepSeek V4 -# launch shape for this host (2x R9700 gfx1201, 246 GiB RAM). -# -# The image's own entrypoint (server/scripts/entrypoint.sh) only forwards a -# fixed Qwen/DFlash flag set and cannot express the DS4 flags, so this unit -# bypasses it (PodmanArgs=--entrypoint) and execs the binary directly. The -# C++ server reads LUCE_* environment variables straight from the process -# environment, so no env plumbing is needed here. -# -# SOURCE BRANCH. The `deepseek41` architecture exists only on the -# `feat/ds41-engine` branch of Luce-Org/lucebox: main carries `deepseek4` -# alone and rejects this checkpoint with "unsupported model architecture -# 'deepseek41'" (server/src/common/backend_plan.cpp). The image is built from -# that branch — see serverAppConfigs-docs/LLM/frank-primary-lucebox-ds41.md. -# -# LAUNCH SHAPE (server/docs/DS41.md — the V4.1 page, not the V4 DS4.md one): -# --ds4-prefill exact THE ONLY prefill this placement can run. Any -# approximate mode (dense, sparse) makes -# DeepSeek4Backend::requires_monolithic_model() true, -# and a monolithic load asks ggml for the whole 187 GiB -# expert set as ONE device-0 buffer — measured -# 2026-09-26: "allocating 191932.25 MiB on device 0: -# cudaMalloc failed: out of memory", then -# "monolithic HIP load required for dense prefill". -# DS41.md's dense-prefill section measures that mode -# where the experts fit (in-process owners); on a -# 32 GiB target it cannot. exact feeds the prompt token -# by token, which is what the hybrid tier streams for. -# --chunk 1024 DS41.md's cold-expert chunk. The runtime caps the -# chunk to what fits after headroom regardless. -# --max-ctx 131072 the V4.1 default window. The MLA compressed caches -# are F16 by design and stay small; --cache-type-k/v -# are ignored by the DeepSeek4 backend, so they are -# not passed. -# --prefix-cache-slots 0 --prefill-cache-slots 0 -# the DS41.md bring-up recipe (the server default is -# 32 prefix slots). Lift them once capacity under load -# has been measured, not before. -# -# EXPERT PLACEMENT. 377 GiB of weights against 64 GiB of VRAM: only the dense -# weights, the compressed caches and a hot expert stack fit on hip:0, and the -# remaining routed experts are streamed out of the mmapped page cache/NVMe by -# the hybrid expert tier. LUCE_EXPERT_STREAM_DEVICE=1 moves the streamed- -# expert slot cache onto the second R9700 — it defaults to the target device, -# where it would compete with the hot stack. Both the owner stack and the -# slot cache then auto-size to their device's free memory minus the 1.5 GiB -# discrete headroom. -# -# No DSpark draft is published for DeepSeek-V4.1-Flash, so this is plain -# autoregressive decode (no --draft, no LUCE_DS4_SPEC). -set -euo pipefail - -MODEL="${FRANK_PRIMARY_MODEL:-/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf}" -BIN="${FRANK_PRIMARY_BIN:-/opt/lucebox-hub/server/build/luce_server}" - -# The second R9700 holds the streamed-expert slot cache and computes them. -export LUCE_EXPERT_STREAM_DEVICE="${LUCE_EXPERT_STREAM_DEVICE:-1}" -# The hot expert stack must be sized EXPLICITLY. Left alone the runtime -# budgets gpu_free - kv - warm - safety (22.35 GiB here), which leaves only -# its 0.50 GiB safety margin free while the fit check then demands the 1.50 -# GiB discrete headroom — its own auto-sizing can never pass on a 32 GiB -# target: "device 0 memory: 30.97 of 31.86 GiB used, 0.89 GiB free (headroom -# 1.50 GiB): does not fit". 18000 MB is the value DS41.md measured its R9700 -# rows at; it puts ~1500 experts (37/layer) resident and leaves device 0 -# ~5 GiB free, so the fit check passes with room for transient buffers. -export LUCE_EXPERT_BUDGET_MB="${LUCE_EXPERT_BUDGET_MB:-18000}" -# rocBLAS/BLAS dispatch and queue depth, as the fleet's other ROCm servers. -export ROCBLAS_USE_HIPBLASLT="${ROCBLAS_USE_HIPBLASLT:-0}" -export GPU_MAX_HW_QUEUES="${GPU_MAX_HW_QUEUES:-8}" - -if [[ ! -f "$MODEL" ]]; then - echo "frank-primary: model missing at $MODEL" >&2 - exit 1 -fi -if [[ ! -x "$BIN" ]]; then - echo "frank-primary: binary missing at $BIN (image build failed?)" >&2 - exit 1 -fi - -exec "$BIN" "$MODEL" \ - --host 0.0.0.0 \ - --port 7770 \ - --model-name frank-primary \ - --target-device hip:0 \ - --max-ctx 131072 \ - --chunk 1024 \ - --ds4-prefill exact \ - --prefix-cache-slots 0 \ - --prefill-cache-slots 0 diff --git a/dankserver/frank-primary/frank-primary-healthcheck.py b/dankserver/frank-primary/frank-primary-healthcheck.py deleted file mode 100755 index 1decc9f..0000000 --- a/dankserver/frank-primary/frank-primary-healthcheck.py +++ /dev/null @@ -1,14 +0,0 @@ -#!/usr/bin/env python3 -"""frank-primary health probe: 200 on /health only once the engine serves. - -File-based (not an inline HealthCmd) because the podman 5.x v1 quadlet -generator splits HealthCmd on spaces into an argv array. -""" -import sys -import urllib.request - -try: - with urllib.request.urlopen("http://127.0.0.1:7770/health", timeout=10) as r: - sys.exit(0 if r.status == 200 else 1) -except Exception: - sys.exit(1) diff --git a/dankserver/frank-primary/lucebox-gfx1201.patch b/dankserver/frank-primary/lucebox-gfx1201.patch deleted file mode 100644 index 10bba3a..0000000 --- a/dankserver/frank-primary/lucebox-gfx1201.patch +++ /dev/null @@ -1,40 +0,0 @@ -diff --git a/Dockerfile.rocm b/Dockerfile.rocm -index 4ea7a1d7..0e22355b 100644 ---- a/Dockerfile.rocm -+++ b/Dockerfile.rocm -@@ -59,18 +59,17 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ - curl \ - git \ - git-lfs \ -- hipblas-dev \ -- hipblaslt-dev \ -- hipcub-dev \ - libcurl4-openssl-dev \ - ninja-build \ - pkg-config \ - python3 \ -- rocblas-dev \ -- rocprim-dev \ -- rocwmma-dev \ - && rm -rf /var/lib/apt/lists/* - -+# Ubuntu jammy ships cmake 3.22, which predates the block command that -+# ROCm 7.14 hipblaslt-config.cmake requires (CMake >= 3.25). Install the -+# official 3.31 binary; /usr/local/bin precedes /usr/bin on PATH. -+RUN curl -fsSL https://github.com/Kitware/CMake/releases/download/v3.31.6/cmake-3.31.6-linux-x86_64.tar.gz | tar -xz -C /usr/local --strip-components=1 -+ - WORKDIR /src - - # COPY ordering mirrors the CUDA Dockerfile: C++ build inputs first so the -@@ -152,11 +151,8 @@ ENV PATH=/opt/rocm/bin:${PATH} - RUN apt-get update && apt-get install -y --no-install-recommends \ - ca-certificates \ - curl \ -- hipblas \ -- hipblaslt \ - libgomp1 \ - pciutils \ -- rocblas \ - && rm -rf /var/lib/apt/lists/* - - # uv (pinned) manages Python 3.12 + resolves the luce + pflash diff --git a/dankserver/frank-primary/quadlet/frank-primary.container b/dankserver/frank-primary/quadlet/frank-primary.container deleted file mode 100644 index 700a639..0000000 --- a/dankserver/frank-primary/quadlet/frank-primary.container +++ /dev/null @@ -1,90 +0,0 @@ -# Quadlet unit: frank-primary (rootless, boot-persistent). -# -# Install path on dankserver: ~/.config/containers/systemd/frank-primary.container -# Unit name: frank-primary.service (container name: frank-primary) -# -# The dankserver LLM endpoint (:7770, alias frank-primary): the Lucebox -# runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm: -# gfx1201 built from Dockerfile.rocm with LUCE_HIP_ARCHES=gfx1201 and -# ROCM_VERSION=7.14.1-full to match the host driver's ROCm 7.x line) -# serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 40-layer MoE, -# 384 routed experts + 1 shared per layer, top-6 routing, Engram tables at -# layers 1 and 14) on the two Radeon AI PRO R9700s (gfx1201). The model -# cannot be resident: dense weights, the compressed MLA caches and a hot -# expert stack sit on hip:0, the streamed-expert slot cache on hip:1, and -# every other routed expert is pulled out of the mmapped page cache (246 GiB -# RAM holds most of the 179 GiB expert set; the rest reads from NVMe). Lazy -# mmap page-in is the engine default — no flag. -# -# The image's entrypoint only forwards a fixed Qwen/DFlash flag set and -# cannot express the DS4 flags, so the unit bypasses it (--entrypoint) and -# runs the repo's frank-primary-entrypoint.sh, which execs the native -# luce_server with the DS4 launch shape (see that script's header). -# -# SHAPE. Host network, :7770, served model name frank-primary, OpenAI /v1 -# (Chat Completions with tool metadata, Responses, Anthropic Messages, -# Open WebUI metadata) — the fleet contract is unchanged from -# frank-medium. -# -# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack. Both R9700s are -# visible (indices 0 and 1; the Raphael iGPU enumerates as index 2 and stays -# out). They are NOT driven as a symmetric pair: the in-process expert- -# parallel path (LUCE_DS4_MOE_TP_INPROC) is qualified for gfx1151+gfx1201 -# pairs, not gfx1201+gfx1201. Instead hip:0 owns the dense weights and the -# hot expert stack while LUCE_EXPERT_STREAM_DEVICE=1 (entrypoint) gives hip:1 -# the streamed-expert slot cache, so each card carries one tier and neither -# waits on the other. -# -# MOUNTS. /models (ro — the GGUF is mmapped, nothing writes), and the -# entrypoint + healthcheck scripts straight out of this checkout -# (deploy.sh git-pulls the checkout before dispatching). -# -# IMAGE. localhost/-aliased tag: this host's podman quadlet path cannot -# resolve registry-prefixed refs from local storage at create time (same -# as frank-medium). Pull=never keeps boots off the registry. -# -# CUTOVER. This unit REPLACES frank-medium on :7770. The deploy handler -# (deploy_llm_primary in scripts/deploy-remote.sh) stops and masks the -# radiance unit before starting this one; the radiance unit is retired to -# the home trash after frank-primary is verified (trash policy, never rm). -# -# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the -# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident). -[Unit] -Description=frank-primary — Lucebox DeepSeek-V4.1-Flash-ROCMFP23 on R9700 (gfx1201), experts in RAM -After=network-online.target - -[Container] -ContainerName=frank-primary -Image=localhost/lucebox-rocm:gfx1201 -# Host network: luce_server binds 0.0.0.0:7770 directly; no PublishPort. -AddDevice=/dev/kfd -AddDevice=/dev/dri -GroupAdd=keep-groups -PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host --entrypoint /usr/local/bin/frank-primary-entrypoint.sh -# Both R9700s: hip:0 target, hip:1 the streamed-expert slot cache. Index 2, -# the Raphael iGPU, stays out. -Environment=HIP_VISIBLE_DEVICES=0,1 -Environment=ROCR_VISIBLE_DEVICES=0,1 -# The 377 GiB GGUF, mmapped; pages pull from disk/page cache on demand. -Volume=/home/dank/models:/models:ro -# Entrypoint + healthcheck from this checkout, mounted read-only. -Volume=/home/dank/serverAppConfigs/dankserver/frank-primary/frank-primary-entrypoint.sh:/usr/local/bin/frank-primary-entrypoint.sh:ro -Volume=/home/dank/serverAppConfigs/dankserver/frank-primary/frank-primary-healthcheck.py:/usr/local/bin/frank-primary-healthcheck.py:ro -# 200 only once the engine serves; first start page-ins the dense weights -# and builds the graphs, which takes a while. HealthCmd is split on spaces -# into an argv array by the podman 5.x v1 generator, hence the file probe. -HealthCmd=python3 /usr/local/bin/frank-primary-healthcheck.py -HealthInterval=60s -HealthTimeout=30s -HealthRetries=3 -HealthStartPeriod=1800s -Pull=never - -[Service] -TimeoutStartSec=1800 -Restart=on-failure -RestartSec=10 - -[Install] -WantedBy=default.target diff --git a/scripts/deploy-plan.py b/scripts/deploy-plan.py index 4c289f6..4e0afac 100755 --- a/scripts/deploy-plan.py +++ b/scripts/deploy-plan.py @@ -371,13 +371,11 @@ def services_for_path(path: str, target: str) -> set[str]: if name == "node-exporter": _add(services, "node-exporter") return services - if len(parts) >= 2 and parts[1] == "frank-primary": - # frank-primary is the Lucebox DeepSeek V4 Quadlet (2026-09-26 - # cutover, replacing frank-medium on :7770): its entrypoint and - # healthcheck live under this directory and the container mounts - # them straight out of this checkout, so any file here changes - # deployed state. - _add(services, "frank-primary") + # frank-primary's quadlet-era directory left the tree on 2026-09-26 + # (the Lucebox DS4.1 unit was retired for the llama.cpp root unit — + # the trio in scripts/ now maps it). Nothing lives under + # dankserver/frank-primary/ except dated trash, and trash plans + # nothing (the retire rule above the service blocks). return services if parts[0] == "frankbook-pro": @@ -421,6 +419,13 @@ def services_for_path(path: str, target: str) -> set[str]: # what the live frank-large endpoint renders, so it plans # frank-large. _add(services, "frank-large") + if name in {"frank-primary.env", "frank-primary.service", "serve-frank-primary.sh"}: + # llama.cpp frank-primary (2026-09-26, replacing the Lucebox + # DS4.1 quadlet) keeps its unit, env and serve files here the + # same way: a root unit on the unsloth b11030-mix prebuilt + # serving Qwen3.8-Flash-Next Q8_0. The download script fetches + # the payload, which is host-side state, so it selects nothing. + _add(services, "frank-primary") # Deployment/test/benchmark scripts and other repo tooling do not # restart a runtime service. return services diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 7e309ea..059431d 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1178,52 +1178,98 @@ deploy_llm_small() { wait_for_health frank-small http://127.0.0.1:7771/health 900 } -# deploy_llm_primary — frank-primary is the Lucebox DeepSeek V4 Quadlet -# (the 2026-09-26 cutover that replaces frank-medium on :7770). The model -# is a 377 GiB GGUF and the image is a local build, so the gates are: -# image present, model file present at its declared byte count (a partial -# download must never be served). +# deploy_llm_primary — frank-primary is unsloth llama.cpp (the b11030-mix +# prebuilt, ROCm gfx120X, bundled runtime) serving Qwen3.8-Flash-Next at +# Q8_0 on the two R9700s, as a ROOT unit — the same shape as frank-large on +# Dankclaw: a host binary, no container image, no generated unit. +# relink_unit points /etc/systemd/system/frank-primary.service at the repo +# file and boot persistence is that unit's [Install] section. +# +# The 2026-09-26 swap that replaced the Lucebox DS4.1 Quadlet on this port. +# The payload is six Q8_0 split parts plus the shared MTP head and the +# projector, so the gate walks every artifact at its declared byte count (a +# partial download must never be served — and stat -c %s, never wc -c, which +# read the whole 377 GiB lucebox file for minutes on 2026-09-26). +LLM_PRIMARY_RUNTIME=/home/dank/GitPackages/unsloth-llama-b11030/llama-server +LLM_PRIMARY_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF +LLM_PRIMARY_PART_SIZES=( + 10946624 + 682434912 + 54400261312 + 49446841216 + 49668930400 + 34015618784 +) +LLM_PRIMARY_DRAFT="$LLM_PRIMARY_ROOT/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" +LLM_PRIMARY_DRAFT_SIZE=2786568256 +LLM_PRIMARY_MMPROJ="$LLM_PRIMARY_ROOT/mmproj-BF16.gguf" +LLM_PRIMARY_MMPROJ_SIZE=907542944 + +# The lucebox quadlet held this port until the 2026-09-26 cutover. Its +# generated user unit is host state, not repo state: stop it and move the +# installed .container into dated home trash (the same retire pattern as +# retire_halogen_large_quadlet). The image and the 377 GiB payload stay on +# disk as the rollback target. +retire_lucebox_primary_quadlet() { + local trash="$HOME/.AGENTTRASH/20260926-llamacpp-cutover/quadlet" + systemctl --user stop frank-primary.service 2>/dev/null || true + if [[ -e "$quadlet_unit_dir/frank-primary.container" ]]; then + mkdir -p "$trash" + mv -- "$quadlet_unit_dir/frank-primary.container" "$trash/frank-primary.container" + echo "retired quadlet frank-primary.container -> $trash" + fi + systemctl --user daemon-reload +} + deploy_llm_primary() { - local missing=0 - local model="/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf" - local want_size=405349859328 - if ! podman image exists localhost/lucebox-rocm:gfx1201; then - echo "ERROR: lucebox image absent (localhost/lucebox-rocm:gfx1201)" >&2 - echo "frank-primary: podman build --build-arg LUCE_HIP_ARCHES=gfx1201 --build-arg ROCM_VERSION=7.14.1-full -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm . (see runbook)" >&2 + # Refuse rather than restart-loop. A unit whose runtime or payload is + # missing never becomes healthy, and a green deploy would leave the + # endpoint dark with nothing in the deploy log to say why. + local missing=0 i part want_size got_size + if [[ ! -x "$LLM_PRIMARY_RUNTIME" ]]; then + echo "ERROR: runtime absent at $LLM_PRIMARY_RUNTIME" >&2 + echo "frank-primary: fetch the unsloth b11030-mix gfx120X prebuilt (see the runbook)" >&2 missing=1 fi - # stat -c %s, never `wc -c < FILE`. wc falls back to reading the whole - # stream whenever its fstat shortcut does not apply, and this payload is - # 377 GiB: on 2026-09-26 a deploy sat in wc for minutes (30 s of CPU, - # NVMe-bound) before it could even reach the cutover. - got_size="$(stat -c %s "$model" 2>/dev/null || echo 0)" - if [[ "$got_size" != "$want_size" ]]; then - echo "ERROR: model absent or incomplete at $model (have $got_size, want $want_size)" >&2 - missing=1 - fi - [[ "$missing" -eq 0 ]] || return 1 - - # Cutover COMPLETED 2026-09-26: frank-primary took :7770 and the radiance - # unit is retired. The host-side unit, its mask and the repo files were all - # moved to trash by the parent session, so there is nothing left to stop or - # mask here — masking a name that no longer exists would only resurrect it - # in the unit search path. The port gate stays, because 7770 is a - # host-network bind: nothing new can be created until the holder released it. - # 7770 is a host-network bind: nothing new can be created until the old - # holder released it (the same gate as the 2026-09-16 cutover). - sleep 2 - wait_port_free 7770 || return 1 - - install_quadlet frank-primary dankserver/frank-primary/quadlet frank-primary - # No enable: generated units cannot be enabled on this host (v1 quadlet - # generator); boot persistence is the unit's [Install] WantedBy=default.target. - systemctl --user start frank-primary.service - # First start page-ins the dense weights and builds the graphs; - # HealthStartPeriod in the unit is 1800s to match. - wait_container_healthy frank-primary 1800 || { - podman logs --tail 40 frank-primary 2>&1 || true + for i in 1 2 3 4 5 6; do + part="$LLM_PRIMARY_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-0000$i-of-00006.gguf" + want_size="${LLM_PRIMARY_PART_SIZES[$((i-1))]}" + got_size="$(stat -c %s "$part" 2>/dev/null || echo 0)" + if [[ "$got_size" != "$want_size" ]]; then + echo "ERROR: part $i absent or incomplete at $part (have $got_size, want $want_size)" >&2 + missing=1 + fi + done + for part in "$LLM_PRIMARY_DRAFT:$LLM_PRIMARY_DRAFT_SIZE" \ + "$LLM_PRIMARY_MMPROJ:$LLM_PRIMARY_MMPROJ_SIZE"; do + local file="${part%%:*}" want="${part##*:}" + got_size="$(stat -c %s "$file" 2>/dev/null || echo 0)" + if [[ "$got_size" != "$want" ]]; then + echo "ERROR: artifact absent or incomplete at $file (have $got_size, want $want)" >&2 + missing=1 + fi + done + [[ "$missing" -eq 0 ]] || { + echo "frank-primary: run scripts/download-frank-primary.sh on this host" >&2 return 1 } + + retire_lucebox_primary_quadlet + # On a re-deploy the predecessor is THIS unit: stop it before waiting on + # the port, or the wait deadlocks on the server it just stopped serving. + run_privileged systemctl stop frank-primary.service 2>/dev/null || true + # Nothing starts until the port is released: the lucebox cutover on + # 2026-09-26 and the frank-large cutover on 2026-09-16 both failed + # exactly here when the next engine bound 7770 a second too early. + wait_port_free 7770 || return 1 + relink_unit frank-primary + run_privileged systemctl daemon-reload + run_privileged systemctl enable frank-primary.service + run_privileged systemctl restart frank-primary.service + # Cold start page-ins 175 GiB of expert pages and builds the graphs; + # 1800s parity with the lucebox HealthStartPeriod this port has always + # needed. + wait_for_health frank-primary http://127.0.0.1:7770/health 1800 } wait_for_health() { diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env new file mode 100644 index 0000000..a135875 --- /dev/null +++ b/scripts/frank-primary.env @@ -0,0 +1,78 @@ +# frank-primary — unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X) +# serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 on the two Radeon AI PRO +# R9700s (gfx1201), with the shared Q8_0 MTP drafter head, the BF16 vision +# projector, and the draft-mtp + ngram-mod spec chain. +# +# These are the tunables read by scripts/serve-frank-primary.sh. The flag +# list is owned by the serve script; this file only sets fleet values. +# +# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP +# runtime reads those bare (the 2026-08-13 TEMP incident). The bind address +# is BIND_HOST, not HOST — frank-large's env predates that lesson and gets +# away with it; do not copy the shape. + +# The payload (scripts/download-frank-primary.sh): the serve script refuses +# to start without all six Q8_0 parts, the MTP head and the projector. +MODEL_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF + +# The runtime: the unsloth b11030-mix prebuilt for gfx120X (the same +# b11030-mix build frank-large runs on gfx1151 — it bundles its own ROCm +# runtime, no host SDK gate). +RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030 + +# Fleet endpoint: the port and alias the lucebox DS4.1 quadlet held before +# the 2026-09-26 swap, so no client, probe or models.yml entry changes +# address. +BIND_HOST=0.0.0.0 +PORT=7770 + +# One conversation, the model's full native 262144 window. +CONTEXT_SIZE=262144 +PARALLEL_SLOTS=1 + +# Spec chain: draft-mtp (the shared Q8_0 head) at depth 2, with the +# draftless ngram-mod matcher chained behind it. Depth 2 is the measured +# optimum class for this pair at long contexts (the retired Dankclaw +# Flash-Next deployment ran exactly this head at depth 2 with 0.83-0.92 +# acceptance); each extra verified position re-batches the CPU expert reads, +# so depth buys less than bandwidth here. +SPEC_DRAFT_N_MAX=2 + +# Device placement. -ngl 999 puts every layer's non-overridden tensors on +# the GPUs; the big overrides then move the bandwidth-huge tensors to host +# RAM (computed there — the x4-pair measurement: streaming over PCIe loses, +# computing in RAM does not). OT_OVERRIDES is comma-separated +# = pairs passed as repeated --override-tensor +# flags: +# * exps=CPU — the routed experts. ~100 GiB at Q8, read once per token; +# the CPU backend computes them from page cache. +# * the n-gram embedding block (the 51B-param PLE table) — sparse row +# lookups, tiny PCIe traffic from CPU RAM; NEVER pin with a GPU target +# (the 23 -> 13.4 t/s regression). +# * per_layer_token_embd.weight=CPU — REQUIRED by the shared MTP head's +# borrow path: GPU-resident, it crashes the draft load with +# "invalid vector subscript" (llama.cpp PR #28243). +# The exact tensor pattern for the n-gram block was discovered at load time +# on 2026-09-26 (see the runbook); this list is the deployed set. +NGL=999 +OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU +TENSOR_SPLIT=1,1 + +# The drafter lives on the first R9700 (the card the load log shows holding +# the shared embedding borrow path). +SPEC_DRAFT_DEVICE=ROCm0 + +# Batch geometry: logical batch 8192 with ubatch 512 — the measured optimum +# class on this fleet's ROCm stack (a bigger logical batch is free; compute +# buffers reserve at max graph size). ubatch 512 balances prefill +# throughput against the asymmetric compute-buffer cost on the card that +# holds the output layers and the drafter. +BATCH_SIZE=8192 +UBATCH_SIZE=512 + +# Prompt-cache budget: 16 GiB. The KV itself is small (the hybrid +# DeltaNet/QSA pattern keeps a 262144 window a fraction of the full-ctx +# x all-layers number), so the prompt cache is the RAM-side spend that +# actually moves prefill on repeat traffic. Host RAM budget: ~175 GiB of +# expert pages + this cache + the server's own arenas inside 246 GiB. +CACHE_RAM=16384 diff --git a/scripts/frank-primary.service b/scripts/frank-primary.service new file mode 100644 index 0000000..4e4aa0a --- /dev/null +++ b/scripts/frank-primary.service @@ -0,0 +1,21 @@ +[Unit] +Description=unsloth llama.cpp (b11030-mix, ROCm gfx120X) Qwen3.8-Flash-Next Q8_0, shared-Q8_0 MTP + ngram-mod spec chain, BF16 vision, 262144 ctx / 1 slot, root systemd unit (dankserver, frank-primary, 2x Radeon AI PRO R9700) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=dank +Group=dank +# The model is mmap'd (175 GiB of expert pages + the 51B-param n-gram +# table live in page cache); an RLIMIT_MEMLOCK smaller than what the +# runtime pins would fail the load mid-start. Parity with the other LLM +# units on this fleet. +LimitMEMLOCK=infinity +Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env" +ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=multi-user.target diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh new file mode 100755 index 0000000..58ecfb2 --- /dev/null +++ b/scripts/serve-frank-primary.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# Serve unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 as frank-primary on +# Dankserver's two Radeon AI PRO R9700s (gfx1201), using the unsloth +# b11030-mix prebuilt (ROCm gfx120X, bundled runtime). +# +# Shape: one stream, the model's full 262144 context, all non-overridden +# tensors on the GPUs (--n-gpu-layers 999 across both cards), with the +# routed experts and the 51B-param n-gram embedding table computed in host +# RAM via --override-tensor (the x4-pair law: streaming over PCIe loses, +# computing in RAM does not care). mmap stays ON (the PLE table must stay +# page-cache-backed; --no-mmap forces it into anonymous memory and the +# cgroup kill is silent). +# +# Spec chain: draft-mtp (the shared Q8_0 head, loaded via -md) chained with +# the draftless ngram-mod matcher. The shared head borrows the target's +# embedding tensors, which is why per_layer_token_embd.weight must be +# CPU-resident (see frank-primary.env). depth 2. +# +# Vision: the BF16 projector, loaded explicitly via --mmproj (this build +# never autoloads projectors — the frank-medium 2026-09-17 trap: a server +# started without it comes up healthy and text-only, and answers image +# requests with 500). +# +# Template: the GGUF's embedded chat template via --jinja. The Qwen3.8 +# family template gates reasoning_effort to {xhigh, medium, low} — the +# harness folds its levels onto that set in models.yml +# (reasoningEffortMap). If the embedded template ever falls back to +# peg-native and bloats prompts (~73,600 tokens for a ~250-token request), +# switch to --chat-template-file scripts/qwen-sharp-v22.4.0.jinja the way +# frank-large does. +# +# Tunables come from frank-primary.env unless FRANK_PRIMARY_ENV_FILE +# overrides it; anything not set there falls through to the defaults below. +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}" +if [[ -f "$ENV_FILE" ]]; then + set -a + source "$ENV_FILE" + set +a +fi + +# --- defaults (used when the env file does not set them) ------------------ +: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}" +: "${MODEL_ROOT:=/home/dank/models/Qwen3.8-Flash-Next-GGUF}" +: "${BIND_HOST:=0.0.0.0}" +: "${PORT:=7770}" +: "${CONTEXT_SIZE:=262144}" +: "${PARALLEL_SLOTS:=1}" +: "${SPEC_DRAFT_N_MAX:=2}" +: "${NGL:=999}" +: "${OT_OVERRIDES:=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU}" +: "${TENSOR_SPLIT:=1,1}" +: "${SPEC_DRAFT_DEVICE:=ROCm0}" +: "${BATCH_SIZE:=8192}" +: "${UBATCH_SIZE:=512}" +: "${CACHE_RAM:=16384}" + +SERVER_BIN="$RUNTIME_DIR/llama-server" +MODEL_PARTS=( + "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf" + "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00002-of-00006.gguf" + "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf" + "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00004-of-00006.gguf" + "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00005-of-00006.gguf" + "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00006-of-00006.gguf" +) +DRAFTER_FILE="$MODEL_ROOT/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" +MMPROJ_FILE="$MODEL_ROOT/mmproj-BF16.gguf" + +if [[ ! -x "$SERVER_BIN" ]]; then + echo "runtime not found at $SERVER_BIN" >&2 + echo " fetch the unsloth b11030-mix gfx120X prebuilt (see the runbook)" >&2 + exit 1 +fi +# Every part is checked: part 1 is the -m target and parts 2-6 are followed +# by the split-file loader, so a missing tail part fails at LOAD time with +# a confusing partial-mmap error unless it is caught here. +for f in "${MODEL_PARTS[@]}" "$DRAFTER_FILE" "$MMPROJ_FILE"; do + if [[ ! -f "$f" ]]; then + echo "missing asset: $f" >&2 + echo " run scripts/download-frank-primary.sh on this host" >&2 + exit 1 + fi +done + +# --- env (the unsloth build's bundled ROCm runtime) ----------------------- +# Both R9700s (gfx1201, HIP devices 0 and 1); the host's Raphael iGPU is +# device 2 and must stay hidden from the enumeration. +export HIP_VISIBLE_DEVICES=0,1 +export LD_LIBRARY_PATH="$RUNTIME_DIR:${LD_LIBRARY_PATH:-}" + +# --- the serve command --------------------------------------------------- +ARGS=( + --host "$BIND_HOST" + --port "$PORT" + --model "${MODEL_PARTS[0]}" + --alias frank-primary + --ctx-size "$CONTEXT_SIZE" + --parallel "$PARALLEL_SLOTS" + --n-gpu-layers "$NGL" + --tensor-split "$TENSOR_SPLIT" + --jinja + --reasoning auto + --reasoning-format deepseek + --reasoning-budget -1 + --flash-attn on + --batch-size "$BATCH_SIZE" + --ubatch-size "$UBATCH_SIZE" + --cache-ram "$CACHE_RAM" + # Vision: explicit, never autoloaded by this build + --mmproj "$MMPROJ_FILE" + # Spec chain: the shared MTP head + the draftless ngram matcher + --spec-type draft-mtp,ngram-mod + --model-draft "$DRAFTER_FILE" + --spec-draft-n-max "$SPEC_DRAFT_N_MAX" + --spec-draft-device "$SPEC_DRAFT_DEVICE" + # Greedy server default: deterministic e2e, and the harness overrides + # sampling per-request (models.yml compat.extraBody carries Qwen's + # recommended temp 1.0 / top_p 0.95 / top_k 20). + --temp 0 + --top-p 1.0 + --top-k 0 + --min-p 0.0 +) + +# Comma-separated = pairs -> repeated --override-tensor. +IFS=',' read -ra _ots <<< "$OT_OVERRIDES" +for ot in "${_ots[@]}"; do + ARGS+=(--override-tensor "$ot") +done + +exec "$SERVER_BIN" "${ARGS[@]}" diff --git a/scripts/test-changed.sh b/scripts/test-changed.sh index 68752ac..198e456 100755 --- a/scripts/test-changed.sh +++ b/scripts/test-changed.sh @@ -187,16 +187,16 @@ for path in "${CHANGED[@]}"; do # as additions, with "no focused-test mapping". .AGENTTRASH/*|*/.AGENTTRASH/*) ;; - # frank-primary is the lucebox DS4.1 Quadlet (2026-09-26 cutover on - # :7770): its entrypoint and healthcheck live under - # dankserver/frank-primary/ and the container mounts them straight out - # of the checkout, so any file here changes deployed state. The - # radiance quadlet that held this endpoint before that date is archive - # material in dankserver/.AGENTTRASH/ and maps to nothing. - dankserver/frank-primary/*) + # frank-primary is llama.cpp (the unsloth b11030-mix prebuilt) on the + # two R9700s as a ROOT unit — the 2026-09-26 swap that replaced the + # Lucebox DS4.1 quadlet on :7770. Its env, unit and serve script all + # live in scripts/, which is why this case has to precede the generic + # scripts/*.service and scripts/* fallbacks below. + scripts/frank-primary.*|scripts/serve-frank-primary.sh) add_test tests/frank_primary_unit.bats + add_test tests/llm_units.bats add_test tests/deployment_scoping.bats - add_test tests/docs_completeness.bats + add_test tests/scripts_syntax.bats ;; # frank-large's chat template lives in scripts/ (moved there on # 2026-09-26 when its old home was retired): a template edit changes diff --git a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md new file mode 100644 index 0000000..eaab636 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md @@ -0,0 +1,135 @@ +--- +type: configuration +title: frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook +description: ACTIVE frank-primary runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment. +tags: + - frank-primary + - dankserver + - llamacpp + - unsloth + - qwen38-flash-next + - q8_0 + - mtp + - ngram-mod + - spec-decode + - vision + - rocm + - gfx1201 + - r9700 +timestamp: 2026-09-26 +--- + +# frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook + +The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**): +unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8** +(the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s, +with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with +**`ngram-mod`**, and **vision** through `mmproj-BF16.gguf`. It replaces the +Lucebox DS4.1 quadlet that held the port for hours earlier the same day +([[frank-primary-lucebox-ds4]]), which itself replaced +[[frank-medium-radiance-mxfp4]]. + +## Why this shape + +- **One runtime for the fleet.** All three endpoints are llama.cpp now — + one metric namespace (`llamacpp:*`), one reasoning contract + (`reasoning_content`), one tool contract (`--jinja`). The two-day + experiment with a single-vendor engine (lucebox) ended because Qwen + Flash-Next at Q8 on llama.cpp delivers better quality than the DS4.1 + fp2/fp3 stream, plus vision, plus the full 262k window, at comparable + speed. +- **The placement law, not a preference:** the routed experts and the + 51B-param n-gram (PLE) embedding table are **computed in host RAM** + (`-ot exps=CPU,ngram_embeddings=CPU`) and never streamed over PCIe. The + x4-pair measurement on the same model shape: streaming loses (3.7 t/s), + computing in RAM does not care (20+ t/s). The dense weights, attention, + projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of + expert pages live in mmap page cache — `--no-mmap` is forbidden (it + forces the table into anonymous memory; the cgroup kill is silent) and + pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried. +- **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root + ports with full PCIe atomics (the ProArt B850-Creator swap, + 2026-09-20), but this deployment uses llama.cpp's layer split — + `-ot` tensor overrides do not compose with tensor-parallel mode, and + `exps=CPU` is the override the placement law needs. The 27B-dense + equivalent on this host measured 1991 t/s prefill / 18.8 t/s decode in + this shape; Flash-Next Q8 with CPU experts lands in the same class + (measured numbers below). +- **Spec chain: `draft-mtp,ngram-mod`.** MTP is a property of the FILE: + the shared-Q8_0 head (`mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`, + 2,786,568,256 B) is loaded via `--model-draft` and borrows the target's + embedding tensors — which is why `per_layer_token_embd.weight` must be + CPU-resident in the override set (a GPU-resident borrow crashes the + draft load with `invalid vector subscript`). `ngram-mod` is draftless, + so it chains legally behind the drafter. Depth 2: each longer + verification round re-batches the CPU expert reads, so depth buys less + than bandwidth here. +- **Vision, explicitly.** This build never autoloads projectors — without + `--mmproj` the server comes up healthy and text-only and answers image + requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector + (907,542,944 B) is named in the serve script. + +## Deployment + +|Piece|Value| +|---|---| +|Unit|`scripts/frank-primary.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`| +|Serve script|`scripts/serve-frank-primary.sh` — validates binary + all 6 parts + drafter + mmproj before exec| +|Env tunables|`scripts/frank-primary.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)| +|Runtime|`/home/dank/GitPackages/unsloth-llama-b11030/llama-server` — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via `LD_LIBRARY_PATH`| +|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-primary.sh` (sha256 + byte-count verified)| +|Port / alias|7770 / `frank-primary` — unchanged from the lucebox and radiance eras; no client moves| +|Context|262,144 × 1 slot (`PARALLEL_SLOTS=1`, one stream, the model's full native window)| +|Sampling|Greedy server default (deterministic e2e); Qwen-recommended sampling rides `models.yml` `compat.extraBody` per request| +|Template|The GGUF's embedded template via `--jinja`; reasoning → `reasoning_content` (`--reasoning-format deepseek`)| +|Observability|`frank-primary` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`| + +### Cutover mechanics + +The deployer (`deploy_llm_primary` in `scripts/deploy-remote.sh`) gates on +every artifact at its declared byte count (`stat -c %s`, never `wc -c` — +which read the whole 377 GiB lucebox file for minutes on 2026-09-26), +retires the lucebox user quadlet to dated home trash +(`~/.AGENTTRASH/20260926-llamacpp-cutover/`), waits for port 7770 to be +truly free, relinks + enables + restarts the root unit, and waits up to +1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs). + +## Verification (2026-09-26 deploy) + +- `/v1/models` reports `frank-primary`. +- Chat completion returns `content` + `reasoning_content`. +- Tool call returns well-formed `tool_calls` (`--jinja`). +- Real image request answered from the projector (not a health check — + health cannot see vision). +- Spec chain live: `draft acceptance` lines in the unit journal; ngram-mod + has no startup banner (identify behaviourally: `draft_n` > 0 where text + repeats). +- `reasoning_effort: high` → 200 (the template's `{xhigh, medium, low}` + gate is folded by `reasoningEffortMap` in `models.yml`). +- VRAM/RAM headroom: `rocm-smi` per card + `free -g` after the first long + completion. + +_Measured numbers land here after the deploy battery._ + +## Rollback + +The lucebox DS4.1 material is fully retained on the host: image +`localhost/lucebox-rocm:gfx1201`, the 378 GiB payload, and the quadlet-era +repo files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see +[[frank-primary-lucebox-ds4]]). Restore = move the `.container` back, +uninstall the root unit (`systemctl disable --now frank-primary` — the +link, not the file; never `systemctl disable` a unit whose file must +survive), `daemon-reload`, start the user quadlet. The Qwen payload +(~179 GiB) stays fetched, so a forward re-swap is a redeploy, not a +refetch. + +## Fleet context + +- [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same + b11030-mix binary family on gfx1151. +- [[frank-small-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the + 3090; the only audio-capable endpoint. +- The harness contract lives in `/home/blank/.omp/agent/models.yml` + (`llamacpp (dankserver)` provider, `contextWindow 262144`, + `maxTokens 65536`, `input: [text, image]`). diff --git a/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md b/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md index 56446ce..f3cfd93 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md +++ b/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md @@ -1,7 +1,7 @@ --- type: configuration title: frank-primary Lucebox DeepSeek V4 Runbook -description: ACTIVE frank-primary runbook — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaces frank-medium (vllm-radiance TP=2) on :7770, cutover 2026-09-26. +description: RETIRED 2026-09-26 frank-primary runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-primary-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target. tags: - frank-primary - dankserver @@ -18,14 +18,22 @@ timestamp: 2026-09-26 # frank-primary Lucebox DeepSeek V4 Runbook -The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**): the +> [!warning] RETIRED 2026-09-26 (same day it went live) +> Replaced by [[frank-primary-llamacpp-qwen38-flash-next-q8]] — unsloth +> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime +> served for hours; the rollback material (image `localhost/lucebox-rocm: +> gfx1201` and the 378 GiB payload) is retained on the host, and the +> quadlet-era files live in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`. + +The dankserver LLM endpoint (:7770, alias **frank-primary**) as it stood on +2026-09-26: the [Lucebox](https://github.com/Luce-Org/lucebox) runtime serving `Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328 bytes, LFS oid `2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a -single Radeon AI PRO R9700 (gfx1201). It replaces -[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired on -2026-09-26. +single Radeon AI PRO R9700 (gfx1201). It replaced +[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired earlier the +same day. ## Why this shape diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 11ff2d8..3bd55ee 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,14 +1,14 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-09-21): -**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel -process across both Radeon AI PRO R9700 cards (just1moremodel -Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the -R4D one-shot allreduce), and TWO endpoints on Dankclaw — stock llama.cpp -(unsloth b11030, ROCm gfx1151; frank-large, unsloth Qwen3.8-27B UD-Q8_K_XL + -the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, port 7770) and mainline -llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on -the RTX 3090, port 7771). +Documentation for the LLM inference fleet on the two hosts (2026-09-26): +**llama.cpp on all three endpoints** — Dankserver runs unsloth llama.cpp +(the b11030-mix prebuilt, ROCm gfx120X) serving frank-primary +(unsloth Qwen3.8-Flash-Next Q8_0 + the shared Q8_0 MTP head + ngram-mod + +vision) across both Radeon AI PRO R9700s on :7770, and Dankclaw runs two +endpoints — stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, +unsloth Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the +gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS +Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -22,11 +22,10 @@ graph LR subgraph Dankserver[AMD Dankserver - Ubuntu 26.04] A[2x Radeon AI PRO R9700 gfx1201, 32 GiB each] - B2[vllm-radiance 0.9.3, TP=2 -frank-medium, port 7770] - C[Qwen3.8-27B MXFP4 + DFlash2 drafter -ONE process on both cards, 262144 ctx, up to 8 seqs] - A --> B2 --> C + B2[unsloth llama.cpp b11030-mix, ROCm gfx120X +frank-primary, port 7770] + C[Qwen3.8-Flash-Next Q8_0 + shared-Q8_0 MTP head +layer split on both cards, 262144 ctx / 1 slot, ngram-mod, vision] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] @@ -53,20 +52,34 @@ frank-small, port 7771] | Server | GPU | VRAM / Memory | OS | | --------------------------------------- | ----------------------------- | -------------------- | ------------ | -| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 60 GB RAM | Ubuntu 26.04 | +| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 246 GB RAM | Ubuntu 26.04 | | [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 | ## Active Deployment -- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: - vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving - just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8) with the - z-lab DFlash2-FP8 drafter as ONE tensor-parallel process (TP=2) behind - :7770, now on unquantized bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 to test - whether the endpoint's historical fp8 KV had been degrading W4A8 quality). - The FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 09-20→09-21 - shape) is the on-disk rollback. The three-arm battery that chose TP=2, the - auto-sized KV pool, and verified capabilities (tools, reasoning) +- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE + frank-primary runbook (2026-09-26, root systemd unit, port 7770): + unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime) + serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s — + routed experts and the 51B-param n-gram table computed in host RAM + (`-ot exps=CPU,ngram_embeddings=CPU`), the shared Q8_0 MTP head + (depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token + conversation, `reasoning_content`, the placement law, the cutover and + rollback +- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap, + rollback target; active for hours): the Lucebox DS4.1 engine + (localhost/lucebox-rocm:gfx1201) serving + Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts — + the fleet's only non-llama.cpp runtime detour +- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by + [[frank-primary-lucebox-ds4]], then + [[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3 + serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark + W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process + (TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut + 2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm + battery that chose TP=2, the auto-sized KV pool, and the verified + capability battery (tools, reasoning) - [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium: the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the @@ -181,9 +194,9 @@ frank-small, port 7771] on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model) - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22) -- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium) -- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the - Kairic Edge runtime since 2026-09-21, frank-small on the RTX 3090) +- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary) +- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on + the unsloth b11030 runtime since 2026-09-21, frank-small on the RTX 3090) - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as the retirement record — the IOMMU/SVA err=-19 root cause and where diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 7d4baa0..2d62143 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -124,12 +124,18 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-primary-lucebox-ds4]] — the ACTIVE frank-primary runbook: - the Lucebox runtime serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB - GGUF, 43-layer MoE, 256 routed experts top-6) on Dankserver's R9700 - with the routed experts computed on CPU from the mmapped page cache — - replaces frank-medium on :7770 (cutover 2026-09-26) -- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: +- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE + frank-primary runbook: unsloth llama.cpp (b11030-mix prebuilt, ROCm + gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3 + GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained + behind ngram-mod, BF16 vision, the experts and the 51B n-gram table + computed in host RAM, the placement law, the cutover, and the rollback +- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap; + rollback target): the Lucebox runtime serving + DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed + experts top-6) on Dankserver's R9700 with the routed experts computed + on CPU from the mmapped page cache — held :7770 for hours +- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's diff --git a/shared/grafana/provisioning/dashboards/llm-fleet.json b/shared/grafana/provisioning/dashboards/llm-fleet.json index d5319e3..7411d87 100644 --- a/shared/grafana/provisioning/dashboards/llm-fleet.json +++ b/shared/grafana/provisioning/dashboards/llm-fleet.json @@ -1,7 +1,7 @@ { "uid": "llm-fleet", "title": "LLM fleet \u2014 capacity and reuse", - "description": "The signals that explain the LLM endpoints under load. One metric namespace: llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU). frank-primary (Dankserver, the Lucebox DS4.1 engine since the 2026-09-26 cutover) exports no metrics and has no panels here. The endpoint picker selects between them. See serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split and the addressability rules.", + "description": "The signals that explain the LLM endpoints under load. One metric namespace: llamacpp:* across all three endpoints — frank-small (Dankclaw, RTX 3090), frank-large (Dankclaw, gfx1151 iGPU) and frank-primary (Dankserver, 2x R9700, llama.cpp again since the 2026-09-26 swap back from Lucebox DS4.1). The endpoint picker selects between them. See serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split and the addressability rules.", "tags": [ "llm", "capacity" @@ -96,7 +96,7 @@ }, { "expr": "job:llm_spec_accept_length:rate5m{job=~\"$job\"}", - "legendFormat": "{{job}} mean accepted length (vLLM)", + "legendFormat": "{{job}} mean accepted length", "refId": "B" }, { @@ -110,7 +110,7 @@ "id": 3, "type": "timeseries", "title": "Requests waiting (queue depth)", - "description": "llama.cpp: anything above zero means the engine is serialising (with --parallel 1 a second concurrent request always shows up here). vLLM: the scheduler's own waiting/running gauges against MAXSEQS=8.", + "description": "llama.cpp: anything above zero means the engine is serialising (with --parallel 1 a second concurrent request always shows up here). All three endpoints are llama.cpp since the 2026-09-26 frank-primary swap.", "gridPos": { "h": 8, "w": 12, @@ -138,7 +138,7 @@ "id": 4, "type": "timeseries", "title": "Decode and prefill throughput", - "description": "llama.cpp: llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. vLLM: the token counters, rate'd. The series carry the job label of their scrape, so the endpoint picker selects between the runtimes.", + "description": "llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. The series carry the job label of their scrape, so the endpoint picker selects between the endpoints.", "gridPos": { "h": 8, "w": 12, @@ -166,7 +166,7 @@ "id": 5, "type": "timeseries", "title": "Decode steps and batch size", - "description": "llama.cpp-only signal \u2014 vLLM does not publish per-step counters. Steps per second and the busy-slot count per step (the effective batch).", + "description": "Per-step counters: steps per second and the busy-slot count per step (the effective batch). With frank-primary's --parallel 1 the slot count stays at 0 or 1 and says nothing about the card; with frank-small's 4 slots it tracks the agent-fleet burst shape.", "gridPos": { "h": 8, "w": 12, diff --git a/shared/homepage/docs.html b/shared/homepage/docs.html index 1168713..455549d 100644 --- a/shared/homepage/docs.html +++ b/shared/homepage/docs.html @@ -36,8 +36,8 @@ Pick a model in the chat's model selector — the three models the fleet runs are described below. Open WebUI keeps your chat history; the raw model endpoints do not.

-

There is no audio input anywhere on the fleet. Text in, and — on - frank-small only — images too.

+

Images go to frank-primary or frank-small + (both see them). Audio (WAV) goes to frank-small only.

@@ -54,12 +54,12 @@
  • Not for: images. It cannot see them.
    • -

    frank-medium — the primary model

    +

    frank-primary — the primary model

      -
    • Model: Qwen3.8-27B-Uncensored (MXFP4/W4A8, 19.4 GiB).
      • -
    • Capacity: 262k-token context shared by up to eight conversations at once; sees images.
      • +
    • Model: Qwen3.8-Flash-Next (Q8_0, 175.3 GiB) with MTP + n-gram speculative drafting on both R9700s.
      • +
    • Capacity: 262k-token context, exactly one conversation at a time; sees images.
    • Inputs: text, image.
      • -
    • Best for: software engineering, system administration, and instruction tasks. Beats Claude Sonnet 5 per Artificial Analysis.
      • +
    • Best for: software engineering, system administration, and instruction tasks.

    frank-small — the fast utility model

    @@ -72,7 +72,7 @@

    Which one to pick? Small utility work, RAG, search, and screenshots: frank-small. Software engineering, sysadmin, and - instruction tasks: frank-medium. Cybersecurity and debugging: + instruction tasks: frank-primary. Cybersecurity and debugging: frank-large (with Frank's permission).

    diff --git a/shared/homepage/services.json b/shared/homepage/services.json index 10bae49..434a452 100644 --- a/shared/homepage/services.json +++ b/shared/homepage/services.json @@ -281,12 +281,12 @@ "showLink": false }, { - "name": "frank-medium", + "name": "frank-primary", "category": "user", "section": "AI", "details": { "role": "Primary model", - "spec": "Qwen3.8-27B-Uncensored (MXFP4/W4A8, 19.4 GiB) — 262k-token context across up to eight conversations; sees images. Beats Claude Sonnet 5 per Artificial Analysis.", + "spec": "Qwen3.8-Flash-Next (Q8_0, 175.3 GiB) — full 262k-token context, one conversation; sees images. Spec-drafted (MTP + n-gram) on both R9700s.", "inputs": ["text", "image"], "use": "Software engineering, system administration, and instruction tasks." }, diff --git a/shared/prometheus/prometheus.yml b/shared/prometheus/prometheus.yml index 94540ed..ed644ef 100644 --- a/shared/prometheus/prometheus.yml +++ b/shared/prometheus/prometheus.yml @@ -79,35 +79,46 @@ scrape_configs: # this container, not the guarded vhosts: these are machine clients and # Prometheus cannot solve an Anubis challenge. # - # One metric namespace: llamacpp:* — frank-large and frank-small, both - # llama.cpp with --metrics and both verified 2026-09-20 to export + # One metric namespace: llamacpp:* — frank-large, frank-small and + # frank-primary, all llama.cpp with /metrics (the family verified from + # /metrics on 2026-09-16 and 2026-09-20) and all exporting # requests_deferred (the queue depth, the only form llama.cpp shows the # capacity wall in), spec_decode_num_drafts_total, tokens_predicted_total, # and the prompt_tokens_cached_total / prompt_tokens_total reuse pair. # - # frank-large is scraped over host.containers.internal, not its own LAN address: - # this Prometheus is a bridge-networked container (Network=dankclaw_default), - # and on this netavark host a container can only reach the host's LAN IP at a - # PUBLISHED port — a host-process bind like llama-server's 0.0.0.0:7770 is - # refused at 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is + # frank-large and frank-small are scraped over host.containers.internal, + # not their own LAN addresses: this Prometheus is a bridge-networked + # container (Network=dankclaw_default), and on this netavark host a + # container can only reach the host's LAN IP at a PUBLISHED port — a + # host-process bind like llama-server's 0.0.0.0:7770 is refused at + # 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is # why frank-small was red here for days (up{job="frank-small"} == 0, # "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and # exporting: the target address was unreachable by construction, not the - # service down. Same host, same rule, so both local endpoints use the gateway. - # (2026-09-26) frank-medium is replaced by frank-primary (lucebox DS4.1), - # which exposes no metrics endpoint (/metrics -> 404), so there is no job - # for it here. The endpoint's liveness is covered by the homepage probe; - # the documented remedy is a blackbox exporter, which is not deployed. + # service down. Same host, same rule, so both local endpoints use the + # gateway. + # + # (2026-09-26) frank-primary is llama.cpp again (the unsloth b11030-mix + # prebuilt serving Qwen3.8-Flash-Next Q8_0) on Dankserver, a DIFFERENT + # machine: the netavark published-port quirk above is same-host only, so + # plain LAN routing reaches 192.168.0.235:7770 and the job returns. - job_name: "frank-small" metrics_path: "/metrics" static_configs: - targets: ["host.containers.internal:7771"] labels: service: frank-small - + - job_name: "frank-large" metrics_path: "/metrics" static_configs: - targets: ["host.containers.internal:7770"] labels: service: frank-large + + - job_name: "frank-primary" + metrics_path: "/metrics" + static_configs: + - targets: ["192.168.0.235:7770"] + labels: + service: frank-primary diff --git a/shared/prometheus/rules/llm-fleet.yml b/shared/prometheus/rules/llm-fleet.yml index 54a9932..d042842 100644 --- a/shared/prometheus/rules/llm-fleet.yml +++ b/shared/prometheus/rules/llm-fleet.yml @@ -1,11 +1,10 @@ # Alerting and recording rules for the LLM fleet. # -# One metric namespace: llamacpp:* — frank-small and frank-large, both -# llama.cpp with --metrics (the family verified from /metrics on 2026-09-16 -# and 2026-09-20). frank-primary (Dankserver, the Lucebox DS4.1 engine since -# the 2026-09-26 cutover) answers 404 on /metrics, so nothing here selects -# it; its liveness is covered by the homepage probe, and the documented -# remedy is a blackbox exporter (not deployed). +# One metric namespace: llamacpp:* — frank-small, frank-large and +# frank-primary, all llama.cpp with /metrics (the family verified from +# /metrics on 2026-09-16 and 2026-09-20). frank-primary is scraped across +# the LAN at 192.168.0.235:7770 (the netavark published-port quirk that +# forces the gateway for the Dankclaw endpoints is same-host only). # # Every expression is keyed to a series that EXISTS. Nothing matches on # absence: when an endpoint's runtime leaves the fleet, its selectors are @@ -111,7 +110,6 @@ groups: description: >- Prometheus cannot reach the LLM endpoint. frank-small and frank-large are root units on Dankclaw: `systemctl is-active - frank-small` / `frank-large`. Both scraped endpoints export the - llamacpp:* family. frank-primary (Dankserver) is not scraped — - its runtime exports no metrics; check the homepage probe or - `systemctl --user is-active frank-primary` on Dankserver. + frank-small` / `frank-large`. frank-primary is a root unit on + Dankserver: `systemctl is-active frank-primary`. All three + scraped endpoints export the llamacpp:* family. diff --git a/tests/changed_tests.bats b/tests/changed_tests.bats index e3a2291..731bb82 100644 --- a/tests/changed_tests.bats +++ b/tests/changed_tests.bats @@ -21,21 +21,28 @@ load helpers [[ "$output" != *"tests/homepage_pages.bats"* ]] } -@test "changed-test wrapper maps the frank-primary dir to its focused tests" { +@test "changed-test wrapper maps the frank-primary trio to its focused tests" { + # The root-unit shape since the 2026-09-26 swap: the trio lives in + # scripts/ and must select the unit seam before the generic + # scripts/*.service fallbacks can swallow it. run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths \ - dankserver/frank-primary/quadlet/frank-primary.container + scripts/frank-primary.env \ + scripts/frank-primary.service \ + scripts/serve-frank-primary.sh [ "$status" -eq 0 ] [[ "$output" == *"tests/frank_primary_unit.bats"* ]] + [[ "$output" == *"tests/llm_units.bats"* ]] [[ "$output" == *"tests/deployment_scoping.bats"* ]] - [[ "$output" == *"tests/docs_completeness.bats"* ]] [[ "$output" != *"tests/homepage_pages.bats"* ]] - # The retired frank-medium tree must no longer select any LLM seam: the - # mappings were removed with the service, and a stale case would keep - # firing tests for an archived runtime. Both archive shapes — the radiance - # Quadlet era and the llama.cpp layersplit trio before it — map to nothing. + # The retired trees must no longer select any LLM seam: the mappings + # were removed with the services, and a stale case would keep firing + # tests for archived runtimes. All three archive shapes — the radiance + # Quadlet era, the llama.cpp layersplit trio, and the lucebox quadlet + # dir — map to nothing. run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths \ dankserver/.AGENTTRASH/20260926-frank-medium-radiance/frank-medium.container \ - scripts/.AGENTTRASH/20260920-llama-layersplit/serve-frank-medium.sh.llamacpp-layersplit + scripts/.AGENTTRASH/20260920-llama-layersplit/serve-frank-medium.sh.llamacpp-layersplit \ + dankserver/.AGENTTRASH/20260926-llamacpp-cutover/frank-primary.container [ "$status" -eq 0 ] [[ "$output" != *"tests/frank_medium_unit.bats"* ]] [[ "$output" != *"tests/frank_primary_unit.bats"* ]] diff --git a/tests/deployment_scoping.bats b/tests/deployment_scoping.bats index 9458ac3..b8df710 100644 --- a/tests/deployment_scoping.bats +++ b/tests/deployment_scoping.bats @@ -63,13 +63,23 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" [ -z "$output" ] } -@test "deployment planner maps the frank-primary container dir to frank-primary only" { +@test "deployment planner maps the frank-primary trio to frank-primary only" { + # The root-unit shape since the 2026-09-26 swap: unit, env and serve + # script live in scripts/ and select exactly frank-primary. The + # download script fetches host-side state and must select nothing, and + # the lucebox quadlet-era tree in dated trash plans nothing either. run python3 "$PLANNER" --target dankserver --paths \ - dankserver/frank-primary/quadlet/frank-primary.container \ - dankserver/frank-primary/frank-primary-entrypoint.sh \ - dankserver/frank-primary/frank-primary-healthcheck.py + scripts/frank-primary.env \ + scripts/frank-primary.service \ + scripts/serve-frank-primary.sh [ "$status" -eq 0 ] [ "$output" = "frank-primary" ] + + run python3 "$PLANNER" --target dankserver --paths \ + scripts/download-frank-primary.sh \ + dankserver/.AGENTTRASH/20260926-llamacpp-cutover/frank-primary.container + [ "$status" -eq 0 ] + [ -z "$output" ] } @test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" { diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 3746526..a19a446 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -1,94 +1,168 @@ #!/usr/bin/env bats # Seam: frank-primary — the Dankserver LLM endpoint on :7770. # -# This file asserts the LUCEBOX shape (the 2026-09-26 cutover that replaced -# frank-medium): a rootless Quadlet running the lucebox DS4.1 engine on the -# two R9700s, with the model mmapped out of /home/dank/models and the -# entrypoint/healthcheck mounted straight out of this checkout. +# This file asserts the llama.cpp ROOT-UNIT shape (the 2026-09-26 swap that +# replaced the Lucebox DS4.1 Quadlet): the unsloth b11030-mix prebuilt on +# the two R9700s (gfx1201) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 +# with the shared Q8_0 MTP head, the BF16 projector, and the +# draft-mtp + ngram-mod chain. The seam pins what the flags alone do not +# say: the CPU-override placement law (experts and the 51B n-gram table +# are COMPUTED in host RAM, never streamed), mmap staying on, the +# shared-head borrow requirement (per_layer_token_embd on CPU), the +# explicit projector (never autoloaded by this build), HIP env hygiene, +# and the deployer's byte-count gates over a six-part payload. load helpers -UNIT="$REPO_ROOT/dankserver/frank-primary/quadlet/frank-primary.container" -EP="$REPO_ROOT/dankserver/frank-primary/frank-primary-entrypoint.sh" -HC="$REPO_ROOT/dankserver/frank-primary/frank-primary-healthcheck.py" +UNIT="$REPO_ROOT/scripts/frank-primary.service" +SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh" +ENV_FILE="$REPO_ROOT/scripts/frank-primary.env" +REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" -@test "the frank-primary quadlet file and its mounted scripts exist" { +@test "frank-primary unit, serve script and env exist in scripts/" { [ -f "$UNIT" ] - [ -x "$EP" ] - [ -f "$HC" ] - # The healthcheck is python; a syntax error would leave the container - # permanently unhealthy with the failure hidden inside HealthCmd. - python3 -m py_compile "$HC" + [ -x "$SERVE" ] + [ -f "$ENV_FILE" ] } -@test "frank-primary is boot-persistent and self-healing" { - # Rootless Podman does not auto-start bare containers and restart: - # unless-stopped does not survive shutdown's explicit stop. The unit must - # carry both the boot symlink and a restart policy (2026-08-26 incident: - # a compose-only Caddy never came back after a reboot). - grep -q '^WantedBy=default.target' "$UNIT" - grep -qE '^Restart=(always|on-failure)' "$UNIT" +@test "frank-primary is boot-persistent, self-healing and unprivileged" { + # Root unit parity with frank-large/frank-small: enable is the [Install] + # section, the restart policy survives a crash, and the mmap'd payload + # needs an unbounded memlock. + grep -q '^WantedBy=multi-user.target$' "$UNIT" + grep -q '^Restart=on-failure$' "$UNIT" + grep -q '^RestartSec=' "$UNIT" + grep -q '^User=dank$' "$UNIT" + grep -q '^LimitMEMLOCK=infinity$' "$UNIT" } -@test "frank-primary binds the host network and needs no published port" { - # luce_server binds 0.0.0.0:7770 directly, so the container takes the - # host netns. A PublishPort would be dead config. - grep -q -- '--network=host' "$UNIT" - ! grep -q '^PublishPort=' "$UNIT" +@test "frank-primary runs the repo's own serve script with the repo's env" { + grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh$' "$UNIT" + grep -q 'FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env' "$UNIT" } -@test "frank-primary reaches both R9700s and hides the Raphael iGPU" { - # /dev/kfd + /dev/dri via AddDevice; HIP_VISIBLE_DEVICES=0,1 keeps the - # host's Raphael iGPU (index 2) out of the enumeration so the engine's - # device picks cannot land on it. - grep -q '^AddDevice=/dev/kfd' "$UNIT" - grep -q '^AddDevice=/dev/dri' "$UNIT" - grep -q '^Environment=HIP_VISIBLE_DEVICES=0,1$' "$UNIT" +@test "frank-primary drafts with the shared MTP head chained behind ngram-mod" { + # The chain is draft-mtp (context-backed) + ngram-mod (draftless): the + # only legal way to layer two spec implementations in one process. The + # shared Q8_0 head borrows the target's embedding tensors — which is + # why per_layer_token_embd.weight must be CPU-resident (a GPU-resident + # borrow crashes the draft load with "invalid vector subscript"). + grep -q -- '--spec-type draft-mtp,ngram-mod' "$SERVE" + grep -q '^SPEC_DRAFT_N_MAX=2$' "$ENV_FILE" + grep -q -- '--spec-draft-n-max "$SPEC_DRAFT_N_MAX"' "$SERVE" + grep -q 'DRAFTER_FILE="$MODEL_ROOT/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf"' "$SERVE" + grep -q -- '--spec-draft-device "$SPEC_DRAFT_DEVICE"' "$SERVE" } -@test "frank-primary never names a bare HIP-reserved env var" { - # The HIP runtime reads bare TEMP for its JIT cache path (the 2026-08-13 - # incident: TEMP=1.0 sent the JIT cache to path "1.0" and segfaulted every - # load). HOST and GPU_MAX_HEAP_SIZE are the same class. The unit's - # Environment= lines must not define them; the entrypoint exports - # GPU_MAX_HW_QUEUES (explicitly not GPU_MAX_HEAP_SIZE). - ! grep -qE '^Environment=(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$UNIT" - ! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$EP" +@test "the bandwidth-huge tensors are computed in host RAM, not streamed" { + # The x4-pair law: streaming experts over PCIe loses (3.7 tok/s) while + # computing them in RAM does not care (20+ tok/s on the same model + # shape). exps=CPU plus the n-gram embedding table on CPU; the PLE + # table must NEVER carry a GPU target (the 23 -> 13.4 t/s regression). + grep -q '^OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU$' "$ENV_FILE" + grep -q -- '--override-tensor "$ot"' "$SERVE" + # ...and mmap must stay on: --no-mmap forces the 51B-param table into + # anonymous memory and the cgroup kill is silent. + ! grep -q -- '--no-mmap' "$SERVE" + ! grep -q '^MMPROJ_ON_GPU' "$ENV_FILE" } -@test "the unit mounts the entrypoint and healthcheck from the checkout" { - # deploy.sh git-pulls this checkout before dispatching, so a Volume - # reference is how an edit here reaches the container without an image - # rebuild. Absolute host paths, read-only. - grep -q 'Volume=/home/dank/serverAppConfigs/dankserver/frank-primary/frank-primary-entrypoint.sh:/usr/local/bin/frank-primary-entrypoint.sh:ro' "$UNIT" - grep -q 'Volume=/home/dank/serverAppConfigs/dankserver/frank-primary/frank-primary-healthcheck.py:/usr/local/bin/frank-primary-healthcheck.py:ro' "$UNIT" +@test "the projector is explicit, never autoloaded" { + # This build never autoloads mmproj from the model directory: without + # the flag the server comes up healthy and text-only, answering image + # requests with 500 (the frank-medium 2026-09-17 trap). + grep -q -- '--mmproj "$MMPROJ_FILE"' "$SERVE" + grep -q 'MMPROJ_FILE="$MODEL_ROOT/mmproj-BF16.gguf"' "$SERVE" } -@test "first-start patience is configured on both sides of the health check" { - # Cold start page-ins the dense weights and builds the graphs; the - # container must not be killed as unhealthy while that happens, and the - # deployer must wait the same order of magnitude. - grep -q '^HealthStartPeriod=1800s' "$UNIT" - grep -q '^TimeoutStartSec=1800$' "$UNIT" - grep -q 'wait_container_healthy frank-primary 1800' \ - "$REPO_ROOT/scripts/deploy-remote.sh" +@test "frank-primary is vision-serving but holds the single fleet port" { + grep -q '^PORT=7770$' "$ENV_FILE" + grep -q '^BIND_HOST=0.0.0.0$' "$ENV_FILE" + grep -q -- '--alias frank-primary' "$SERVE" + grep -q -- '--host "$BIND_HOST"' "$SERVE" + grep -q -- '--port "$PORT"' "$SERVE" } -@test "the deployer refuses to start without image and complete payload" { - # A 377 GiB partial download must never be served: the handler gates on - # the image being present and the model file existing at its exact - # declared byte count (stat -c %s, never wc -c — wc read the whole stream - # for minutes on 2026-09-26). - grep -q 'deploy_llm_primary() {' "$REPO_ROOT/scripts/deploy-remote.sh" - grep -q 'podman image exists localhost/lucebox-rocm:gfx1201' \ - "$REPO_ROOT/scripts/deploy-remote.sh" - grep -q 'want_size=405349859328' "$REPO_ROOT/scripts/deploy-remote.sh" - grep -q 'stat -c %s' "$REPO_ROOT/scripts/deploy-remote.sh" +@test "one stream, the model's full native window" { + local ctx parallel + ctx="$(sed -nE 's/^CONTEXT_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" + parallel="$(sed -nE 's/^PARALLEL_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")" + [ -n "$ctx" ] && [ -n "$parallel" ] + [ "$ctx" -eq 262144 ] + [ "$parallel" -eq 1 ] + grep -q -- '--ctx-size "$CONTEXT_SIZE"' "$SERVE" + grep -q -- '--parallel "$PARALLEL_SLOTS"' "$SERVE" } -@test "the deployer gates on the port actually being free" { - # 7770 is a host-network bind: the previous holder must release it before - # the new container can be created, or pasta's bind fails at create time - # and the failure hides behind Restart=. - grep -q 'wait_port_free 7770' "$REPO_ROOT/scripts/deploy-remote.sh" +@test "reasoning is extracted to reasoning_content and tools are legal" { + # The harness contract: llama.cpp endpoints emit reasoning_content + # (the deepseek format is the one that fills it), and tools require + # --jinja (the ik_llama 500 lesson). The Qwen3.8 template gates + # reasoning_effort to {xhigh, medium, low} — folded in models.yml. + grep -q -- '--jinja' "$SERVE" + grep -q -- '--reasoning auto' "$SERVE" + grep -q -- '--reasoning-format deepseek' "$SERVE" + grep -q -- '--reasoning-budget -1' "$SERVE" +} + +@test "the greedy server default is explicit" { + # Deterministic e2e probes; the harness overrides sampling per request. + grep -q -- '--temp 0' "$SERVE" + grep -q -- '--top-p 1.0' "$SERVE" + grep -q -- '--top-k 0' "$SERVE" + grep -q -- '--min-p 0.0' "$SERVE" +} + +@test "both R9700s are visible and the Raphael iGPU is not" { + # The host has a third (Raphael) GPU the server must never place + # tensors on; the bundled ROCm runtime is the only one linked. + grep -q 'export HIP_VISIBLE_DEVICES=0,1$' "$SERVE" + grep -q 'export LD_LIBRARY_PATH="$RUNTIME_DIR' "$SERVE" +} + +@test "no bare HIP-reserved env var is defined" { + # HIP reads bare TEMP for its JIT cache path (2026-08-13: TEMP=1.0 + # segfaulted every load), and HOST/GPU_MAX_HEAP_SIZE are the same + # class. The bind address is BIND_HOST, not HOST. + ! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE" + grep -q '^BIND_HOST=' "$ENV_FILE" + ! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE" +} + +@test "the serve script refuses to start on a half-finished download" { + # Six split parts plus the head and the projector: a missing tail part + # fails at LOAD time with a confusing partial-mmap error, so every + # artifact is checked before the server execs. + grep -q 'missing asset' "$SERVE" + grep -q 'download-frank-primary.sh' "$SERVE" +} + +@test "the prompt-cache cap is set below the RAM the host can spare" { + grep -q '^CACHE_RAM=16384$' "$ENV_FILE" + grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE" +} + +@test "the deployer retires the lucebox quadlet, gates every part and waits out the port" { + # The cutover order: the user-scope quadlet is stopped and its + # generated .container moved to dated home trash BEFORE the root unit + # binds 7770 (nothing starts until the port is released — the 2026-09-26 + # and 2026-09-16 cutover failures were both here). + grep -q 'retire_lucebox_primary_quadlet' "$REMOTE_DEPLOY" + grep -q 'systemctl --user stop frank-primary.service' "$REMOTE_DEPLOY" + grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY" + grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY" + grep -q 'systemctl restart frank-primary.service' "$REMOTE_DEPLOY" + grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY" + grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY" +} + +@test "the deployer gates the six-part payload at declared byte counts" { + # stat -c %s, never wc -c — wc read the whole 377 GiB lucebox file for + # minutes on 2026-09-26 before it could reach the cutover. A partial + # download must never be served. + grep -q 'LLM_PRIMARY_PART_SIZES' "$REMOTE_DEPLOY" + grep -q 'stat -c %s' "$REMOTE_DEPLOY" + grep -q '2786568256' "$REMOTE_DEPLOY" + grep -q '907542944' "$REMOTE_DEPLOY" + grep -q 'wait_for_health frank-primary http://127.0.0.1:7770/health 1800' "$REMOTE_DEPLOY" } diff --git a/tests/homepage_config.bats b/tests/homepage_config.bats index 93033a8..2c3baeb 100644 --- a/tests/homepage_config.bats +++ b/tests/homepage_config.bats @@ -61,17 +61,18 @@ PY @test "the LLM status cards track the fleet, not retired service names" { # This drifted twice: the card probing Dankserver's :7770 was still named - # frank-small long after frank-medium took that endpoint, and Dankclaw's - # second endpoint now exists again as a real service on its own port. The - # probe target is what matters; the name has to follow it. + # frank-small long after frank-medium took that endpoint, and the endpoint + # itself has since been frank-medium (radiance), frank-primary (lucebox) + # and frank-primary (llama.cpp). The probe target is what matters; the + # name has to follow it. python3 - "$REPO_ROOT/shared/homepage/services.json" <<'PY' import json, sys services = json.load(open(sys.argv[1])) by_probe = {s["probe"]: s["name"] for s in services} -medium = by_probe.get("192.168.0.235:7770") -assert medium == "frank-medium", f"Dankserver :7770 card is named {medium!r}" -# Dankclaw serves two endpoints on two ports: frank-large (halogen, 7770) and -# frank-small (llama.cpp on the RTX 3090, 7771). +primary = by_probe.get("192.168.0.235:7770") +assert primary == "frank-primary", f"Dankserver :7770 card is named {primary!r}" +# Dankclaw serves two endpoints on two ports: frank-large (unsloth llama.cpp +# on the iGPU, 7770) and frank-small (llama.cpp on the RTX 3090, 7771). assert by_probe.get("127.0.0.1:7770") == "frank-large", by_probe.get("127.0.0.1:7770") assert by_probe.get("127.0.0.1:7771") == "frank-small", by_probe.get("127.0.0.1:7771") PY diff --git a/tests/observability_quadlet.bats b/tests/observability_quadlet.bats index ea8d27a..bbff296 100755 --- a/tests/observability_quadlet.bats +++ b/tests/observability_quadlet.bats @@ -315,24 +315,29 @@ PAIRS @test "prometheus scrapes every LLM endpoint that exports metrics" { # The LLM endpoints are the services whose behaviour this fleet argues about - # most (KV pressure, prefix-cache reuse, queue depth). Only the two Dankclaw - # llama.cpp endpoints export metrics today: frank-primary (Dankserver, the - # Lucebox DS4.1 engine since the 2026-09-26 cutover) answers 404 on - # /metrics, so a job for it would be a permanently-down target — no job, - # and its LAN address must not sneak back in with one. - # The two scraped endpoints are HOST processes reached through + # most (KV pressure, prefix-cache reuse, queue depth). All three export + # the llamacpp:* family since 2026-09-26, when frank-primary (Dankserver) + # went back to llama.cpp — the unsloth b11030-mix prebuilt serving + # Qwen3.8-Flash-Next Q8_0 — after two days of the Lucebox DS4.1 engine, + # which exported nothing (404) and had no job. + # The two Dankclaw endpoints are HOST processes reached through # host.containers.internal: this Prometheus is a bridge-networked # container, and on this netavark host a container can only reach the # host's LAN IP at a PUBLISHED port — a host-process bind like # llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it # answers at 169.254.1.2:7770 (why frank-small stayed red by construction - # for days until the 2026-09-20 /targets audit). + # for days until the 2026-09-20 /targets audit). frank-primary is a + # DIFFERENT machine, so that quirk does not apply: plain LAN routing + # reaches 192.168.0.235:7770, and the job must say so (a gateway target + # would point at Dankclaw's own :7770 — frank-large — and double-scrape + # it under the wrong label). local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml" - ! grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-primary is scraped but exports no metrics"; return 1; } - grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; } grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; } - for job in frank-small frank-large; do + grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-primary not scraped"; return 1; } + grep -q '"host.containers.internal:7770,192.168.0.235:7770"' "$cfg" \ + && { echo "frank-primary folded into a Dankclaw gateway target"; return 1; } + for job in frank-small frank-large frank-primary; do grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; } done # None of them is reached through a guarded vhost: Prometheus cannot solve