dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: swap Dankserver to llama.cpp Qwen3.8-Flash-Next Q8_0

Browse Source 
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).

The deployed contract:

- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
  and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
  -ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
  over PCIe loses at 3.7 t/s, computing in RAM does not care).
  per_layer_token_embd.weight=CPU is required by the shared MTP head's
  borrow path (a GPU-resident borrow crashes the draft load).
  mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
  (mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
  draftless ngram-mod matcher chained behind it — the only legal way to
  layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
  autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
  Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
  by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
  with Qwen-recommended sampling riding compat.extraBody.

The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.

Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.

Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).

models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].

Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).

Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
This commit is contained in:
Frank committed 2026-09-26 19:45:28 -07:00
1 parent d89813363e
commit 8500600d09
25 files changed
+856 -544
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+75 -81
View File
@@ -9,7 +9,7 @@ services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22
merge absorbed the former `vllmScripts` repo, history preserved).
- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 |
| Dankserver | `frank-primary` | unsloth llama.cpp b11030-mix prebuilt (ROCm gfx120X, bundled runtime, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q_8 (175.30 GiB, 6 split parts) with the shared Q_8 MTP head (depth 2) chained behind `ngram-mod`, BF16 vision (mmproj), layer split `-ts 1,1` across both R9700s with the routed experts + 51B-param n-gram table computed in host RAM (`-ot exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU` — the shared-head borrow path needs the last), 262144 ctx single stream (1 slot), mmap on, `--cache-ram 16384` | 7770 |
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
@@ -61,16 +61,15 @@ All edits flow through git. Never edit files on a remote machine directly.
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-medium
./scripts/deploy.sh dankserver --service frank-primary
./scripts/deploy.sh dankclaw --dry-run
```
The planner maps changed paths to services; the remote runner restarts only
the selected service and records its revision outside the checkout.
Documentation-only changes are a no-op. LLM services are selected only by
their own files — `frank-large`/`frank-small` by their unit/env/serve trio in
`scripts/`, `frank-medium` by its Quadlet unit, entrypoint and healthcheck under
`dankserver/frank-medium/` — so
their own files — every endpoint (`frank-large`/`frank-small`/`frank-primary`)
by its unit/env/serve trio in `scripts/` — so
unrelated
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
runs on a host without passwordless sudo; the value never enters the repo.
@@ -232,9 +231,8 @@ merely wedged kernel usually still answers ARP.
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
- Dankserver `frank-medium` (**rootless Quadlet since 2026-09-20** — the
system unit is retired, so `sudo systemctl` sees nothing):
`systemctl --user stop frank-medium && systemctl --user mask frank-medium`
- Dankserver `frank-primary` (root unit, llama.cpp on the 2x R9700):
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
2. Bench.
3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.
@@ -247,9 +245,8 @@ deletes the unit file from `/etc/systemd/system/`, not just the symlink.
The harness reaches the fleet through `.omp/agent/models.yml` in the
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
where the harness runs; there is no copy on the servers. Each provider is one
HOST:PORT; each model `id` is the served alias: `frank-medium` →
HOST:PORT; each model `id` is the served alias: `frank-primary` →
`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` →
`192.168.0.143:7771`. Per-role routing lives beside it in `config.yml` under
`modelRoles`.
Resolution rules: the default model resolves as a `defaultProvider` +
@@ -262,16 +259,19 @@ When renaming a service, retiring an endpoint, or moving a model: update the
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a provider the file does not define fails at call time. As
of 2026-09-20 both `frank-small` and `frank-medium` accept images
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
Gemma4 merge path with no separate mmproj, verified on held-out synthetic
images; `frank-large` remains text-only because its stock llama.cpp deployment loads no
of 2026-09-26 both `frank-small` and `frank-primary` accept images
(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
(this build never autoloads projectors); `frank-large` remains text-only
because its stock llama.cpp deployment loads no
vision projector (the mmproj is not loaded). Note also that
the reasoning channel differs by runtime: llama.cpp endpoints emit
`reasoning_content`, vLLM emits `reasoning`.
`reasoning_content`, vLLM emits `reasoning` — academic since the
2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
There is **no audio**
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
2026-09-11.
input anywhere in the fleet except frank-small: that capability left with the
FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
projector's audio encoder (frank-small transcribes WAV verbatim even though
`audio` is omitted from its `input` list — OMP's schema rejects it).
### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
@@ -341,73 +341,66 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]].
### frank-medium — radiance TP=2, DFlash2 drafter (Dankserver, ACTIVE)
### frank-primary — llama.cpp Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet**
`dankserver/frank-medium/quadlet/frank-medium.container`) serving
`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4/quark W4A8, 19.37
GiB) with the **z-lab DFlash2 external drafter** (`num_speculative_tokens: 7`,
live mean accept 4.24/7) in **one tensor-parallel process across both
R9700s** (`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk
8192, **unquantized bf16 KV** (`KV_DTYPE=auto`, cut 2026-09-21 from the
FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output
quality), prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool
auto-sized to 486,464 tokens (1.86× the full window). DFlash2 was A/B tested
against MTP self-speculation on this checkpoint and wins on decode (100.0 vs
85.9 t/s) — see the runbook. Tools via `qwen3_coder`, reasoning via `qwen3`
(the field is `reasoning`, not llama.cpp's `reasoning_content`), vision through
the Gemma4 merge path, chat template Qwen-Sharp v22.4.0 (froggeric,
repo-tracked at `dankserver/frank-medium/`, terse system prompt by default).
unsloth llama.cpp **b11030-mix prebuilt** (ROCm gfx120X, commit `6ba30d05b`,
bundled ROCm runtime at `/home/dank/GitPackages/unsloth-llama-b11030/`, **root
systemd unit** `scripts/frank-primary.service`, `User=dank`) serving
`unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8** — 175.30 GiB across six split
parts under `/home/dank/models/Qwen3.8-Flash-Next-GGUF/Q8_0/`, fetched by
`scripts/download-frank-primary.sh` (per-file sha256 + byte-count verified) —
on **both** Radeon AI PRO R9700s. Replaced the Lucebox DS4.1 quadlet (hours
old) on 2026-09-26, which had replaced radiance frank-medium the same day.
**The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads
`quantization_config.quant_method` from the checkpoint's `config.json` and
serves the plain FP8 path (no MXFP4 patches, `--quantization fp8`,
`ROCM_AITER_UNIFIED_ATTN`, the DFlash2 drafter) or the MXFP4 patch-and-serve
path (the DFlash2 external drafter). A checkpoint swap is therefore four quadlet
env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`) — the deploy handler reads
the serving shape from the quadlet and gates on the right assets. The MXFP4
checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` + the z-lab
DFlash2-FP8 drafter) remains on disk as the rollback target. Full swap
process + toolchain: [[frank-medium-model-swap]].
**Placement: compute the experts in RAM, never stream them.** Layer split
`-ts 1,1` with `-ngl 999`; the routed experts and the 51B-param n-gram
(PLE) table are overridden to CPU (`-ot
exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU`) and computed
from the mmap page cache — the x4-pair law (streaming over PCIe loses at
3.7 t/s; computing in RAM does not care). `per_layer_token_embd.weight=CPU`
is REQUIRED by the shared MTP head's borrow path (a GPU-resident borrow
crashes the draft load with `invalid vector subscript`). Dense weights,
attention, the BF16 projector and the MTP head sit on the cards (~29 GiB);
mmap must stay on (`--no-mmap` forces the table into anonymous memory →
silent cgroup kill) and the PLE tensor must never carry a GPU target
(23 → 13.4 t/s when tried). `--cache-ram 16384`; KV is the hybrid
DeltaNet/QSA pattern, a fraction of the dense-model cost.
**TP=2 runs because the board changed.** The predecessor board (TUF GAMING
B650-E) put card #2 behind chipset bridges advertising `AtomicOpsCap: Routing-`
and no collective could be dispatched there. The ProArt B850-Creator WiFi NEO
routes both cards to CPU GPP root ports (`00:01.1 → 03:00.0`,
`00:01.3 → 07:00.0`, Gen5 x8, every hop `Routing+`, both endpoints `ReqEn+`).
The collective in use is vLLM's **R4D one-shot allreduce**, which never touches
RCCL (TheRock RCCL has no gfx1201 device code; the Ubuntu RCCL 2.27 build does,
and that is what `scripts/build-llama-rdna4.sh GGML_HIP_RCCL=ON` links).
**Spec chain `draft-mtp,ngram-mod`, depth 2.** The shared Q_8 head
(`MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`, 2,786,568,256 B — unsloth's
recommended pairing) loads via `--model-draft` and borrows the target's
embedding tensors; `ngram-mod` chains behind it (draftless, so legal).
The dankclaw UD-IQ4_XS deployment of the same model family ran this head
at depth 2 with 0.83–0.92 acceptance. ngram-mod prints no startup banner —
identify it behaviourally (`draft_n` > 0 where text repeats).
**Measured, not assumed** (`scripts/bench-conversation.py`, 10 real prompts,
max_tokens 512, greedy, salted per arm, service stopped between arms):
radiance TP=2 95.8 t/s decode geomean / 0.21 s TTFT / 259.4 t/s burst-4, vs the
llama.cpp layer split's 45.7 / 0.72 / 71.7 and llama.cpp `--split-mode tensor`'s
30.5 / 0.71 / 54.3. Honest caveat: the arms also differ in quantization (MXFP4
vs Q8_0), so part of the margin is the quant, not the parallelism. Re-measured
against production after deploy: 92.4 t/s, 0.22 s, 248.9.
**Vision explicit, reasoning deepseek-format.** `--mmproj` names the BF16
projector (this build never autoloads one — a server without the flag comes
up healthy and text-only and answers image requests 500). `--jinja` with the
GGUF's embedded template; `--reasoning-format deepseek` fills
`reasoning_content`. The Qwen3.8 template gates `reasoning_effort` to
`{xhigh, medium, low}` — folded by `reasoningEffortMap` in `models.yml`.
One stream, full 262,144 ctx (`PARALLEL_SLOTS=1`); greedy server default,
Qwen-recommended sampling rides `compat.extraBody` per request. `--flash-attn
on`, batch 8192 / ubatch 512 (the measured ROCm optimum class), port 7770,
alias `frank-primary`.
Files: `dankserver/frank-medium/quadlet/frank-medium.container`,
`dankserver/frank-medium/radiance-entrypoint.sh` (quantization-aware),
`dankserver/frank-medium/frank-medium-healthcheck.py`. Swap toolchain:
`scripts/hf-manifest.py` (generate a `<filename> <size> <sha256>` manifest
from the live repo tree), `scripts/hf-fetch.sh` (resumable, byte + sha256
verified, idempotent fetch), `scripts/swap-frank-medium.sh` (generate
manifest → download → detect quantization → edit the quadlet → deploy). The
llama.cpp layer-split trio (`scripts/frank-medium.service`, `.env`,
`serve-frank-medium.sh`) is archived in `scripts/.AGENTTRASH/
20260920-llama-layersplit/`; `deploy-remote.sh`'s `retire_llama_medium_unit`
moved the installed unit symlink there rather than `systemctl disable`, which
would delete it. The deploy handler reads `SNAP`/`SPEC_METHOD` from the
quadlet, gates on the image tag + the checkpoint's `config.json` (drafter only
for `dflash`, `libr4d` only for `mxfp4`/`quark`), then waits for :7770 and
gates on `/health`.
Files: `scripts/frank-primary.service` (root unit, relinked to
`/etc/systemd/system/`), `scripts/serve-frank-primary.sh` (validates binary
+ all 6 parts + drafter + mmproj, then `exec`s llama-server),
`scripts/frank-primary.env` (tunables; the bind address is `BIND_HOST` —
never a bare `HOST`, HIP reads it). The deployer's `deploy_llm_primary`
gates every artifact at its declared byte count (`stat -c %s`, never `wc -c`),
retires the lucebox quadlet to `~/.AGENTTRASH/20260926-llamacpp-cutover/`,
waits out port 7770, relinks/enables/restarts the root unit and waits up to
1800 s for `/health` (cold start page-ins ~175 GiB).
Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` (serving
details); swap process + toolchain: `serverAppConfigs-docs/LLM/
frank-medium-model-swap.md`.
Rollback: the MXFP4 checkpoint on disk (`SPEC_METHOD=dflash`, `SPEC=7`);
before that, [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20).
Runbook: `serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md`.
Rollback: the lucebox DS4.1 material on host (image
`localhost/lucebox-rocm:gfx1201` + the 377 GiB payload + the archived
quadlet files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`) — see
[[frank-primary-lucebox-ds4]]; the Qwen payload stays fetched, so a
forward re-swap is a redeploy.
## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
@@ -756,8 +749,7 @@ Active:
|---------|---------|
| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
| frank-primary (llama.cpp Qwen3.8-Flash-Next Q8_0 + shared MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -767,6 +759,8 @@ Retired (rollback targets and history):
|---------|------|--------|---------|
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` |
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-primary`) | Dankserver | 2026-09-26 (same-day swap) | `frank-primary-lucebox-ds4.md` |
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |