llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot

The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.

serverAppConfigs-docs/LLM/frank-archon-lucebox-ds4.md

@@ -0,0 +1,215 @@
+---
+type: configuration
+title: frank-archon Lucebox DeepSeek V4 Runbook
+description: RETIRED 2026-09-26 frank-archon runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-archon-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.
+tags:
+  - frank-archon
+  - dankserver
+  - lucebox
+  - deepseek-v4
+  - ds4
+  - rocmfp23
+  - gfx1201
+  - r9700
+  - quadlet
+  - moe
+timestamp: 2026-09-26
+---
+
+# frank-archon Lucebox DeepSeek V4 Runbook
+
+> [!warning] RETIRED 2026-09-26 (same day it went live)
+> Replaced by [[frank-archon-llamacpp-qwen38-flash-next-q8]] — unsloth
+> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
+> served for hours; the rollback material (image `localhost/lucebox-rocm:
+> gfx1201` and the 378 GiB payload) is retained on the host, and the
+> quadlet-era files live in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`.
+
+The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood on
+2026-09-26: the
+[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
+`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
+bytes, LFS oid
+`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a
+single Radeon AI PRO R9700 (gfx1201). It replaced
+[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired earlier the
+same day.
+
+## Why this shape
+
+DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert
+per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3
+down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB
+the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM,
+so it is served the way llama.cpp serves big models by default: the GGUF is
+mmapped, pages pull from disk/page cache on demand, dense work (attention,
+shared expert, embeddings, output projection) and the MLA KV cache run on
+the R9700, and the routed experts are computed on CPU. The 246 GiB RAM
+(upgraded for this deployment) holds the working set of active experts; the
+rest streams from the 1.8 T NVMe.
+
+Lucebox's `--ds4-fused-decode` requests a monolithic GPU load; on a model
+this size that allocation fails and the backend falls back to **hybrid
+expert placement + layered decode** — that fallback is the intended serving
+mode here, not an error.
+
+## Image
+
+`localhost/lucebox-rocm:gfx1201`, built on the host from
+`github.com/Luce-Org/lucebox` `Dockerfile.rocm`:
+
+```
+git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
+cd lucebox
+# Three local modifications (repo-tracked at
+# dankserver/frank-archon/lucebox-gfx1201.patch):
+#   1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
+#      base already ships them in /opt/rocm; the apt repo does not serve
+#      them) and the three runtime math libs.
+#   2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
+#      3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
+#      requires, CMake >= 3.25).
+#   3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
+#      `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
+#      unconditionally, but the macro is never defined by the build
+#      system, so the in-guard definition was dead and the use failed.
+git apply ../lucebox-gfx1201.patch
+podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
+             --build-arg ROCM_VERSION=7.14.1-full \
+             -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .
+```
+
+- The prebuilt `ghcr.io/luce-org/lucebox-hub:rocm` is **not usable as-is**:
+  it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not
+  code-object compatible.
+- `ROCM_VERSION=7.14.1-full` matches the host driver's ROCm 7.x line (the
+  radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace
+  on a 7.x driver can segfault at model load. Only `-full` tags exist for
+  7.14.x.
+- The binary lands at `/opt/lucebox-hub/server/build/luce_server`.
+
+## Model
+
+`/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf`,
+fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
+400 against HF's xethub CDN for this repo; see the fetch note in
+`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
+`deploy_llm_archon`.
+
+## Deployed flags (2026-09-26)
+
+From `dankserver/frank-archon/frank-archon-entrypoint.sh` (the unit
+bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
+flag set and cannot express the DS4 flags):
+
+| Setting | Value |
+|---|---|
+| Model | `DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB) |
+| Device | `--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out) |
+| Window | `--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small) |
+| Chunk | `--chunk 8192` (the qualified profile's chunk) |
+| KV dtype | `--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch) |
+| Decode | `--ds4-fused-decode` (falls back to hybrid expert placement on this host) |
+| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
+| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
+| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
+| Identity | `--model-name frank-archon`, `--host 0.0.0.0 --port 7770` |
+
+## API surface
+
+OpenAI Chat Completions (streaming + tool metadata), `/v1/models`,
+`/health`, OpenAI Responses, Anthropic Messages, and Open WebUI model
+metadata — the fleet contract is unchanged from frank-medium.
+
+## Cutover procedure
+
+1. Image built and model fetched (gates in `deploy_llm_archon`).
+2. `deploy.sh` → `deploy_llm_archon`: stops and **masks** the radiance
+   unit (`frank-medium`), waits for :7770 to free, installs
+   `dankserver/frank-archon/quadlet/frank-archon.container`, starts
+   `frank-archon.service`, waits for health (1800 s start period — first
+   start page-ins the dense weights and builds the graphs).
+3. Verify: `/health` 200 is not enough — a real chat completion, a tool
+   call, and an Open WebUI completion.
+4. Retire the radiance unit to the home trash (never `rm`); the MXFP6
+   checkpoint payload stays on disk.
+
+## Rollback
+
+Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
+systemctl --user start frank-medium`) and stop/mask frank-archon. The
+`localhost/radiance-vllm:0.13.0-e060895` image and the
+`/models/Qwen3.8-27B-PARO-MXFP6` checkpoint remain on disk.
+
+## Verification (2026-09-26)
+
+_Filled in after the cutover: health, real completion, tool call, Open WebUI
+completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
+state._
+
+## Dual-R9700 follow-up (2026-09-28 probe era)
+
+After this single-card runbook was retired, the DS41 engine got a dual-R9700
+attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
+`unsupported model architecture 'deepseek41'`). Findings that outlive the
+probe:
+
+**Budget both cards explicitly, or you waste half the box.** The engine
+sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
+every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
+looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
+cache of things still streaming from NVMe. Pin all four:
+
+| env | card | meaning |
+|---|---|---|
+| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
+| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
+| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
+| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
+
+Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
+card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
+
+**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
+`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
+gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
+tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
+secondary=hip:1`. It is the only lever that materially raises resident
+expert count, and resident count is the entire bottleneck. Confirm every run
+with `expert ownership:` and `hybrid experts ready:` lines.
+
+**The placement file must live under the mounted models dir.** JSON
+`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
+`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
+keeps what fits its budget and demotes the rest. Anywhere outside
+`/models/...` gives `cannot open expert placement` and a SILENT single-tier
+fallback — a silent-wrongness failure, not a crash.
+
+**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
+exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
+is refused there. With ~22% of experts resident each token pulls ~187 cold
+experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
+the model is larger than the box's RAM and you are in page-cache thrash at
+the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
+with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
+decode, not a broken run.
+
+**Harness traps, each of which produced a wrong number:** never `env -i` the
+podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
+PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
+before the container exists — pass container env with `-e`); never put a
+large prompt on curl's argv (`Argument list too long` at ~128 KB — use
+`--data-binary @file`); size prompts by growing the file until `wc -c`
+reaches the byte target AND assert depth against the engine's own
+`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
+that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
+30%); the server runs as the container's PID 1, so tear down with
+`podman rm -f`, never host `pkill`; drop page cache between arms
+(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
+every later one is a lie.
+
+Read the placement log — it is the whole diagnosis
+(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
+hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
+… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
+mapped, so they do not evict the expert page cache).