From efff47a75ddd1e66cfb53c6b91d5abd892bb344d Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 27 Sep 2026 04:39:30 -0700 Subject: [PATCH] AGENTS.md: frank-primary section rewritten for the R9V cutover Fleet topology row + full endpoint section now describe the R9V radiance shape (on-host 104-key state, shim laws, launcher-driven serve, docker-wait self-healing, measured numbers); llama.cpp section retired into the runbook pointer. Bench note: mask fails on a relinked unit symlink - stop is enough, redeploy restores. --- AGENTS.md | 122 +++++++++++++++++++++++++++++------------------------- 1 file changed, 66 insertions(+), 56 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 054bd07..b6730b0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | unsloth llama.cpp b11030-mix prebuilt (ROCm gfx120X, bundled runtime, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q_8 (175.30 GiB, 6 split parts) with the shared Q_8 MTP head (depth 2) chained behind `ngram-mod`, BF16 vision (mmproj), layer split `-ts 1,1` across both R9700s with the routed experts + 51B-param n-gram table computed in host RAM (`-ot exps=CPU,per_layer_token_embd.weight=CPU` — the shared-head borrow path needs the last), 262144 ctx single stream (1 slot), mmap on, `--cache-ram 16384` | 7770 | +| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8 MTP4 head (depth 4) and F16 mmproj, TP=2 tiered experts (UVA pinned host copy 43 GiB rank 0 + 16 GiB PLE pinned reserve; PLE n-gram `ssd` mmap; budgets 146.25/78.75 GiB by device), CED on, 196608 ctx single stream, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 | | Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | @@ -231,9 +231,13 @@ merely wedged kernel usually still answers ARP. `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - - Dankserver `frank-primary` (root unit, llama.cpp on the 2x R9700): + - Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700): `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary` 2. Bench. + NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in + `/etc/systemd/system/` is itself a symlink into the checkout — mask + refuses to replace it). Stop is enough for the bench window; the deploy + restores the unit, so re-deploy after benching instead of unmasking. 3. Unmask and start; verify `curl http://192.168.0.:7770/health`. **CRITICAL:** never `systemctl disable` a system unit on these hosts — it @@ -341,66 +345,72 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-primary — llama.cpp Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE) +### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (Dankserver, ACTIVE) -unsloth llama.cpp **b11030-mix prebuilt** (ROCm gfx120X, commit `6ba30d05b`, -bundled ROCm runtime at `/home/dank/GitPackages/unsloth-llama-b11030/`, **root -systemd unit** `scripts/frank-primary.service`, `User=dank`) serving -`unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8** — 175.30 GiB across six split -parts under `/home/dank/models/Qwen3.8-Flash-Next-GGUF/Q8_0/`, fetched by -`scripts/download-frank-primary.sh` (per-file sha256 + byte-count verified) — -on **both** Radeon AI PRO R9700s. Replaced the Lucebox DS4.1 quadlet (hours -old) on 2026-09-26, which had replaced radiance frank-medium the same day. +The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned +prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving +`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` (abliterated weights, +92.39 GiB, 3 GGUF shards) with the blockfp8 MTP4 head (depth 4) and F16 +mmproj on **both** Radeon AI PRO R9700s (gfx1201, BDFs `0000:03:00.0` / +`0000:07:00.0`), one stream at **196,608 ctx**, **TP=2 tiered experts** +(UVA pinned host copy ~43 GiB on rank 0 + 16 GiB PLE pinned reserve; PLE +n-gram table on `ssd` mmap residency; per-device offload budgets +146.25/78.75 GiB), **CED on** (split-16 projector engagement at >=8192-token +prompts), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut +over 2026-09-27 from the llama.cpp b11030 endpoint (downtime ~1 h 24 m) at +Frank's direction. -**Placement: compute the experts in RAM, never stream them.** Layer split -`-ts 1,1` with `-ngl 999`; the routed experts and the 51B-param n-gram -(PLE) table are overridden to CPU (`-ot -exps=CPU,per_layer_token_embd.weight=CPU`) and computed -from the mmap page cache — the x4-pair law (streaming over PCIe loses at -3.7 t/s; computing in RAM does not care). `per_layer_token_embd.weight=CPU` -is REQUIRED by the shared MTP head's borrow path (a GPU-resident borrow -crashes the draft load with `invalid vector subscript`). Dense weights, -attention, the BF16 projector and the MTP head sit on the cards (~29 GiB); -mmap must stay on (`--no-mmap` forces the table into anonymous memory → -silent cgroup kill) and the PLE tensor must never carry a GPU target -(23 → 13.4 t/s when tried). `--cache-ram 16384`; KV is the hybrid -DeltaNet/QSA pattern, a fraction of the dense-model cost. +**The tuned shape lives on-host, not in the repo.** The profile's setup run +qualified a 104-key `R9V_*` config (placement, budgets, parsers, CED) into +`/home/dank/r9v-state/uncensored/setup.json` (`ready: true`); the repo env +file `scripts/frank-primary.env` carries ONLY production overrides +(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`, +`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`, +`R9V_EXPECTED_GPU_BDFS`). Never copy tuned values from the state into the +repo — it would drift from the qualification. -**Spec chain `draft-mtp,ngram-mod`, depth 2.** The shared Q_8 head -(`MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`, 2,786,568,256 B — unsloth's -recommended pairing) loads via `--model-draft` and borrows the target's -embedding tensors; `ngram-mod` chains behind it (draftless, so legal). -The dankclaw UD-IQ4_XS deployment of the same model family ran this head -at depth 2 with 0.83–0.92 acceptance. ngram-mod prints no startup banner — -identify it behaviourally (`draft_n` > 0 where text repeats). +**Serving = the fork's own launcher through a docker→podman shim.** The R9V +tooling is docker-pinned; the shim at +`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries +three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys +the image by its OCI manifest digest while podman keys by config digest but +preserves the manifest digest in `RepoDigests` — the shim answers +`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run +refs to `repo@sha256:X`; (2) podman inspect answers +`HostConfig.LogConfig.Config: null` where docker returns the log-opts map +(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs +and normalizes inspect output; (3) podman lacks docker's `SecurityOptions` +/`ServerVersion` info fields — the shim answers them. -**Vision explicit, reasoning deepseek-format.** `--mmproj` names the BF16 -projector (this build never autoloads one — a server without the flag comes -up healthy and text-only and answers image requests 500). `--jinja` with the -GGUF's embedded template; `--reasoning-format deepseek` fills -`reasoning_content`. The Qwen3.8 template gates `reasoning_effort` to -`{xhigh, medium, low}` — folded by `reasoningEffortMap` in `models.yml`. -One stream, full 262,144 ctx (`PARALLEL_SLOTS=1`); greedy server default, -Qwen-recommended sampling rides `compat.extraBody` per request. `--flash-attn -on`, batch 8192 / ubatch 512 (the measured ROCm optimum class), port 7770, -alias `frank-primary`. +**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`, +`User=dank`, `LimitMEMLOCK=infinity`) — the tiered-expert UVA host copy pins +~43 GiB and the PLE reserve 16 GiB, and a root unit grants the pin budget +via PID1 (a rootless quadlet would be capped by the user manager's limit). +The serve script (`scripts/serve-frank-primary.sh`) gates the setup state +(ready + paths), the pinned image (RepoDigests), the PLE byte count +(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes, +clears a stale container, then execs the fork's `scripts/launch.sh` through +the shim and **blocks on `docker wait`** — the launcher `--detach`es, so +without the wait a crashed engine is a green unit; `docker wait` cannot stop +the container child, hence `ExecStop=podman stop -t 120 +frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/ +PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/ +restarts the unit and health-gates with the MainPID check (the false-green +stray law). -Files: `scripts/frank-primary.service` (root unit, relinked to -`/etc/systemd/system/`), `scripts/serve-frank-primary.sh` (validates binary -+ all 6 parts + drafter + mmproj, then `exec`s llama-server), -`scripts/frank-primary.env` (tunables; the bind address is `BIND_HOST` — -never a bare `HOST`, HIP reads it). The deployer's `deploy_llm_primary` -gates every artifact at its declared byte count (`stat -c %s`, never `wc -c`), -retires the lucebox quadlet to `~/.AGENTTRASH/20260926-llamacpp-cutover/`, -waits out port 7770, relinks/enables/restarts the root unit and waits up to -1800 s for `/health` (cold start page-ins ~175 GiB). +Verified at cutover: setup PASS=28/WARN=3/FAIL=0 (20/20 artifacts +sha256-verified, PLE at pinned size); first-start doctor battery PASS=34/ +WARN=8/FAIL=0 including pinned-UVA reads, H2D copies and the TP all-reduce +verified INSIDE serving workers; wire: alias `frank-primary`, `/health` + +`/metrics` 200, non-thinking PONG, thinking → `reasoning`, tool_calls +round-trip, vision ('red' for a red PNG), needle recall exact at 111,647 +prompt tokens (95% depth, ~4.5k tok/s effective prefill), decode 59–96 +tok/s (draft-acceptance dependent). MemAvailable floor 56.3 GiB. -Runbook: `serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md`. -Rollback: the lucebox DS4.1 material on host (image -`localhost/lucebox-rocm:gfx1201` + the 377 GiB payload + the archived -quadlet files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`) — see -[[frank-primary-lucebox-ds4]]; the Qwen payload stays fetched, so a -forward re-swap is a redeploy. +Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-uncensored.md`. +Rollback: the llama.cpp trio at commit `4992e5a` (unit/env/serve) and the +GGUF payload on disk — see [[frank-primary-llamacpp-qwen38-flash-next-q8]]; +a rollback is a redeploy. ## Key Lessons Learned Durable operational lessons from the fleet's history. Full detail in the