feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets
The radiance TP=2 cutover changed the metric namespace frank-medium exports (llamacpp:* -> vllm:*) and left two scrape targets addressing host processes by an address a container cannot reach. - shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never had), the job:llm_vllm_prefix_cache_reuse:rate5m and job:llm_spec_accept_length:rate5m recording rules, and re-cut LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with per-family traffic floors so an idle endpoint cannot fire them. Every expression validated against live Prometheus. - shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771 and frank-large scraped at host.containers.internal:7770. Both are HOST processes on Dankclaw and netavark DNATs published container ports only, so the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp build does export the llamacpp:* family, contra the old comment. - tests/observability_quadlet.bats: the two seams now encode the new contract (per-endpoint reachability by process type; every named series must be on the measured list for its engine; engine-independent rules must name both). - homepage docs.html/services.json: frank-medium sees pictures, frank-large is CIRU Orca with a single seat (was halogen w4b, two seats, 200k). - docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified capabilities, rollback, replica-era sections labelled); llama.cpp runbook carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured verdict; Observability records the namespace split and the DNAT lesson; AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook index flipped; both indexes updated; work-log entry added. Also repaired agent-pass damage: prose prepended above frontmatter openers, duplicated runbook bodies, a literal tool-call token pasted into a runbook, and an index destroyed by diff artifacts (restored from HEAD, flips re-applied). Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap — atomic routing verified on both chains), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770|
|
||||
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | Qwen3.8-27B-Uncensored MXFP4-awq with the z-lab DFlash2-FP8 drafter (depth 7, live mean accept length ~4.1), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 943,581-token KV pool = 3.59x the window), fp8 KV, prefix caching, vision | 7770 |
|
||||
| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -68,7 +68,9 @@ All edits flow through git. Never edit files on a remote machine directly.
|
||||
The planner maps changed paths to services; the remote runner restarts only
|
||||
the selected service and records its revision outside the checkout.
|
||||
Documentation-only changes are a no-op. LLM services are selected only by
|
||||
their own files — all three by their unit/env/serve trio in `scripts/` — so
|
||||
their own files — `frank-large`/`frank-small` by their unit/env/serve trio in
|
||||
`scripts/`, `frank-medium` by its Quadlet unit, entrypoint and healthcheck under
|
||||
`dankserver/frank-medium/` — so
|
||||
unrelated
|
||||
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
||||
runs on a host without passwordless sudo; the value never enters the repo.
|
||||
@@ -230,8 +232,9 @@ merely wedged kernel usually still answers ARP.
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-medium` (system unit):
|
||||
`sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
|
||||
- Dankserver `frank-medium` (**rootless Quadlet since 2026-09-20** — the
|
||||
system unit is retired, so `sudo systemctl` sees nothing):
|
||||
`systemctl --user stop frank-medium && systemctl --user mask frank-medium`
|
||||
2. Bench.
|
||||
3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.
|
||||
|
||||
@@ -259,9 +262,14 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
|
||||
`frank-large` is text-only because its CIRU deployment loads no vision
|
||||
projector (the F16 mmproj is on disk, one env flip away). There is **no audio**
|
||||
of 2026-09-20 both `frank-small` and `frank-medium` accept images
|
||||
(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
|
||||
Gemma4 merge path with no separate mmproj, verified on held-out synthetic
|
||||
images; `frank-large` remains text-only because its CIRU deployment loads no
|
||||
vision projector (the F16 mmproj is on disk, one env flip away). Note also that
|
||||
the reasoning channel differs by runtime: llama.cpp endpoints emit
|
||||
`reasoning_content`, vLLM emits `reasoning`.
|
||||
There is **no audio**
|
||||
input anywhere in the fleet: that capability left with the FLM NPU endpoint on
|
||||
2026-09-11.
|
||||
|
||||
@@ -326,63 +334,49 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
### frank-medium — llama.cpp layer split, DFlash2 drafter (Dankserver, ACTIVE)
|
||||
### frank-medium — radiance TP=2, DFlash2-FP8 drafter (Dankserver, ACTIVE)
|
||||
|
||||
Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
|
||||
systemd unit `scripts/frank-medium.service`) serving
|
||||
`orcarouter/Qwen3.8-27B-Uncensored-GGUF` at **Q8_0** (27.0 GiB) with the
|
||||
**z-lab DFlash2 `Q4_K_M` drafter** (1.06 GiB). The target file also carries the
|
||||
model's own MTP head (`blk.<n>.nextn.*`, `nextn_predict_layers=1`), so
|
||||
`draft-mtp` is one `SPEC_TYPE` change away — the drafter file stays on disk. In
|
||||
**one process across both R9700s**, `--split-mode layer` at a **32:33 tensor
|
||||
split** — balanced (operator call 2026-09-19, the most even split 65 layers
|
||||
allow; ROCm1, the last-layers card that also carries the drafter and output
|
||||
layers, holds the extra one).
|
||||
The predecessor's 35:30 skew (2026-09-17) leaned work onto the wider card
|
||||
because ROCm1 (then `0b:00.0`) rode the B650 chipset at Gen4 x1 (~1.97 GB/s)
|
||||
against ROCm0's CPU-root-port Gen4 x16 (~31.5 GB/s); the balanced point traded
|
||||
that lean for symmetry and was accepted. The 2026-09-20 board swap (ASUS ProArt
|
||||
B850-Creator WiFi NEO) removed the other axis: both cards are now CPU-root-port
|
||||
Gen5 x8 (~31.5 GB/s each), at `03:00.0` and `07:00.0`. 327680 ctx
|
||||
over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so
|
||||
no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16
|
||||
KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was
|
||||
tried in the drafter's place on 2026-09-16 and reverted: it declares no sliding
|
||||
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
|
||||
layers of the split.
|
||||
vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet**
|
||||
`dankserver/frank-medium/quadlet/frank-medium.container`) serving
|
||||
`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4) with the
|
||||
**z-lab DFlash2 FP8 drafter** (`num_speculative_tokens: 7`, live mean accept
|
||||
~4.1) in **one tensor-parallel process across both R9700s**
|
||||
(`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8
|
||||
KV, prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool auto-sized
|
||||
to 943,581 tokens (3.59× the full window). Tools via `qwen3_coder`, reasoning
|
||||
via `qwen3` (the field is `reasoning`, not llama.cpp's `reasoning_content`),
|
||||
vision through the Gemma4 merge path with `MM_MERGE_ENGINE=AXROW_SEPARATE`.
|
||||
|
||||
**Layer split is why both cards serve ONE model today.** It was chosen because
|
||||
tensor parallelism was then impossible (see the PCIe lesson below) and so was
|
||||
vLLM's pipeline parallelism — its stage handoff is an RCCL `send`/`recv` of the
|
||||
hidden state, the exact primitive that could not be dispatched on the old
|
||||
board's card #2. llama.cpp's layer split needs
|
||||
no collective at all: layers are distributed in order and the boundary
|
||||
activation is a per-backend copy, once per forward pass. Measured on this host
|
||||
(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor — but that
|
||||
tensor number is the cost of a butterfly-fallback allreduce with `GGML_HIP_RCCL`
|
||||
OFF, not what an RCCL-backed tensor split would do. Since the 2026-09-20 board
|
||||
swap cleared the atomics gate, that comparison is stale and the probe is the
|
||||
open work; see [[Dual-R9700-Cooperation]].
|
||||
**TP=2 runs because the board changed.** The predecessor board (TUF GAMING
|
||||
B650-E) put card #2 behind chipset bridges advertising `AtomicOpsCap: Routing-`
|
||||
and no collective could be dispatched there. The ProArt B850-Creator WiFi NEO
|
||||
routes both cards to CPU GPP root ports (`00:01.1 → 03:00.0`,
|
||||
`00:01.3 → 07:00.0`, Gen5 x8, every hop `Routing+`, both endpoints `ReqEn+`).
|
||||
The collective in use is vLLM's **R4D one-shot allreduce**, which never touches
|
||||
RCCL (TheRock RCCL has no gfx1201 device code; the Ubuntu RCCL 2.27 build does,
|
||||
and that is what `scripts/build-llama-rdna4.sh GGML_HIP_RCCL=ON` links).
|
||||
|
||||
This is the box's **only** LLM runtime: the vllm-radiance replica pair and its
|
||||
routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went
|
||||
live, with their units moved to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
|
||||
**Measured, not assumed** (`scripts/bench-conversation.py`, 10 real prompts,
|
||||
max_tokens 512, greedy, salted per arm, service stopped between arms):
|
||||
radiance TP=2 95.8 t/s decode geomean / 0.21 s TTFT / 259.4 t/s burst-4, vs the
|
||||
llama.cpp layer split's 45.7 / 0.72 / 71.7 and llama.cpp `--split-mode tensor`'s
|
||||
30.5 / 0.71 / 54.3. Honest caveat: the arms also differ in quantization (MXFP4
|
||||
vs Q8_0), so part of the margin is the quant, not the parallelism. Re-measured
|
||||
against production after deploy: 92.4 t/s, 0.22 s, 248.9.
|
||||
|
||||
Files: `scripts/build-llama-rdna4.sh` (builds the runtime, gfx1201 only, private
|
||||
ROCm 7.14 toolchain bundled beside the binary), `scripts/download-frank-medium-gguf.sh`
|
||||
(resumable, byte-verified payload), `scripts/frank-medium.env`,
|
||||
`scripts/serve-frank-medium.sh`, `scripts/frank-medium.service`. The deploy
|
||||
handler refuses to start a unit whose runtime or payload is missing, retires
|
||||
the vLLM quadlets, waits for 7770 to be released (the retired proxy held it) and
|
||||
gates on `/health`.
|
||||
Files: `dankserver/frank-medium/quadlet/frank-medium.container`,
|
||||
`dankserver/frank-medium/radiance-entrypoint.sh`,
|
||||
`dankserver/frank-medium/frank-medium-healthcheck.py`. The llama.cpp layer-split
|
||||
trio (`scripts/frank-medium.service`, `.env`, `serve-frank-medium.sh`) is
|
||||
archived in `scripts/.AGENTTRASH/20260920-llama-layersplit/` and is the
|
||||
rollback target; `deploy-remote.sh`'s `retire_llama_medium_unit` moved the
|
||||
installed unit symlink there rather than `systemctl disable`, which would delete
|
||||
it. The deploy handler gates on the image tag, both model configs, the patch
|
||||
clone + jinja template, and `libr4d`, then waits for :7770 and gates on
|
||||
`/health`.
|
||||
|
||||
The ref is pinned by two features and must not float: **DFlash2**
|
||||
(ggml-org/llama.cpp#27342) and **qwen35 MTP**. The ROCmFPX fork the endpoint
|
||||
used earlier that day has neither.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md`.
|
||||
Retired runtimes: [[frank-medium-radiance-mxfp4]] (vLLM replicas, same day),
|
||||
[[frank-medium-rocmfpx]] (llama.cpp fork, earlier that day).
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
|
||||
Rollback: [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20).
|
||||
|
||||
## Key Lessons Learned
|
||||
Durable operational lessons from the fleet's history. Full detail in the
|
||||
@@ -412,9 +406,14 @@ linked runbooks.
|
||||
every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
|
||||
precondition the old test plan named. Bandwidth was never the constraint (a
|
||||
27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
|
||||
and under 0.4 ms at Gen5 x8). **What is NOT yet re-measured:** no collective
|
||||
has been dispatched since the swap, so TP=2 is unblocked, not proven. Full
|
||||
evidence matrix and the standing probe plan: [[Dual-R9700-Cooperation]].
|
||||
and under 0.4 ms at Gen5 x8).
|
||||
**Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and
|
||||
frank-medium serves on it; llama.cpp's RCCL path was built too
|
||||
(`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower
|
||||
shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201
|
||||
device code (`invalid device function`), which is why R4D is the path vLLM
|
||||
uses at all. Evidence matrix and the measured three-arm verdict:
|
||||
[[Dual-R9700-Cooperation]].
|
||||
- **When a GPU collective dies, read the runtime, not the framework.** The
|
||||
message that identified the cause came from `AMD_LOG_LEVEL=3`
|
||||
(`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
|
||||
@@ -428,7 +427,11 @@ linked runbooks.
|
||||
collective: layers are distributed in order and only the boundary activation
|
||||
crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
|
||||
18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
|
||||
fallback). That is what `frank-medium` runs — see
|
||||
fallback).
|
||||
This was what `frank-medium` ran until 2026-09-20, when the board swap made
|
||||
collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s
|
||||
decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see
|
||||
[[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target
|
||||
[[frank-medium-llamacpp-dflash2]].
|
||||
- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
|
||||
combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
|
||||
@@ -467,9 +470,10 @@ linked runbooks.
|
||||
|
||||
**ROCm / GPU**
|
||||
|
||||
- **Layer split is the deployed shape on both multi-device llama.cpp services,
|
||||
for two different reasons.** On Dankserver it is the *only* way both R9700s
|
||||
serve one model (no collective can be dispatched on card #2). On a
|
||||
- **Layer split was the deployed shape on both multi-device llama.cpp services,
|
||||
for two different reasons.** On Dankserver it was the *only* way both R9700s
|
||||
served one model (no collective could be dispatched on card #2) until the
|
||||
2026-09-20 board swap; frank-medium now runs radiance TP=2 instead. On a
|
||||
RCCL-less fork build it is also the fast one: tensor parallel falls back to
|
||||
a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s).
|
||||
Pin the spec-decode drafter to one device — an unpinned draft under tensor
|
||||
@@ -717,7 +721,7 @@ Active:
|
||||
|---------|---------|
|
||||
| frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
|
||||
| frank-medium (radiance TP=2 + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
@@ -739,6 +743,7 @@ Retired (rollback targets and history):
|
||||
| FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
|
||||
| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
|
||||
| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
|
||||
| llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
|
||||
|
||||
Retired runbooks are under `serverAppConfigs-docs/LLM/`. Retired scripts live
|
||||
in `old/` and `.AGENTTRASH/`.
|
||||
|
||||
Reference in New Issue
Block a user