feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank committed 2026-09-20 19:35:56 -07:00
1 parent c2116658f8
commit 08f1c7ad09
13 files changed
+687 -286

No files matched your search

+30 -25
View File
@@ -1,13 +1,13 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on the two hosts (2026-09-19):
**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split
process across both Radeon AI PRO R9700 cards (orcarouter Qwen3.8-27B-
Uncensored Q8_0 + the z-lab DFlash2 Q4_K_M drafter, balanced 32:33), and TWO
endpoints on Dankclaw — the CIRU v4.4.1 runtime (frank-large, jcbtc
Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU, port 7770) and mainline
llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
the RTX 3090, port 7771).
Documentation for the LLM inference fleet on the two hosts (2026-09-20):
**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
process across both Radeon AI PRO R9700 cards (just1moremodel
Qwen3.8-27B-Uncensored MXFP4-awq + the z-lab DFlash2-FP8 drafter, TP=2 over the
R4D one-shot allreduce), and TWO endpoints on Dankclaw — the CIRU v4.4.1
runtime (frank-large, jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT
Q4_K_M with MTP and vision on the RTX 3090, port 7771).
> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -21,10 +21,10 @@ graph LR
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
B2[llama.cpp, layer split
B2[vllm-radiance 0.9.3, TP=2
frank-medium, port 7770]
C[orcarouter Qwen3.8-27B Q8_0 + DFlash2 Q4_K_M
ONE process on both cards, 2x163840 ctx, balanced 32:33]
C[Qwen3.8-27B MXFP4-awq + DFlash2-FP8
ONE process on both cards, 262144 ctx, up to 8 seqs]
A --> B2 --> C
end
@@ -46,6 +46,7 @@ frank-small, port 7771]
CLI[LLM Clients / API] -->|HTTP :7770| B2
CLI -->|HTTP :7770| E
CLI -->|HTTP :7771| G
```
## Servers
@@ -56,21 +57,25 @@ frank-small, port 7771]
## Active Deployment
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving
orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M
drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process
behind :7770 (tensor parallelism was impossible on the old board's second
card; the 2026-09-20 swap cleared that gate but the runtime still ships
RCCL-less and the collective is unprobed), balanced 32:33 over two 163,840-token
slots
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
2026-09-20: vllm-radiance MXFP4 (stilldeadcode/vllm-radiance:0.9.3 +
ggz14/radiance-vllm-mxfp4 patches) serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas
behind a round-robin proxy; kept as the record of the deployment it
succeeded, plus the PCIe-atomics evidence that ruled out TP=2 on the previous
board — that gate cleared on 2026-09-20, see [[Dual-R9700-Cooperation]]
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the z-lab DFlash2-FP8
drafter as ONE tensor-parallel process (TP=2) behind :7770 — the three-arm
battery that chose it, the auto-sized 943,581-token KV pool, verified
capabilities (tools, reasoning, vision), and the record of the 2026-09-16
two-replica era
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
163,840-token slots); superseded by the radiance TP=2 cutover, and the
rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
arithmetic
- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
(radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
that put TP=2 in production
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
2026-09-15): charlie12345/ROCmFPX serving
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the