feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets
The radiance TP=2 cutover changed the metric namespace frank-medium exports (llamacpp:* -> vllm:*) and left two scrape targets addressing host processes by an address a container cannot reach. - shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never had), the job:llm_vllm_prefix_cache_reuse:rate5m and job:llm_spec_accept_length:rate5m recording rules, and re-cut LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with per-family traffic floors so an idle endpoint cannot fire them. Every expression validated against live Prometheus. - shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771 and frank-large scraped at host.containers.internal:7770. Both are HOST processes on Dankclaw and netavark DNATs published container ports only, so the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp build does export the llamacpp:* family, contra the old comment. - tests/observability_quadlet.bats: the two seams now encode the new contract (per-endpoint reachability by process type; every named series must be on the measured list for its engine; engine-independent rules must name both). - homepage docs.html/services.json: frank-medium sees pictures, frank-large is CIRU Orca with a single seat (was halogen w4b, two seats, 200k). - docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified capabilities, rollback, replica-era sections labelled); llama.cpp runbook carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured verdict; Observability records the namespace split and the DNAT lesson; AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook index flipped; both indexes updated; work-log entry added. Also repaired agent-pass damage: prose prepended above frontmatter openers, duplicated runbook bodies, a literal tool-call token pasted into a runbook, and an index destroyed by diff artifacts (restored from HEAD, flips re-applied). Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
1 parent
c2116658f8
commit
08f1c7ad09
13 files changed
+687
-286
No files matched your search
@@ -1,13 +1,13 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-19):
|
||||
**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split
|
||||
process across both Radeon AI PRO R9700 cards (orcarouter Qwen3.8-27B-
|
||||
Uncensored Q8_0 + the z-lab DFlash2 Q4_K_M drafter, balanced 32:33), and TWO
|
||||
endpoints on Dankclaw — the CIRU v4.4.1 runtime (frank-large, jcbtc
|
||||
Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU, port 7770) and mainline
|
||||
llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
|
||||
the RTX 3090, port 7771).
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-20):
|
||||
**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
|
||||
process across both Radeon AI PRO R9700 cards (just1moremodel
|
||||
Qwen3.8-27B-Uncensored MXFP4-awq + the z-lab DFlash2-FP8 drafter, TP=2 over the
|
||||
R4D one-shot allreduce), and TWO endpoints on Dankclaw — the CIRU v4.4.1
|
||||
runtime (frank-large, jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
|
||||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT
|
||||
Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -21,10 +21,10 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[llama.cpp, layer split
|
||||
B2[vllm-radiance 0.9.3, TP=2
|
||||
frank-medium, port 7770]
|
||||
C[orcarouter Qwen3.8-27B Q8_0 + DFlash2 Q4_K_M
|
||||
ONE process on both cards, 2x163840 ctx, balanced 32:33]
|
||||
C[Qwen3.8-27B MXFP4-awq + DFlash2-FP8
|
||||
ONE process on both cards, 262144 ctx, up to 8 seqs]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
@@ -46,6 +46,7 @@ frank-small, port 7771]
|
||||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||||
CLI -->|HTTP :7770| E
|
||||
CLI -->|HTTP :7771| G
|
||||
```
|
||||
|
||||
## Servers
|
||||
|
||||
@@ -56,21 +57,25 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
|
||||
2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving
|
||||
orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M
|
||||
drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process
|
||||
behind :7770 (tensor parallelism was impossible on the old board's second
|
||||
card; the 2026-09-20 swap cleared that gate but the runtime still ships
|
||||
RCCL-less and the collective is unprobed), balanced 32:33 over two 163,840-token
|
||||
slots
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
|
||||
deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
|
||||
2026-09-20: vllm-radiance MXFP4 (stilldeadcode/vllm-radiance:0.9.3 +
|
||||
ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas
|
||||
behind a round-robin proxy; kept as the record of the deployment it
|
||||
succeeded, plus the PCIe-atomics evidence that ruled out TP=2 on the previous
|
||||
board — that gate cleared on 2026-09-20, see [[Dual-R9700-Cooperation]]
|
||||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the z-lab DFlash2-FP8
|
||||
drafter as ONE tensor-parallel process (TP=2) behind :7770 — the three-arm
|
||||
battery that chose it, the auto-sized 943,581-token KV pool, verified
|
||||
capabilities (tools, reasoning, vision), and the record of the 2026-09-16
|
||||
two-replica era
|
||||
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
|
||||
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
|
||||
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
|
||||
163,840-token slots); superseded by the radiance TP=2 cutover, and the
|
||||
rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
|
||||
arithmetic
|
||||
- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
|
||||
RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
|
||||
B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
|
||||
(radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
|
||||
that put TP=2 in production
|
||||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
|
||||
2026-09-15): charlie12345/ROCmFPX serving
|
||||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||||
|
||||
Reference in new issue
Block a user