docs: frank-primary runs FreeToken (PR587 ROCm) — runbook, index, AGENTS, log
New runbook frank-primary-freetoken-pr587 (build laws, shape, measured cutover numbers, TP=2 non-boot signature, rollback); LLM/index.md intro + mermaid + active/retired entries; AGENTS.md fleet row + section + bench and model-config mentions + runbook index; log.md entry. Seam pins the measured KV pool 299968 with the multiple-of-64 guard. NOTE: intended PGP signature skipped — key 0x76EA601A11B178D7 has no secret key in this machine's keyring (fresh ~/.gnupg); the PGP identity lives elsewhere.
This commit is contained in:
1 parent
6ba19f6f48
commit
e2308dff71
5 files changed
+295
-139
No files matched your search
@@ -1,18 +1,17 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-03):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **shape E since
|
||||
2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
|
||||
conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
|
||||
decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
|
||||
(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
|
||||
(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
|
||||
mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
|
||||
`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
|
||||
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-05):
|
||||
**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
|
||||
image `frank-primary-freetoken:pr587-77a7201` built on the host from
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
|
||||
`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
|
||||
driving `podman run` directly, serving orcarouter
|
||||
/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
|
||||
pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
|
||||
`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
|
||||
decode slots, parsers AUTO (thinking → `reasoning_content` natively;
|
||||
measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
|
||||
runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw
|
||||
runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
@@ -34,12 +33,12 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
B2[FreeToken PR587 ROCm, local image
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8, no spec decode
|
||||
TP=2, expert offload automatic, 3 x 200704 ctx,
|
||||
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
|
||||
expert-precise-memory arena 7,522 slots, /root/.cache mounted]
|
||||
C[Qwen3.8-Flash-Next FP8
|
||||
1x R9700, experts in pinned host RAM + GPU LRU 10%,
|
||||
KV pool 299,968 tokens, 2 slots, parsers AUTO,
|
||||
TP=2 untested on ROCm; second R9700 idle]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -95,8 +94,22 @@ frank-small, port 7771]
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
|
||||
(branch rocm714_torch214) built on the host into the local image
|
||||
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||||
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
|
||||
(~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
|
||||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
|
||||
frank-primary-vllm-shapeG-20261003
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||||
|
||||
Reference in new issue
Block a user