docs: frank-primary runs FreeToken (PR587 ROCm) — runbook, index, AGENTS, log

New runbook frank-primary-freetoken-pr587 (build laws, shape, measured
cutover numbers, TP=2 non-boot signature, rollback); LLM/index.md intro +
mermaid + active/retired entries; AGENTS.md fleet row + section + bench
and model-config mentions + runbook index; log.md entry. Seam pins the
measured KV pool 299968 with the multiple-of-64 guard.

NOTE: intended PGP signature skipped — key 0x76EA601A11B178D7 has no
secret key in this machine's keyring (fresh ~/.gnupg); the PGP identity
lives elsewhere.
This commit is contained in:
Frank committed 2026-10-05 19:35:31 -07:00
1 parent 6ba19f6f48
commit e2308dff71
5 files changed
+295 -139

No files matched your search

+33 -20
View File
@@ -1,18 +1,17 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on the two hosts (2026-10-03):
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
on :7770, TP=2 with automatic tiered-expert offload, **shape E since
2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
Documentation for the LLM inference fleet on the two hosts (2026-10-05):
**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
image `frank-primary-freetoken:pr587-77a7201` built on the host from
`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
driving `podman run` directly, serving orcarouter
/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
decode slots, parsers AUTO (thinking → `reasoning_content` natively;
measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
and Dankclaw
runs two endpoints — **halogen container (frank-large, port 7770:
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
@@ -34,12 +33,12 @@ graph LR
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
B2[FreeToken PR587 ROCm, local image
podman run, frank-primary, port 7770]
C[Qwen3.8-Flash-Next FP8, no spec decode
TP=2, expert offload automatic, 3 x 200704 ctx,
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
expert-precise-memory arena 7,522 slots, /root/.cache mounted]
C[Qwen3.8-Flash-Next FP8
1x R9700, experts in pinned host RAM + GPU LRU 10%,
KV pool 299,968 tokens, 2 slots, parsers AUTO,
TP=2 untested on ROCm; second R9700 idle]
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -95,8 +94,22 @@ frank-small, port 7771]
target; payload on disk; owns the measured boot-pair table)**: the same
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
104.8 tok/s, 3,571 MiB true free
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
(root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
(branch rocm714_torch214) built on the host into the local image
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
curated --no-deps install per docs/install_amd.md), serving the SAME
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
(~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
frank-primary-vllm-shapeG-20261003
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
(rollback target; the vLLM trio + env lore live in the tag)**: the
tcclaviger RDNA4
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),