docs: frank-primary runs FreeToken (PR587 ROCm) — runbook, index, AGENTS, log

New runbook frank-primary-freetoken-pr587 (build laws, shape, measured
cutover numbers, TP=2 non-boot signature, rollback); LLM/index.md intro +
mermaid + active/retired entries; AGENTS.md fleet row + section + bench
and model-config mentions + runbook index; log.md entry. Seam pins the
measured KV pool 299968 with the multiple-of-64 guard.

NOTE: intended PGP signature skipped — key 0x76EA601A11B178D7 has no
secret key in this machine's keyring (fresh ~/.gnupg); the PGP identity
lives elsewhere.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
+| Dankserver | `frank-primary` | **FreeToken PR #587** (branch `rocm714_torch214`, head `77a7201`; local image `frank-primary-freetoken:pr587-77a7201` built on the host from `scripts/docker/frank-primary-freetoken.Dockerfile` — digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards, modelopt block-FP8, loaded straight from safetensors), **ONE R9700 (second card idle)**, all experts in **pinned host RAM** (~120 GiB) behind a GPU LRU (`--moe-cache-rate 0.10`), PLE table 47.7 GiB pinned (root unit + `LimitMEMLOCK=infinity`), **KV pool `--num-tokens 299968`** (24 KiB/token, multiple-of-64 law of the `qsa_sparse` auto backend), 2 slots, `--max-output-tokens 36864`, **parsers AUTO** (thinking → `reasoning_content`, tools qwen3.8 dialect), vision tower host-streamed. Measured: boot ~2.3 min cold / 92 s warm, decode **17.1 tok/s** single-stream; **TP=2 does not boot on ROCm** (pynccl JIT); `/metrics` 404 (Prometheus scrape reads down). Deploy gate reads the `/health` BODY (`status:"ok"`) — FreeToken 200s while loading. Rollback tag `frank-primary-vllm-shapeG-20261003` |7770|
 | Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
 | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
 
@@ -260,7 +260,7 @@ merely wedged kernel usually still answers ARP.
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
    - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
      `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
+   - Dankserver `frank-primary` (root unit, FreeToken on one R9700):
      `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
    NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -295,8 +295,9 @@ a role pointing at a provider the file does not define fails at call time. As
 of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
 image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
 ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
-`frank-primary` takes them through a BF16 mmproj loaded explicitly (this build never
+`frank-primary` takes them natively (FreeToken builds the qwen3.8 vision
-autoloads projectors), and `frank-small` through
+tower and streams its block weights from host RAM two at a time by default),
+and `frank-small` through
 `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
 GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
 **200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the
@@ -443,122 +444,63 @@ pairs with Q5 only; SGLang rollback in
 `/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
 `~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).
 
-### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
+### frank-primary — FreeToken (PR #587 ROCm), Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
 
-The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
+The **FreeToken engine at PR #587 head `77a7201`** (branch
-digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
+`rocm714_torch214`, built on the host into the local image
-(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
+`localhost/frank-primary-freetoken:pr587-77a7201` by
-**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
+`scripts/docker/frank-primary-freetoken.Dockerfile` — the PR's own
-2026-10-03 the **shape-G** geometry (shape E + MNBT 2048 + the CUDA-graph plan
+docs/install_amd.md stack: digest-pinned
-term at the measured floor): `--max-model-len 200704` × 3 slots with
+`rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0` base =
-`--kv-cache-memory-bytes 3,166,500,000`/rank (~400,700 KV tokens — the engine
+torch 2.11.0+rocm7.14/triton 3.7, `--no-build-isolation --no-deps -e .`,
-prices this pool at 7,891.8 B/token here), util **0.982** (the law — see the
+curated portable deps, `flashlib==0.3.0 --no-deps`, and a `libhiprtc.so`
-unbootable-1.0 lesson below), `--expert-precise-memory`,
+dev-symlink the pip-ROCm wheel lacks) serving
-`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`,
+**`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards,
-`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries
+standard modelopt block-FP8 — 75,264 `weight_scale_inv` tensors, no
-`contextWindow: 200704` for this endpoint — the served window, not the model's
+`input_scale`, loaded directly from HF safetensors, no FTW conversion) on
-native 262144, so the harness compacts before vLLM answers a longer prompt
+**ONE R9700 (gfx1201)** — the second card sits idle. Plain `podman run`
-with a 400.
+through `scripts/serve-frank-primary.sh` under the root systemd unit
+(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
+container name is the unit name (`frank-primary`; the vLLM-era
+`frank-primary-qwen` is in the legacy cleanup list). Cutover 2026-10-05;
+rollback tag `frank-primary-vllm-shapeG-20261003` (the tcclaviger vLLM trio
+and its 362-line env lore live in that tag; the tcclaviger image stays in
+dank's rootless storage).
 
-**`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by
+**Shape**: all expert weights in **pinned host RAM** (~120 GiB) behind a
-single-variable boot probe against the then-deployed shape: SEQS 2 fails at
+GPU LRU (`--moe-cache-rate 0.10`, ~12 GiB — the single-card ceiling given
-BOTH MNBT 1024 and 2048 with `torch.fx.experimental.symbolic_shapes.Constraint
+the KV contract), the ~47.7 GiB PLE ngram table pinned in host RAM
-ViolationError: Constraints violated (L['query_start_loc'].size()[0]) …
+(→ root unit + memlock law), **KV pool `--num-tokens 299968`**
-specialized it to be a constant (3)`; SEQS 3 passes at both. `model_runner.py`
+(24 KiB/token f16 across the 12 full-attention layers ≈ 7.0 GiB; **must be
-sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
+a multiple of 64** — the auto attention backend is `qsa_sparse`, which
-`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
+overrides page size to 64; 300,000 was refused at first boot), 2 decode
-`SEQS + 1` for any `MNBT ≥ SEQS` — MNBT only moves the compile RANGE
+slots, `--max-output-tokens 36864` (the harness maxTokens; requests that
-((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
+omit one get this default). **Parsers are AUTO**: thinking lands in
-graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
+`reasoning_content` natively, tool calls in the qwen3.8 dialect — models.yml
+flips `reasoningContentField` to `reasoning_content` on the harness side.
+Vision works (models.yml keeps `input: [text, image]`); the vision tower's
+blocks stream from host RAM two at a time by default.
 
-**The expert arena has THREE reachable levers: `max_model_len`, the overhead
+**Measured at cutover**: boot ~2.3 min cold-ish / 92 s warm → /health
-term, and the CUDA-graph estimate.**
+status ok; decode **17.1 tok/s** single-stream temp 0 (the vLLM incumbent
-The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
+measured ~30 tok/s on TWO cards; the KV contract caps the single-card LRU,
-graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
+and **TP=2 exists but does not boot on ROCm** — the TP allreduce JIT
-`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
+(`pynccl.cu`) can't resolve `cuda_runtime.h` for gfx1201; probed
-200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
+2026-10-06, signature in the runbook). `/metrics` is a 404 — the Dankclaw
-0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
+Prometheus llm-fleet scrape of :7770 reads down until repointed.
-headroom (the shape it replaced died with `0 bytes free` on a real 995-token
+`ft ctl cache --kv/--moe` resizes pools live (grow only; shrink needs a
-request). The second lever is `--expert-precise-memory`, **ON since
+restart); `ft ctl stats` reports occupancy.
-2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted +
-compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB
-difference lands in the arena — measured plan lines at 200704 × 3: overhead
-5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region
-13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9
-deliberately: because the pool is not an arena lever, cutting it would cost
-concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots.
-The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
-headroom collapses the offload hit rate.
 
-**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).**
+**Deploy gate law (changed for FreeToken)**: `/health` returns **200 with
-`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the
+`{"status":"loading"}`** while the engine loads — the FastAPI front end
-planner charges `budget = total × util` against terms that overstate reality:
+binds before the scheduler is up, so a bare `curl` false-greens (measured:
-the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while
+"healthy after 6 s" mid-page-in). The deployer loops until the body says
-the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25
+`"status":"ok"`; the MainPID stray-port guard is unchanged. The image gate
-GiB) is far above the idle-measured figure. So the visible free space is the
+is `podman image exists` (a local build carries no RepoDigests — the old
-estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena
+digest grep refused exactly this shape; pinning lives in the Dockerfile).
-lever (each +0.01 = +0.32 GiB) but **0.982 is the law, and 1.00 is UNBOOTABLE
-on this fork — measured 2026-10-03**: the worker init demands
-`free > budget`, and at `le=1` budget == device total, so both ranks raise
-`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` before a
-single weight loads, twice (2-restart crash loop, zero payload progress). The
-CLI field accepting `le=1` is the trap; Frank's 0.982 is where it always
-ships. What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
-`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the
-analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's
-own post-capture line reported 0.31 GiB, and a second boot (with the plan
-already charging 0.32 GiB) reported **-0.06 GiB**, so the 256 MiB base pool is
-pure over-charge: shape G charges `0 base + 12 MiB/graph` = 0.07 GiB, which
-still prices per graph for a future 18-graph spec-decode shape (0.21 GiB).
-Verify every deploy by comparing the `Estimated CUDA graph memory`
-line with `Graph capturing finished … took X GiB`; if capture exceeds the
-estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO
-arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)`
-anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the
-fitted figure was fitted AT 4096.
 
-**A container's compile cache does NOT survive `podman rm -f`, and on this
+Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
-endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
+Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
-`/root/.cache` at all** (fresh container: `No such file or directory`); after a
-boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus
-`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine
-prints `torch.compile took 181.20 s in total` (173.19 s of it one graph,
-`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and
-reboot re-pays it. The serve script now mounts
-`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT /
-torch_compile tree (`VLLM_CACHE_ROOT`, whose default is
-`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and
-nothing in the image is hidden because the image has no such directory. The
-compile RANGE is part of the cache key, so an MNBT change costs one
-recompilation, not a permanent tax. Same class of fix as the recipe's
-`/cache/{triton,vllm,inductor}` volumes. **Proven live 2026-10-02 on
-consecutive boots:** cold `Compiling a graph for compile range (1, 1024) takes
-179.24 s` / `torch.compile took 188.26 s`; warm `Directly load the compiled
-graph(s) for compile range (1, 1024) from the cache, took 2.143 s` /
-`torch.compile took 3.97 s in total` / `init engine (profile, create kv cache,
-warmup model) took 41.13 s (compilation: 3.98 s)`. Time to `/health` is
-dominated by the 172.8 GiB payload load, so the visible win is the ~184 s of
-inductor/AOT build, not a proportional boot cut.
-
-Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
-**37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;
-2-way concurrency sums BELOW parity (25.49 vs 26.8 single) — slots are a
-capacity feature, not a throughput one. V-only q8 KV is NOT possible via
-config: stock `--kv-cache-dtype` is K+V together; a V-only quant needs a fork
-patch.
-
-Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
-frank-primary-qwen`. The serve script drives `podman run` directly, gates
-the digest-pinned image, the payload's `config.json` and its exact
-131-shard count and the GPU device nodes, clears stale container
-identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
-same, waits out :7770, relinks/enables/restarts the unit and health-gates
-with the MainPID check. Boot cost ~7 min warm (the 172.8 GiB payload load; the
-~184 s inductor/AOT build inside it is what the mounted cache now skips — see
-the compile-cache lesson below); health returns 503 while loading.
-
-Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
 Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
 
 ## Key Lessons Learned
@@ -1005,7 +947,7 @@ Active:
 |---------|---------|
 | frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
 | frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
-| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| frank-primary (FreeToken PR #587 ROCm, Qwen3.8-Flash-Next FP8, 1×R9700, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 
@@ -1014,6 +956,7 @@ Retired (rollback targets and history):
 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
 | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
+| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
 | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
 | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |

serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md

@@ -0,0 +1,170 @@
+# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
+
+**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
+
+The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
+`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
+on the host into the local image
+`localhost/frank-primary-freetoken:pr587-77a7201` by
+`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served
+alias (`frank-primary`), same 131-shard payload
+(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
+git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore),
+runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
+
+## Why FreeToken
+
+The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
+0.982 was a hard law (1.00 was **unbootable** — worker init raises
+`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
+total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
+still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
+fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
+GPU cache, sizes **KV as an explicit token pool**, and boots in one
+deterministic pass — no per-boot estimator to fight.
+
+## The build (on dankserver, rootless podman as dank)
+
+```bash
+cd /home/dank/serverAppConfigs
+podman build -t frank-primary-freetoken:pr587-77a7201 \
+  -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
+```
+
+Laws baked into the Dockerfile (each one cost a failure):
+
+- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
+  the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
+  rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
+  rocm10.1.0 — an untested pair for this PR).
+- **`--no-build-isolation --no-deps -e .`** — a plain
+  `pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
+  build (the dependency set is authored for NVIDIA). Runtime deps are
+  curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
+  nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
+  **flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
+  FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
+  are exactly those backends).
+- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
+  versioned `libhiprtc.so.7` in
+  `/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
+  extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
+  the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
+- FreeToken source is checked out at the **pinned PR head**, not the branch
+  (a future push to `rocm714_torch214` cannot change the deployed engine).
+- The stale Docker Hub login (`docker login` creds at
+  `~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
+  with "invalid username/password" — `docker logout` fixes the pull. Hit
+  2026-10-05 while pulling the base image.
+
+## Measured at cutover (2026-10-05/06)
+
+
+- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
+  pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
+  **1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
+  pinned expert/PLE slabs)**, 123 GB still available.
+- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
+  1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
+  FreeToken single-card is the deployed baseline.
+- **The single-card LRU ceiling**: the KV contract (harness worst case
+  177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
+  the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
+  `ft ctl cache --moe` cannot raise it far on one card. The paths to a
+  better decode rate are TP=2 (below) or a smaller pool (contract
+  violation — the 177,356 floor is real traffic, not padding).
+- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
+  zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
+  allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
+  resolve `cuda_runtime.h` from the hipified include path
+  (`1 error generated when compiling for gfx1201`, ninja stopped). The PR
+  author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
+  edge, not a board or driver fault (radiance TP=2 allreduces fine on this
+  pair since the 2026-09-20 swap). A future fix = patch the JIT include
+  path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
+  then.
+- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
+  (finish stop); thinking split — `reasoning_content` non-empty AND
+  `content` carries the answer (17×24 → "408"); native `tool_calls` with
+  correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
+  an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
+  native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
+  [text,image], sampling defaults from generation_config.json
+  (temp 1.0 / top_p 0.95 / top_k 20).
+- **Thinking counts against max_tokens** (same as the vLLM era): a
+  `max_tokens: 10` probe returns EMPTY content with the budget spent in the
+  think block. Clients must budget for thinking or disable it.
+- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
+  from the llm-fleet job; FreeToken has no /metrics endpoint). The target
+  will read down until the scrape config is repointed or a metrics shim
+  lands — same class as the retired HTML-probe targets
+  (see [[Observability]]).
+
+
+## The shape
+
+| Knob | Value | Why |
+|---|---|---|
+| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
+| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
+| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
+| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
+| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
+| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
+
+KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
+2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
+not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
+case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
+native 262,144-token conversation + full budget. `models.yml`
+`contextWindow: 200704` stays (compacts early, never 400s).
+
+## RAM math (why the root unit + memlock)
+
+246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
+PLE ngram table in host RAM → the unit carries
+`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
+a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
+container's ~170 GiB MUST be down first; `systemctl stop frank-primary`
+frees it (measured: 207 GB free right after the stop).
+
+## Resize live
+
+```bash
+ft ctl cache --kv <tokens>   # grows the KV pool LIVE (shrink needs restart)
+ft ctl cache --moe <rate>    # same for the expert LRU
+ft ctl stats                 # pool occupancy, VRAM, throughput
+```
+
+## Verification (the fleet's real-completion law)
+
+1. `/health` 200 — necessary, not sufficient.
+2. Plain completion returns the asked-for string.
+3. Thinking request: `reasoning_content` non-empty AND `content` carries the
+   answer (the models.yml `reasoningContentField: reasoning_content` flip
+   rides on this).
+4. Vision probe: a solid-color PNG through an `image_url` part answers the
+   exact color (the legacy top-level `images` field trap does not exist
+   here — FreeToken is OpenAI-native).
+5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
+   2026-09-26 stray-port false green).
+
+## Rollback
+
+```bash
+git checkout frank-primary-vllm-shapeG-20261003 -- \
+  scripts/frank-primary.env scripts/serve-frank-primary.sh \
+  scripts/frank-primary.service
+./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \
+  && git push && ./scripts/deploy.sh dankserver --service frank-primary
+```
+
+The tcclaviger image and the FreeToken image coexist in dank's rootless
+storage; the deploy re-pulls the pinned digest if absent.
+
+## Index
+
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
+  (rollback target; its 362-line env lore lives in the tag)
+- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
+- [[LLM/index]] — fleet index

serverAppConfigs-docs/LLM/index.md

@@ -1,18 +1,17 @@
 # LLM Fleet — Knowledge Base
 
-Documentation for the LLM inference fleet on the two hosts (2026-10-03):
+Documentation for the LLM inference fleet on the two hosts (2026-10-05):
-**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
+**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
-digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
+image `frank-primary-freetoken:pr587-77a7201` built on the host from
-orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
+`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
-on :7770, TP=2 with automatic tiered-expert offload, **shape E since
+`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
-2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
+driving `podman run` directly, serving orcarouter
-conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
+/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
-decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
+pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
-(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
+`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
-(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
+decode slots, parsers AUTO (thinking → `reasoning_content` natively;
-mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
+measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
-`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
+runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
-SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
 and Dankclaw
 runs two endpoints — **halogen container (frank-large, port 7770:
 ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
@@ -34,12 +33,12 @@ graph LR
     subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
         A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[tcclaviger RDNA4 vLLM fork, digest-pinned
+        B2[FreeToken PR587 ROCm, local image
 podman run, frank-primary, port 7770]
-        C[Qwen3.8-Flash-Next FP8, no spec decode
+        C[Qwen3.8-Flash-Next FP8
-TP=2, expert offload automatic, 3 x 200704 ctx,
+1x R9700, experts in pinned host RAM + GPU LRU 10%,
-fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
+KV pool 299,968 tokens, 2 slots, parsers AUTO,
-expert-precise-memory arena 7,522 slots, /root/.cache mounted]
+TP=2 untested on ROCm; second R9700 idle]
     end
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -95,8 +94,22 @@ frank-small, port 7771]
   target; payload on disk; owns the measured boot-pair table)**: the same
   runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
   104.8 tok/s, 3,571 MiB true free
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
+- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
-  2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
+  (root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
+  (branch rocm714_torch214) built on the host into the local image
+  `frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
+  rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
+  curated --no-deps install per docs/install_amd.md), serving the SAME
+  orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
+  (~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
+  (multiple-of-64 law), 2 slots, parsers AUTO (thinking →
+  `reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
+  single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
+  not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
+  frank-primary-vllm-shapeG-20261003
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
+  (rollback target; the vLLM trio + env lore live in the tag)**: the
+  tcclaviger RDNA4
   vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
   R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
   slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),

serverAppConfigs-docs/log.md

@@ -1,5 +1,31 @@
 # Bundle Change Log
 
+## 2026-10-05
+- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
+  dankserver.** Built FreeToken at PR head `77a7201` on the host into the
+  local image `frank-primary-freetoken:pr587-77a7201`
+  (digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws
+  bought with failures: the pip-ROCm wheel needs a `libhiprtc.so` dev
+  symlink for the PR's `_cpu_moe` link, and a stale `docker login` at
+  `~/.docker/config.json` poisons podman's ANONYMOUS pulls). Trio rewritten
+  through git (env/serve/unit + seam + deployer): ONE R9700, experts in
+  pinned host RAM behind a 0.10 GPU LRU, PLE 47.7 GiB pinned, KV pool
+  299,968 tokens (multiple-of-64 — the `qsa_sparse` auto backend overrides
+  page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking →
+  `reasoning_content`; models.yml flipped accordingly). Container renamed
+  to the unit name (Frank's call; `frank-primary-qwen` → legacy list);
+  deployer image gate → `podman image exists` (local builds carry no
+  RepoDigests) and health gate reads the BODY — FreeToken 200s with
+  `{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
+  (vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK
+  completion, reasoning split (17×24 → "408"), native `tool_calls`, vision
+  "Red" probe, boot 2.3 min cold / 92 s warm, decode 17.1 tok/s
+  single-stream (vLLM was ~30 on two cards; the KV contract caps the
+  single-card LRU; TP=2 exists but the pynccl JIT can't compile for
+  gfx1201 — probed, signature in the runbook). `/metrics` is a 404 — the
+  Dankclaw Prometheus llm-fleet scrape of :7770 reads down until
+  repointed. Rollback: tag `frank-primary-vllm-shapeG-20261003`.
+  Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
 ## 2026-10-05
 - **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright
   project.** The `11-admin` walk included the six `*arr`/qBittorrent admin

tests/frank_primary_unit.bats

@@ -99,13 +99,17 @@ DOCKERFILE="$REPO_ROOT/scripts/docker/frank-primary-freetoken.Dockerfile"
 @test "the FreeToken engine shape is in the env file" {
     # KV pool: TOKENS, not bytes. 24 KiB/token (12 full-attention layers x
     # 2 KV heads x 256 head_dim x K+V x f16) — the 36 GDN layers use fixed
-    # per-seq state, not per-token KV. 300,000 tokens = 7.2 GiB and clears
+    # per-seq state, not per-token KV. 299,968 tokens = 7.0 GiB and clears
     # the harness worst case (140,492-token prompt + 36,864 max_tokens =
     # 177,356); one NATIVE 262,144-token conversation + full budget also
-    # fits. The arithmetic guard keeps future resizes honest.
+    # fits. The value must stay a MULTIPLE OF 64: the auto attention
-    grep -q '^FRANK_PRIMARY_FT_NUM_TOKENS=300000$' "$ENV_FILE"
+    # backend is qsa_sparse, which overrides page size to 64 (measured
+    # first boot — 300,000 was refused). The arithmetic guard keeps future
+    # resizes honest.
+    grep -q '^FRANK_PRIMARY_FT_NUM_TOKENS=299968$' "$ENV_FILE"
     nt=$(awk -F= '/^FRANK_PRIMARY_FT_NUM_TOKENS=/{print $2}' "$ENV_FILE")
     [ "$nt" -ge 177356 ]
+    [ $((nt % 64)) -eq 0 ]
     # First-boot conservative shape: 2 decode slots, 10% expert LRU
     # (~12 GiB of the ~120 GiB fp8 expert set), output ceiling = the
     # harness maxTokens, single R9700 at index 0.