docs: frank-primary runs FreeToken (PR587 ROCm) — runbook, index, AGENTS, log
New runbook frank-primary-freetoken-pr587 (build laws, shape, measured cutover numbers, TP=2 non-boot signature, rollback); LLM/index.md intro + mermaid + active/retired entries; AGENTS.md fleet row + section + bench and model-config mentions + runbook index; log.md entry. Seam pins the measured KV pool 299968 with the multiple-of-64 guard. NOTE: intended PGP signature skipped — key 0x76EA601A11B178D7 has no secret key in this machine's keyring (fresh ~/.gnupg); the PGP identity lives elsewhere.
This commit is contained in:
Frank
@@ -0,0 +1,170 @@
|
||||
# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
|
||||
|
||||
**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
|
||||
|
||||
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
|
||||
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
|
||||
on the host into the local image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201` by
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served
|
||||
alias (`frank-primary`), same 131-shard payload
|
||||
(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
|
||||
git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore),
|
||||
runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
|
||||
## Why FreeToken
|
||||
|
||||
The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
|
||||
0.982 was a hard law (1.00 was **unbootable** — worker init raises
|
||||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
|
||||
total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
|
||||
still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
|
||||
fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
|
||||
GPU cache, sizes **KV as an explicit token pool**, and boots in one
|
||||
deterministic pass — no per-boot estimator to fight.
|
||||
|
||||
## The build (on dankserver, rootless podman as dank)
|
||||
|
||||
```bash
|
||||
cd /home/dank/serverAppConfigs
|
||||
podman build -t frank-primary-freetoken:pr587-77a7201 \
|
||||
-f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
|
||||
```
|
||||
|
||||
Laws baked into the Dockerfile (each one cost a failure):
|
||||
|
||||
- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
|
||||
the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
|
||||
rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
|
||||
rocm10.1.0 — an untested pair for this PR).
|
||||
- **`--no-build-isolation --no-deps -e .`** — a plain
|
||||
`pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
|
||||
build (the dependency set is authored for NVIDIA). Runtime deps are
|
||||
curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
|
||||
nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
|
||||
**flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
|
||||
FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
|
||||
are exactly those backends).
|
||||
- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
|
||||
versioned `libhiprtc.so.7` in
|
||||
`/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
|
||||
extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
|
||||
the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
|
||||
- FreeToken source is checked out at the **pinned PR head**, not the branch
|
||||
(a future push to `rocm714_torch214` cannot change the deployed engine).
|
||||
- The stale Docker Hub login (`docker login` creds at
|
||||
`~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
|
||||
with "invalid username/password" — `docker logout` fixes the pull. Hit
|
||||
2026-10-05 while pulling the base image.
|
||||
|
||||
## Measured at cutover (2026-10-05/06)
|
||||
|
||||
|
||||
- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
|
||||
pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
|
||||
**1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
|
||||
pinned expert/PLE slabs)**, 123 GB still available.
|
||||
- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
|
||||
1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
|
||||
FreeToken single-card is the deployed baseline.
|
||||
- **The single-card LRU ceiling**: the KV contract (harness worst case
|
||||
177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
|
||||
the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
|
||||
`ft ctl cache --moe` cannot raise it far on one card. The paths to a
|
||||
better decode rate are TP=2 (below) or a smaller pool (contract
|
||||
violation — the 177,356 floor is real traffic, not padding).
|
||||
- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
|
||||
zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
|
||||
allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
|
||||
resolve `cuda_runtime.h` from the hipified include path
|
||||
(`1 error generated when compiling for gfx1201`, ninja stopped). The PR
|
||||
author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
|
||||
edge, not a board or driver fault (radiance TP=2 allreduces fine on this
|
||||
pair since the 2026-09-20 swap). A future fix = patch the JIT include
|
||||
path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
|
||||
then.
|
||||
- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
|
||||
(finish stop); thinking split — `reasoning_content` non-empty AND
|
||||
`content` carries the answer (17×24 → "408"); native `tool_calls` with
|
||||
correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
|
||||
an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
|
||||
native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
|
||||
[text,image], sampling defaults from generation_config.json
|
||||
(temp 1.0 / top_p 0.95 / top_k 20).
|
||||
- **Thinking counts against max_tokens** (same as the vLLM era): a
|
||||
`max_tokens: 10` probe returns EMPTY content with the budget spent in the
|
||||
think block. Clients must budget for thinking or disable it.
|
||||
- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
|
||||
from the llm-fleet job; FreeToken has no /metrics endpoint). The target
|
||||
will read down until the scrape config is repointed or a metrics shim
|
||||
lands — same class as the retired HTML-probe targets
|
||||
(see [[Observability]]).
|
||||
|
||||
|
||||
## The shape
|
||||
|
||||
| Knob | Value | Why |
|
||||
|---|---|---|
|
||||
| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
|
||||
| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
|
||||
| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
|
||||
| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
|
||||
| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
|
||||
| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
|
||||
|
||||
KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
|
||||
2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
|
||||
not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
|
||||
case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
|
||||
native 262,144-token conversation + full budget. `models.yml`
|
||||
`contextWindow: 200704` stays (compacts early, never 400s).
|
||||
|
||||
## RAM math (why the root unit + memlock)
|
||||
|
||||
246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
|
||||
PLE ngram table in host RAM → the unit carries
|
||||
`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
|
||||
a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
|
||||
container's ~170 GiB MUST be down first; `systemctl stop frank-primary`
|
||||
frees it (measured: 207 GB free right after the stop).
|
||||
|
||||
## Resize live
|
||||
|
||||
```bash
|
||||
ft ctl cache --kv <tokens> # grows the KV pool LIVE (shrink needs restart)
|
||||
ft ctl cache --moe <rate> # same for the expert LRU
|
||||
ft ctl stats # pool occupancy, VRAM, throughput
|
||||
```
|
||||
|
||||
## Verification (the fleet's real-completion law)
|
||||
|
||||
1. `/health` 200 — necessary, not sufficient.
|
||||
2. Plain completion returns the asked-for string.
|
||||
3. Thinking request: `reasoning_content` non-empty AND `content` carries the
|
||||
answer (the models.yml `reasoningContentField: reasoning_content` flip
|
||||
rides on this).
|
||||
4. Vision probe: a solid-color PNG through an `image_url` part answers the
|
||||
exact color (the legacy top-level `images` field trap does not exist
|
||||
here — FreeToken is OpenAI-native).
|
||||
5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
|
||||
2026-09-26 stray-port false green).
|
||||
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
git checkout frank-primary-vllm-shapeG-20261003 -- \
|
||||
scripts/frank-primary.env scripts/serve-frank-primary.sh \
|
||||
scripts/frank-primary.service
|
||||
./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \
|
||||
&& git push && ./scripts/deploy.sh dankserver --service frank-primary
|
||||
```
|
||||
|
||||
The tcclaviger image and the FreeToken image coexist in dank's rootless
|
||||
storage; the deploy re-pulls the pinned digest if absent.
|
||||
|
||||
## Index
|
||||
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
|
||||
(rollback target; its 362-line env lore lives in the tag)
|
||||
- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
|
||||
- [[LLM/index]] — fleet index
|
||||
@@ -1,18 +1,17 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-03):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **shape E since
|
||||
2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
|
||||
conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
|
||||
decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
|
||||
(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
|
||||
(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
|
||||
mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
|
||||
`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
|
||||
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-05):
|
||||
**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
|
||||
image `frank-primary-freetoken:pr587-77a7201` built on the host from
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
|
||||
`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
|
||||
driving `podman run` directly, serving orcarouter
|
||||
/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
|
||||
pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
|
||||
`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
|
||||
decode slots, parsers AUTO (thinking → `reasoning_content` natively;
|
||||
measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
|
||||
runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw
|
||||
runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
@@ -34,12 +33,12 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
B2[FreeToken PR587 ROCm, local image
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8, no spec decode
|
||||
TP=2, expert offload automatic, 3 x 200704 ctx,
|
||||
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
|
||||
expert-precise-memory arena 7,522 slots, /root/.cache mounted]
|
||||
C[Qwen3.8-Flash-Next FP8
|
||||
1x R9700, experts in pinned host RAM + GPU LRU 10%,
|
||||
KV pool 299,968 tokens, 2 slots, parsers AUTO,
|
||||
TP=2 untested on ROCm; second R9700 idle]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -95,8 +94,22 @@ frank-small, port 7771]
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
|
||||
(branch rocm714_torch214) built on the host into the local image
|
||||
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||||
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
|
||||
(~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
|
||||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
|
||||
frank-primary-vllm-shapeG-20261003
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||||
|
||||
@@ -1,5 +1,31 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-10-05
|
||||
- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
|
||||
dankserver.** Built FreeToken at PR head `77a7201` on the host into the
|
||||
local image `frank-primary-freetoken:pr587-77a7201`
|
||||
(digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws
|
||||
bought with failures: the pip-ROCm wheel needs a `libhiprtc.so` dev
|
||||
symlink for the PR's `_cpu_moe` link, and a stale `docker login` at
|
||||
`~/.docker/config.json` poisons podman's ANONYMOUS pulls). Trio rewritten
|
||||
through git (env/serve/unit + seam + deployer): ONE R9700, experts in
|
||||
pinned host RAM behind a 0.10 GPU LRU, PLE 47.7 GiB pinned, KV pool
|
||||
299,968 tokens (multiple-of-64 — the `qsa_sparse` auto backend overrides
|
||||
page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`; models.yml flipped accordingly). Container renamed
|
||||
to the unit name (Frank's call; `frank-primary-qwen` → legacy list);
|
||||
deployer image gate → `podman image exists` (local builds carry no
|
||||
RepoDigests) and health gate reads the BODY — FreeToken 200s with
|
||||
`{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
|
||||
(vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK
|
||||
completion, reasoning split (17×24 → "408"), native `tool_calls`, vision
|
||||
"Red" probe, boot 2.3 min cold / 92 s warm, decode 17.1 tok/s
|
||||
single-stream (vLLM was ~30 on two cards; the KV contract caps the
|
||||
single-card LRU; TP=2 exists but the pynccl JIT can't compile for
|
||||
gfx1201 — probed, signature in the runbook). `/metrics` is a 404 — the
|
||||
Dankclaw Prometheus llm-fleet scrape of :7770 reads down until
|
||||
repointed. Rollback: tag `frank-primary-vllm-shapeG-20261003`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
## 2026-10-05
|
||||
- **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright
|
||||
project.** The `11-admin` walk included the six `*arr`/qBittorrent admin
|
||||
|
||||
Reference in New Issue
Block a user