Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
175 lines
9.1 KiB
Markdown
175 lines
9.1 KiB
Markdown
# frank-archon — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
|
||
|
||
**Status: RETIRED 2026-10-05 → (superseded the same day by the radiance
|
||
cutover — see [[frank-archon-radiance-qwen38-native-fp8]]; kept as the
|
||
rollback runbook: the image + orcarouter payload stay on disk until the
|
||
window closes)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
|
||
**GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
|
||
|
||
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
|
||
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
|
||
on the host into the local image
|
||
`localhost/frank-archon-freetoken:pr587-77a7201` by
|
||
`scripts/docker/frank-archon-freetoken.Dockerfile`. Same port, same served
|
||
alias (`frank-archon`), same 131-shard payload
|
||
(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
|
||
git tag `frank-archon-vllm-shapeG-20261003` (the vLLM trio + its env lore),
|
||
runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||
|
||
## Why FreeToken
|
||
|
||
The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
|
||
0.982 was a hard law (1.00 was **unbootable** — worker init raises
|
||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
|
||
total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
|
||
still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
|
||
fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
|
||
GPU cache, sizes **KV as an explicit token pool**, and boots in one
|
||
deterministic pass — no per-boot estimator to fight.
|
||
|
||
## The build (on dankserver, rootless podman as dank)
|
||
|
||
```bash
|
||
cd /home/dank/serverAppConfigs
|
||
podman build -t frank-archon-freetoken:pr587-77a7201 \
|
||
-f scripts/docker/frank-archon-freetoken.Dockerfile scripts/docker/
|
||
```
|
||
|
||
Laws baked into the Dockerfile (each one cost a failure):
|
||
|
||
- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
|
||
the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
|
||
rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
|
||
rocm10.1.0 — an untested pair for this PR).
|
||
- **`--no-build-isolation --no-deps -e .`** — a plain
|
||
`pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
|
||
build (the dependency set is authored for NVIDIA). Runtime deps are
|
||
curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
|
||
nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
|
||
**flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
|
||
FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
|
||
are exactly those backends).
|
||
- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
|
||
versioned `libhiprtc.so.7` in
|
||
`/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
|
||
extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
|
||
the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
|
||
- FreeToken source is checked out at the **pinned PR head**, not the branch
|
||
(a future push to `rocm714_torch214` cannot change the deployed engine).
|
||
- The stale Docker Hub login (`docker login` creds at
|
||
`~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
|
||
with "invalid username/password" — `docker logout` fixes the pull. Hit
|
||
2026-10-05 while pulling the base image.
|
||
|
||
## Measured at cutover (2026-10-05/06)
|
||
|
||
|
||
- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
|
||
pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
|
||
**1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
|
||
pinned expert/PLE slabs)**, 123 GB still available.
|
||
- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
|
||
1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
|
||
FreeToken single-card is the deployed baseline.
|
||
- **The single-card LRU ceiling**: the KV contract (harness worst case
|
||
177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
|
||
the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
|
||
`ft ctl cache --moe` cannot raise it far on one card. The paths to a
|
||
better decode rate are TP=2 (below) or a smaller pool (contract
|
||
violation — the 177,356 floor is real traffic, not padding).
|
||
- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
|
||
zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
|
||
allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
|
||
resolve `cuda_runtime.h` from the hipified include path
|
||
(`1 error generated when compiling for gfx1201`, ninja stopped). The PR
|
||
author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
|
||
edge, not a board or driver fault (radiance TP=2 allreduces fine on this
|
||
pair since the 2026-09-20 swap). A future fix = patch the JIT include
|
||
path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
|
||
then.
|
||
- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
|
||
(finish stop); thinking split — `reasoning_content` non-empty AND
|
||
`content` carries the answer (17×24 → "408"); native `tool_calls` with
|
||
correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
|
||
an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
|
||
native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
|
||
[text,image], sampling defaults from generation_config.json
|
||
(temp 1.0 / top_p 0.95 / top_k 20).
|
||
- **Thinking counts against max_tokens** (same as the vLLM era): a
|
||
`max_tokens: 10` probe returns EMPTY content with the budget spent in the
|
||
think block. Clients must budget for thinking or disable it.
|
||
- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
|
||
from the llm-fleet job; FreeToken has no /metrics endpoint). The target
|
||
will read down until the scrape config is repointed or a metrics shim
|
||
lands — same class as the retired HTML-probe targets
|
||
(see [[Observability]]).
|
||
|
||
|
||
## The shape
|
||
|
||
| Knob | Value | Why |
|
||
|---|---|---|
|
||
| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
|
||
| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
|
||
| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
|
||
| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
|
||
| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
|
||
| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
|
||
|
||
KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
|
||
2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
|
||
not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
|
||
case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
|
||
native 262,144-token conversation + full budget. `models.yml`
|
||
`contextWindow: 200704` stays (compacts early, never 400s).
|
||
|
||
## RAM math (why the root unit + memlock)
|
||
|
||
246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
|
||
PLE ngram table in host RAM → the unit carries
|
||
`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
|
||
a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
|
||
container's ~170 GiB MUST be down first; `systemctl stop frank-archon`
|
||
frees it (measured: 207 GB free right after the stop).
|
||
|
||
## Resize live
|
||
|
||
```bash
|
||
ft ctl cache --kv <tokens> # grows the KV pool LIVE (shrink needs restart)
|
||
ft ctl cache --moe <rate> # same for the expert LRU
|
||
ft ctl stats # pool occupancy, VRAM, throughput
|
||
```
|
||
|
||
## Verification (the fleet's real-completion law)
|
||
|
||
1. `/health` 200 — necessary, not sufficient.
|
||
2. Plain completion returns the asked-for string.
|
||
3. Thinking request: `reasoning_content` non-empty AND `content` carries the
|
||
answer (the models.yml `reasoningContentField: reasoning_content` flip
|
||
rides on this).
|
||
4. Vision probe: a solid-color PNG through an `image_url` part answers the
|
||
exact color (the legacy top-level `images` field trap does not exist
|
||
here — FreeToken is OpenAI-native).
|
||
5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
|
||
2026-09-26 stray-port false green).
|
||
|
||
## Rollback
|
||
|
||
```bash
|
||
git checkout frank-archon-vllm-shapeG-20261003 -- \
|
||
scripts/frank-archon.env scripts/serve-frank-archon.sh \
|
||
scripts/frank-archon.service
|
||
./tests/run.sh && git commit -m "frank-archon: rollback to vLLM shape G" \
|
||
&& git push && ./scripts/deploy.sh dankserver --service frank-archon
|
||
```
|
||
|
||
The tcclaviger image and the FreeToken image coexist in dank's rootless
|
||
storage; the deploy re-pulls the pinned digest if absent.
|
||
|
||
## Index
|
||
|
||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
|
||
(rollback target; its 362-line env lore lives in the tag)
|
||
- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
|
||
- [[LLM/index]] — fleet index
|