serverAppConfigs/serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md

frank-archon — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)

Status: RETIRED 2026-10-05 → (superseded the same day by the radiance cutover — see frank-archon-radiance-qwen38-native-fp8, itself retired 2026-10-08; the payload live today (2026-10-18) is the publisher iq4r container, frank-archon-radiance-qwen38-iq4r). Kept as the rollback runbook: the image is still in dank's rootless storage, the orcarouter payload is not on disk any more) · Host: Dankserver (192.168.0.235) · Port: 7770 · GPU: 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle

The tcclaviger RDNA4 vLLM fork was replaced by FreeToken at PR #587 head 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214), built on the host into the local image localhost/frank-primary-freetoken:pr587-77a7201 (that is the tag podman reports today — the 2026-10-10 rename never retagged it) by scripts/docker/frank-primary-freetoken.Dockerfile, which survives only in the rollback tag: scripts/docker/ is not in the tree today. Same port, same served alias (the wire id was frank-primary until the 2026-10-09 rename), same 131-shard payload (/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8, since deleted from the host). Rollback: git tag frank-archon-vllm-shapeG-20261003 (the vLLM trio + its env lore), runbook dankserver-qwen38-fp8-expert-offload-tuning.

Why FreeToken

The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util 0.982 was a hard law (1.00 was unbootable — worker init raises ValueError: Free memory on device cuda:N (31.29/31.86 GiB) when budget == total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic still degraded ~40k tokens in (the GDN state-fault class, never fixed in the fork). FreeToken keeps all expert weights in pinned host RAM with an LRU GPU cache, sizes KV as an explicit token pool, and boots in one deterministic pass — no per-boot estimator to fight.

The build (on dankserver, rootless podman as dank)

git checkout frank-archon-freetoken-pr587-20261005   # scripts/docker/ exists only in the tag
cd /home/dank/serverAppConfigs
podman build -t frank-primary-freetoken:pr587-77a7201 \
  -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/

(The names above are the ones the tag actually carries — scripts/docker/frank-primary-freetoken.Dockerfile, per git ls-tree frank-archon-freetoken-pr587-20261005. The 2026-10-10 rename swept the docs, not the tag or the image in storage.)

Laws baked into the Dockerfile (each one cost a failure):

• Base digest-pinned: rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1… — the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are rocm10.1.0 — an untested pair for this PR).
• --no-build-isolation --no-deps -e . — a plain pip install -e . on ROCm can REPLACE the image's ROCm torch with a CUDA build (the dependency set is authored for NVIDIA). Runtime deps are curated explicitly; flashlib==0.3.0 --no-deps (its numba / nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT; flashinfer / sgl_kernel intentionally absent (no ROCm builds; FreeToken falls back to its Triton kernels — the PR's 12 pytest failures are exactly those backends).
• libhiprtc.so dev symlink: the pip-ROCm torch ships only the versioned libhiprtc.so.7 in /opt/venv/.../site-packages/_rocm_sdk_core/lib, but the PR's _cpu_moe extension links -lhiprtc. First build died ld: cannot find -lhiprtc; the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
• FreeToken source is checked out at the pinned PR head, not the branch (a future push to rocm714_torch214 cannot change the deployed engine).
• The stale Docker Hub login (docker login creds at ~/.docker/config.json) poisons podman's ANONYMOUS pulls of public images with "invalid username/password" — docker logout fixes the pull. Hit 2026-10-05 while pulling the base image.

Measured at cutover (2026-10-05/06)

• Boot: ~2.3 min container-start → API-ready (131 shards page-in + PLE pin + expert staging + CUDA graphs [1,2]). VRAM after graphs: 1.80 GiB free of 31.86. RAM: 122 GB used + 113 GB shared (the pinned expert/PLE slabs), 123 GB still available.
• Decode: 17.1 tok/s single-stream, temp 0, 400-token probe (clean 1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards; FreeToken single-card is the deployed baseline.
• The single-card LRU ceiling: the KV contract (harness worst case 177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so ft ctl cache --moe cannot raise it far on one card. The paths to a better decode rate are TP=2 (below) or a smaller pool (contract violation — the 177,356 floor is real traffic, not padding).
• TP=2 exists but does not boot on ROCm (probed 2026-10-06, zero-impact window): --tensor-parallel-size 2 --gpu 0,1 dies in the TP allreduce JIT — pynccl.cu compiles per-rank at boot and clang can't resolve cuda_runtime.h from the hipified include path (1 error generated when compiling for gfx1201, ninja stopped). The PR author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm edge, not a board or driver fault (radiance TP=2 allreduces fine on this pair since the 2026-09-20 swap). A future fix = patch the JIT include path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe then.
• Verification gates passed: /health ok; plain completion FLEET-OK (finish stop); thinking split — reasoning_content non-empty AND content carries the answer (17×24 → "408"); native tool_calls with correct name+arguments (finish tool_calls); 256×256 solid-red PNG through an image_url part answers exactly "Red" (prompt_tokens 127); stats: native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities [text,image], sampling defaults from generation_config.json (temp 1.0 / top_p 0.95 / top_k 20).
• Thinking counts against max_tokens (same as the vLLM era): a max_tokens: 10 probe returns EMPTY content with the budget spent in the think block. Clients must budget for thinking or disable it.
• Prometheus scrapes /metrics → 404 (Dankclaw scrapes 192.168.0.235:7770 from the llm-fleet job; FreeToken has no /metrics endpoint). The target will read down until the scrape config is repointed or a metrics shim lands — same class as the retired HTML-probe targets (see Observability).

The shape

Knob	Value	Why
--gpu / HIP_VISIBLE_DEVICES	0 / 0	ONE R9700; visibility pinned so in-container index 0 is the right card
--num-tokens	299,968	KV pool in TOKENS; must be a multiple of 64 (the auto attention backend is qsa_sparse, which overrides page size to 64 — 300,000 was refused: nearest valid values: 299968 or 300032)
--max-running-requests	2	first-boot conservative; raise after real-traffic slot-occupancy look
--max-output-tokens	36,864	harness maxTokens; requests that omit max_tokens get this default
--moe-cache-rate	0.10	GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative
parsers	AUTO	thinking → reasoning_content natively; tools in the qwen3.8 dialect; NO --reasoning-parser/--tool-call-parser flags

KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) × 2 B = 24 KiB/token (f16 KV; the 36 GDN layers use fixed per-seq state, not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE native 262,144-token conversation + full budget. models.yml contextWindow: 200704 stays (compacts early, never 400s).

RAM math (why the root unit + memlock)

246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB PLE ngram table in host RAM → the unit carries LimitMEMLOCK=infinity (root unit; dank's login session caps at 8 MiB and a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM container's ~170 GiB MUST be down first; systemctl stop frank-archon frees it (measured: 207 GB free right after the stop).

Resize live

ft ctl cache --kv <tokens>   # grows the KV pool LIVE (shrink needs restart)
ft ctl cache --moe <rate>    # same for the expert LRU
ft ctl stats                 # pool occupancy, VRAM, throughput

Verification (the fleet's real-completion law)

1. /health 200 — necessary, not sufficient.
2. Plain completion returns the asked-for string.
3. Thinking request: reasoning_content non-empty AND content carries the answer (the models.yml reasoningContentField: reasoning_content flip rides on this).
4. Vision probe: a solid-color PNG through an image_url part answers the exact color (the legacy top-level images field trap does not exist here — FreeToken is OpenAI-native).
5. Unit-managed: systemctl show -p MainPID behind the health answer (the 2026-09-26 stray-port false green).

Rollback

git checkout frank-archon-vllm-shapeG-20261003 -- \
  scripts/frank-archon.env scripts/serve-frank-archon.sh \
  scripts/frank-archon.service
./tests/run.sh && git commit -m "frank-archon: rollback to vLLM shape G" \
  && git push && ./scripts/deploy.sh dankserver --service frank-archon

The tcclaviger image and the FreeToken image coexist in dank's rootless storage; the deploy re-pulls the pinned digest if absent.

Index

• dankserver-qwen38-fp8-expert-offload-tuning — the retired vLLM shape (rollback target; its 362-line env lore lives in the tag)
• freetoken-frank-small — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
• LLM/index — fleet index