docs: frank-primary runs FreeToken (PR587 ROCm) — runbook, index, AGENTS, log
New runbook frank-primary-freetoken-pr587 (build laws, shape, measured cutover numbers, TP=2 non-boot signature, rollback); LLM/index.md intro + mermaid + active/retired entries; AGENTS.md fleet row + section + bench and model-config mentions + runbook index; log.md entry. Seam pins the measured KV pool 299968 with the multiple-of-64 guard. NOTE: intended PGP signature skipped — key 0x76EA601A11B178D7 has no secret key in this machine's keyring (fresh ~/.gnupg); the PGP identity lives elsewhere.
This commit is contained in:
Frank committed
1 parent
6ba19f6f48
commit
e2308dff71
5 files changed
+295
-139
No files matched your search
@@ -0,0 +1,170 @@
|
||||
# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
|
||||
|
||||
**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
|
||||
|
||||
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
|
||||
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
|
||||
on the host into the local image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201` by
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served
|
||||
alias (`frank-primary`), same 131-shard payload
|
||||
(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
|
||||
git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore),
|
||||
runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
|
||||
## Why FreeToken
|
||||
|
||||
The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
|
||||
0.982 was a hard law (1.00 was **unbootable** — worker init raises
|
||||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
|
||||
total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
|
||||
still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
|
||||
fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
|
||||
GPU cache, sizes **KV as an explicit token pool**, and boots in one
|
||||
deterministic pass — no per-boot estimator to fight.
|
||||
|
||||
## The build (on dankserver, rootless podman as dank)
|
||||
|
||||
```bash
|
||||
cd /home/dank/serverAppConfigs
|
||||
podman build -t frank-primary-freetoken:pr587-77a7201 \
|
||||
-f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
|
||||
```
|
||||
|
||||
Laws baked into the Dockerfile (each one cost a failure):
|
||||
|
||||
- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
|
||||
the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
|
||||
rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
|
||||
rocm10.1.0 — an untested pair for this PR).
|
||||
- **`--no-build-isolation --no-deps -e .`** — a plain
|
||||
`pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
|
||||
build (the dependency set is authored for NVIDIA). Runtime deps are
|
||||
curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
|
||||
nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
|
||||
**flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
|
||||
FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
|
||||
are exactly those backends).
|
||||
- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
|
||||
versioned `libhiprtc.so.7` in
|
||||
`/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
|
||||
extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
|
||||
the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
|
||||
- FreeToken source is checked out at the **pinned PR head**, not the branch
|
||||
(a future push to `rocm714_torch214` cannot change the deployed engine).
|
||||
- The stale Docker Hub login (`docker login` creds at
|
||||
`~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
|
||||
with "invalid username/password" — `docker logout` fixes the pull. Hit
|
||||
2026-10-05 while pulling the base image.
|
||||
|
||||
## Measured at cutover (2026-10-05/06)
|
||||
|
||||
|
||||
- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
|
||||
pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
|
||||
**1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
|
||||
pinned expert/PLE slabs)**, 123 GB still available.
|
||||
- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
|
||||
1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
|
||||
FreeToken single-card is the deployed baseline.
|
||||
- **The single-card LRU ceiling**: the KV contract (harness worst case
|
||||
177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
|
||||
the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
|
||||
`ft ctl cache --moe` cannot raise it far on one card. The paths to a
|
||||
better decode rate are TP=2 (below) or a smaller pool (contract
|
||||
violation — the 177,356 floor is real traffic, not padding).
|
||||
- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
|
||||
zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
|
||||
allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
|
||||
resolve `cuda_runtime.h` from the hipified include path
|
||||
(`1 error generated when compiling for gfx1201`, ninja stopped). The PR
|
||||
author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
|
||||
edge, not a board or driver fault (radiance TP=2 allreduces fine on this
|
||||
pair since the 2026-09-20 swap). A future fix = patch the JIT include
|
||||
path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
|
||||
then.
|
||||
- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
|
||||
(finish stop); thinking split — `reasoning_content` non-empty AND
|
||||
`content` carries the answer (17×24 → "408"); native `tool_calls` with
|
||||
correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
|
||||
an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
|
||||
native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
|
||||
[text,image], sampling defaults from generation_config.json
|
||||
(temp 1.0 / top_p 0.95 / top_k 20).
|
||||
- **Thinking counts against max_tokens** (same as the vLLM era): a
|
||||
`max_tokens: 10` probe returns EMPTY content with the budget spent in the
|
||||
think block. Clients must budget for thinking or disable it.
|
||||
- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
|
||||
from the llm-fleet job; FreeToken has no /metrics endpoint). The target
|
||||
will read down until the scrape config is repointed or a metrics shim
|
||||
lands — same class as the retired HTML-probe targets
|
||||
(see [[Observability]]).
|
||||
|
||||
|
||||
## The shape
|
||||
|
||||
| Knob | Value | Why |
|
||||
|---|---|---|
|
||||
| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
|
||||
| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
|
||||
| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
|
||||
| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
|
||||
| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
|
||||
| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
|
||||
|
||||
KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
|
||||
2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
|
||||
not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
|
||||
case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
|
||||
native 262,144-token conversation + full budget. `models.yml`
|
||||
`contextWindow: 200704` stays (compacts early, never 400s).
|
||||
|
||||
## RAM math (why the root unit + memlock)
|
||||
|
||||
246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
|
||||
PLE ngram table in host RAM → the unit carries
|
||||
`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
|
||||
a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
|
||||
container's ~170 GiB MUST be down first; `systemctl stop frank-primary`
|
||||
frees it (measured: 207 GB free right after the stop).
|
||||
|
||||
## Resize live
|
||||
|
||||
```bash
|
||||
ft ctl cache --kv <tokens> # grows the KV pool LIVE (shrink needs restart)
|
||||
ft ctl cache --moe <rate> # same for the expert LRU
|
||||
ft ctl stats # pool occupancy, VRAM, throughput
|
||||
```
|
||||
|
||||
## Verification (the fleet's real-completion law)
|
||||
|
||||
1. `/health` 200 — necessary, not sufficient.
|
||||
2. Plain completion returns the asked-for string.
|
||||
3. Thinking request: `reasoning_content` non-empty AND `content` carries the
|
||||
answer (the models.yml `reasoningContentField: reasoning_content` flip
|
||||
rides on this).
|
||||
4. Vision probe: a solid-color PNG through an `image_url` part answers the
|
||||
exact color (the legacy top-level `images` field trap does not exist
|
||||
here — FreeToken is OpenAI-native).
|
||||
5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
|
||||
2026-09-26 stray-port false green).
|
||||
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
git checkout frank-primary-vllm-shapeG-20261003 -- \
|
||||
scripts/frank-primary.env scripts/serve-frank-primary.sh \
|
||||
scripts/frank-primary.service
|
||||
./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \
|
||||
&& git push && ./scripts/deploy.sh dankserver --service frank-primary
|
||||
```
|
||||
|
||||
The tcclaviger image and the FreeToken image coexist in dank's rootless
|
||||
storage; the deploy re-pulls the pinned digest if absent.
|
||||
|
||||
## Index
|
||||
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
|
||||
(rollback target; its 362-line env lore lives in the tag)
|
||||
- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
|
||||
- [[LLM/index]] — fleet index
|
||||
Reference in new issue
Block a user