diff --git a/AGENTS.md b/AGENTS.md index 77a1f99..60314c0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser … +| Dankserver | `frank-primary` | **FreeToken PR #587** (branch `rocm714_torch214`, head `77a7201`; local image `frank-primary-freetoken:pr587-77a7201` built on the host from `scripts/docker/frank-primary-freetoken.Dockerfile` — digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards, modelopt block-FP8, loaded straight from safetensors), **ONE R9700 (second card idle)**, all experts in **pinned host RAM** (~120 GiB) behind a GPU LRU (`--moe-cache-rate 0.10`), PLE table 47.7 GiB pinned (root unit + `LimitMEMLOCK=infinity`), **KV pool `--num-tokens 299968`** (24 KiB/token, multiple-of-64 law of the `qsa_sparse` auto backend), 2 slots, `--max-output-tokens 36864`, **parsers AUTO** (thinking → `reasoning_content`, tools qwen3.8 dialect), vision tower host-streamed. Measured: boot ~2.3 min cold / 92 s warm, decode **17.1 tok/s** single-stream; **TP=2 does not boot on ROCm** (pynccl JIT); `/metrics` 404 (Prometheus scrape reads down). Deploy gate reads the `/health` BODY (`status:"ok"`) — FreeToken 200s while loading. Rollback tag `frank-primary-vllm-shapeG-20261003` |7770| | Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770| | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 | @@ -260,7 +260,7 @@ merely wedged kernel usually still answers ARP. `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700): + - Dankserver `frank-primary` (root unit, FreeToken on one R9700): `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary` 2. Bench. NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in @@ -295,8 +295,9 @@ a role pointing at a provider the file does not define fails at call time. As of 2026-10-05 **two of the three endpoints accept images** (`input: [text, image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while -`frank-primary` takes them through a BF16 mmproj loaded explicitly (this build never -autoloads projectors), and `frank-small` through +`frank-primary` takes them natively (FreeToken builds the qwen3.8 vision +tower and streams its block weights from host RAM two at a time by default), +and `frank-small` through `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is **200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the @@ -443,122 +444,63 @@ pairs with Q5 only; SGLang rollback in `/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/`). -### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE) +### frank-primary — FreeToken (PR #587 ROCm), Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE) -The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`, -digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** -(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**, -**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since -2026-10-03 the **shape-G** geometry (shape E + MNBT 2048 + the CUDA-graph plan -term at the measured floor): `--max-model-len 200704` × 3 slots with -`--kv-cache-memory-bytes 3,166,500,000`/rank (~400,700 KV tokens — the engine -prices this pool at 7,891.8 B/token here), util **0.982** (the law — see the -unbootable-1.0 lesson below), `--expert-precise-memory`, -`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`, -`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries -`contextWindow: 200704` for this endpoint — the served window, not the model's -native 262144, so the harness compacts before vLLM answers a longer prompt -with a 400. +The **FreeToken engine at PR #587 head `77a7201`** (branch +`rocm714_torch214`, built on the host into the local image +`localhost/frank-primary-freetoken:pr587-77a7201` by +`scripts/docker/frank-primary-freetoken.Dockerfile` — the PR's own +docs/install_amd.md stack: digest-pinned +`rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0` base = +torch 2.11.0+rocm7.14/triton 3.7, `--no-build-isolation --no-deps -e .`, +curated portable deps, `flashlib==0.3.0 --no-deps`, and a `libhiprtc.so` +dev-symlink the pip-ROCm wheel lacks) serving +**`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards, +standard modelopt block-FP8 — 75,264 `weight_scale_inv` tensors, no +`input_scale`, loaded directly from HF safetensors, no FTW conversion) on +**ONE R9700 (gfx1201)** — the second card sits idle. Plain `podman run` +through `scripts/serve-frank-primary.sh` under the root systemd unit +(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the +container name is the unit name (`frank-primary`; the vLLM-era +`frank-primary-qwen` is in the legacy cleanup list). Cutover 2026-10-05; +rollback tag `frank-primary-vllm-shapeG-20261003` (the tcclaviger vLLM trio +and its 362-line env lore live in that tag; the tcclaviger image stays in +dank's rootless storage). -**`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by -single-variable boot probe against the then-deployed shape: SEQS 2 fails at -BOTH MNBT 1024 and 2048 with `torch.fx.experimental.symbolic_shapes.Constraint -ViolationError: Constraints violated (L['query_start_loc'].size()[0]) … -specialized it to be a constant (3)`; SEQS 3 passes at both. `model_runner.py` -sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets -`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is -`SEQS + 1` for any `MNBT ≥ SEQS` — MNBT only moves the compile RANGE -((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA -graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape. +**Shape**: all expert weights in **pinned host RAM** (~120 GiB) behind a +GPU LRU (`--moe-cache-rate 0.10`, ~12 GiB — the single-card ceiling given +the KV contract), the ~47.7 GiB PLE ngram table pinned in host RAM +(→ root unit + memlock law), **KV pool `--num-tokens 299968`** +(24 KiB/token f16 across the 12 full-attention layers ≈ 7.0 GiB; **must be +a multiple of 64** — the auto attention backend is `qsa_sparse`, which +overrides page size to 64; 300,000 was refused at first boot), 2 decode +slots, `--max-output-tokens 36864` (the harness maxTokens; requests that +omit one get this default). **Parsers are AUTO**: thinking lands in +`reasoning_content` natively, tool calls in the qwen3.8 dialect — models.yml +flips `reasoningContentField` to `reasoning_content` on the harness side. +Vision works (models.yml keeps `input: [text, image]`); the vision tower's +blocks stream from host RAM two at a time by default. -**The expert arena has THREE reachable levers: `max_model_len`, the overhead -term, and the CUDA-graph estimate.** -The fork's planner sizes expert cache as `budget − non-expert − KV reserve − -graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token × -`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**; -200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena -0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime -headroom (the shape it replaced died with `0 bytes free` on a real 995-token -request). The second lever is `--expert-precise-memory`, **ON since -2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted + -compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB -difference lands in the arena — measured plan lines at 200704 × 3: overhead -5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region -13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9 -deliberately: because the pool is not an arena lever, cutting it would cost -concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots. -The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV -headroom collapses the offload hit rate. +**Measured at cutover**: boot ~2.3 min cold-ish / 92 s warm → /health +status ok; decode **17.1 tok/s** single-stream temp 0 (the vLLM incumbent +measured ~30 tok/s on TWO cards; the KV contract caps the single-card LRU, +and **TP=2 exists but does not boot on ROCm** — the TP allreduce JIT +(`pynccl.cu`) can't resolve `cuda_runtime.h` for gfx1201; probed +2026-10-06, signature in the runbook). `/metrics` is a 404 — the Dankclaw +Prometheus llm-fleet scrape of :7770 reads down until repointed. +`ft ctl cache --kv/--moe` resizes pools live (grow only; shrink needs a +restart); `ft ctl stats` reports occupancy. -**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).** -`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the -planner charges `budget = total × util` against terms that overstate reality: -the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while -the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25 -GiB) is far above the idle-measured figure. So the visible free space is the -estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena -lever (each +0.01 = +0.32 GiB) but **0.982 is the law, and 1.00 is UNBOOTABLE -on this fork — measured 2026-10-03**: the worker init demands -`free > budget`, and at `le=1` budget == device total, so both ranks raise -`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` before a -single weight loads, twice (2-restart crash loop, zero payload progress). The -CLI field accepting `le=1` is the trap; Frank's 0.982 is where it always -ships. What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` / -`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the -analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's -own post-capture line reported 0.31 GiB, and a second boot (with the plan -already charging 0.32 GiB) reported **-0.06 GiB**, so the 256 MiB base pool is -pure over-charge: shape G charges `0 base + 12 MiB/graph` = 0.07 GiB, which -still prices per graph for a future 18-graph spec-decode shape (0.21 GiB). -Verify every deploy by comparing the `Estimated CUDA graph memory` -line with `Graph capturing finished … took X GiB`; if capture exceeds the -estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO -arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)` -anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the -fitted figure was fitted AT 4096. +**Deploy gate law (changed for FreeToken)**: `/health` returns **200 with +`{"status":"loading"}`** while the engine loads — the FastAPI front end +binds before the scheduler is up, so a bare `curl` false-greens (measured: +"healthy after 6 s" mid-page-in). The deployer loops until the body says +`"status":"ok"`; the MainPID stray-port guard is unchanged. The image gate +is `podman image exists` (a local build carries no RepoDigests — the old +digest grep refused exactly this shape; pinning lives in the Dockerfile). -**A container's compile cache does NOT survive `podman rm -f`, and on this -endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no -`/root/.cache` at all** (fresh container: `No such file or directory`); after a -boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus -`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine -prints `torch.compile took 181.20 s in total` (173.19 s of it one graph, -`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and -reboot re-pays it. The serve script now mounts -`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT / -torch_compile tree (`VLLM_CACHE_ROOT`, whose default is -`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and -nothing in the image is hidden because the image has no such directory. The -compile RANGE is part of the cache key, so an MNBT change costs one -recompilation, not a permanent tax. Same class of fix as the recipe's -`/cache/{triton,vllm,inductor}` volumes. **Proven live 2026-10-02 on -consecutive boots:** cold `Compiling a graph for compile range (1, 1024) takes -179.24 s` / `torch.compile took 188.26 s`; warm `Directly load the compiled -graph(s) for compile range (1, 1024) from the cache, took 2.143 s` / -`torch.compile took 3.97 s in total` / `init engine (profile, create kv cache, -warmup model) took 41.13 s (compilation: 3.98 s)`. Time to `/health` is -dominated by the 172.8 GiB payload load, so the visible win is the ~184 s of -inductor/AOT build, not a proportional boot cut. - -Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode -**37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5; -2-way concurrency sums BELOW parity (25.49 vs 26.8 single) — slots are a -capacity feature, not a throughput one. V-only q8 KV is NOT possible via -config: stock `--kv-cache-dtype` is K+V together; a V-only quant needs a fork -patch. - -Unit and self-healing: root systemd unit (`scripts/frank-primary.service`, -`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120 -frank-primary-qwen`. The serve script drives `podman run` directly, gates -the digest-pinned image, the payload's `config.json` and its exact -131-shard count and the GPU device nodes, clears stale container -identities, then blocks on `podman wait`. `deploy_llm_primary` gates the -same, waits out :7770, relinks/enables/restarts the unit and health-gates -with the MainPID check. Boot cost ~7 min warm (the 172.8 GiB payload load; the -~184 s inductor/AOT build inside it is what the mounted cache now skips — see -the compile-cache lesson below); health returns 503 while loading. - -Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`. +Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`. +Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`. Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`. ## Key Lessons Learned @@ -1005,7 +947,7 @@ Active: |---------|---------| | frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` | | frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` | -| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` | +| frank-primary (FreeToken PR #587 ROCm, Qwen3.8-Flash-Next FP8, 1×R9700, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -1014,6 +956,7 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` | +| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` | | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` | | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | diff --git a/serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md b/serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md new file mode 100644 index 0000000..5c677b1 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md @@ -0,0 +1,170 @@ +# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700) + +**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle + +The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head +`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built +on the host into the local image +`localhost/frank-primary-freetoken:pr587-77a7201` by +`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served +alias (`frank-primary`), same 131-shard payload +(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback: +git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore), +runbook [[dankserver-qwen38-fp8-expert-offload-tuning]]. + +## Why FreeToken + +The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util +0.982 was a hard law (1.00 was **unbootable** — worker init raises +`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget == +total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic +still degraded ~40k tokens in (the GDN state-fault class, never fixed in the +fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU +GPU cache, sizes **KV as an explicit token pool**, and boots in one +deterministic pass — no per-boot estimator to fight. + +## The build (on dankserver, rootless podman as dank) + +```bash +cd /home/dank/serverAppConfigs +podman build -t frank-primary-freetoken:pr587-77a7201 \ + -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/ +``` + +Laws baked into the Dockerfile (each one cost a failure): + +- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` — + the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No + rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are + rocm10.1.0 — an untested pair for this PR). +- **`--no-build-isolation --no-deps -e .`** — a plain + `pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA + build (the dependency set is authored for NVIDIA). Runtime deps are + curated explicitly; `flashlib==0.3.0 --no-deps` (its numba / + nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT; + **flashinfer / sgl_kernel intentionally absent** (no ROCm builds; + FreeToken falls back to its Triton kernels — the PR's 12 pytest failures + are exactly those backends). +- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the + versioned `libhiprtc.so.7` in + `/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe` + extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`; + the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir. +- FreeToken source is checked out at the **pinned PR head**, not the branch + (a future push to `rocm714_torch214` cannot change the deployed engine). +- The stale Docker Hub login (`docker login` creds at + `~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images + with "invalid username/password" — `docker logout` fixes the pull. Hit + 2026-10-05 while pulling the base image. + +## Measured at cutover (2026-10-05/06) + + +- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE + pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs: + **1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the + pinned expert/PLE slabs)**, 123 GB still available. +- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean + 1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards; + FreeToken single-card is the deployed baseline. +- **The single-card LRU ceiling**: the KV contract (harness worst case + 177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for + the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so + `ft ctl cache --moe` cannot raise it far on one card. The paths to a + better decode rate are TP=2 (below) or a smaller pool (contract + violation — the 177,356 floor is real traffic, not padding). +- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06, + zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP + allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't + resolve `cuda_runtime.h` from the hipified include path + (`1 error generated when compiling for gfx1201`, ninja stopped). The PR + author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm + edge, not a board or driver fault (radiance TP=2 allreduces fine on this + pair since the 2026-09-20 swap). A future fix = patch the JIT include + path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe + then. +- **Verification gates passed**: /health ok; plain completion `FLEET-OK` + (finish stop); thinking split — `reasoning_content` non-empty AND + `content` carries the answer (17×24 → "408"); native `tool_calls` with + correct name+arguments (finish tool_calls); 256×256 solid-red PNG through + an `image_url` part answers exactly "Red" (prompt_tokens 127); stats: + native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities + [text,image], sampling defaults from generation_config.json + (temp 1.0 / top_p 0.95 / top_k 20). +- **Thinking counts against max_tokens** (same as the vLLM era): a + `max_tokens: 10` probe returns EMPTY content with the budget spent in the + think block. Clients must budget for thinking or disable it. +- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770 + from the llm-fleet job; FreeToken has no /metrics endpoint). The target + will read down until the scrape config is repointed or a metrics shim + lands — same class as the retired HTML-probe targets + (see [[Observability]]). + + +## The shape + +| Knob | Value | Why | +|---|---|---| +| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card | +| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) | +| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look | +| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default | +| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative | +| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags | + +KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) × +2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state, +not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst +case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE +native 262,144-token conversation + full budget. `models.yml` +`contextWindow: 200704` stays (compacts early, never 400s). + +## RAM math (why the root unit + memlock) + +246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB +PLE ngram table in host RAM → the unit carries +`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and +a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM +container's ~170 GiB MUST be down first; `systemctl stop frank-primary` +frees it (measured: 207 GB free right after the stop). + +## Resize live + +```bash +ft ctl cache --kv # grows the KV pool LIVE (shrink needs restart) +ft ctl cache --moe # same for the expert LRU +ft ctl stats # pool occupancy, VRAM, throughput +``` + +## Verification (the fleet's real-completion law) + +1. `/health` 200 — necessary, not sufficient. +2. Plain completion returns the asked-for string. +3. Thinking request: `reasoning_content` non-empty AND `content` carries the + answer (the models.yml `reasoningContentField: reasoning_content` flip + rides on this). +4. Vision probe: a solid-color PNG through an `image_url` part answers the + exact color (the legacy top-level `images` field trap does not exist + here — FreeToken is OpenAI-native). +5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the + 2026-09-26 stray-port false green). + +## Rollback + +```bash +git checkout frank-primary-vllm-shapeG-20261003 -- \ + scripts/frank-primary.env scripts/serve-frank-primary.sh \ + scripts/frank-primary.service +./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \ + && git push && ./scripts/deploy.sh dankserver --service frank-primary +``` + +The tcclaviger image and the FreeToken image coexist in dank's rootless +storage; the deploy re-pulls the pinned digest if absent. + +## Index + +- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape + (rollback target; its 362-line env lore lives in the tag) +- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090) +- [[LLM/index]] — fleet index diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 9bd3814..4d2586a 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,18 +1,17 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-10-03): -**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the -digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving -orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s -on :7770, TP=2 with automatic tiered-expert offload, **shape E since -2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length -conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative -decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired -(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory` -(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a -mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays -`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3: -SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001), +Documentation for the LLM inference fleet on the two hosts (2026-10-05): +**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local +image `frank-primary-freetoken:pr587-77a7201` built on the host from +`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head +`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 — +driving `podman run` directly, serving orcarouter +/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in +pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool +`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2 +decode slots, parsers AUTO (thinking → `reasoning_content` natively; +measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the +runbook). Rollback tag frank-primary-vllm-shapeG-20261003), and Dankclaw runs two endpoints — **halogen container (frank-large, port 7770: ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai @@ -34,12 +33,12 @@ graph LR subgraph Dankserver[AMD Dankserver - Ubuntu 26.04] A[2x Radeon AI PRO R9700 gfx1201, 32 GiB each] - B2[tcclaviger RDNA4 vLLM fork, digest-pinned + B2[FreeToken PR587 ROCm, local image podman run, frank-primary, port 7770] - C[Qwen3.8-Flash-Next FP8, no spec decode -TP=2, expert offload automatic, 3 x 200704 ctx, -fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers, -expert-precise-memory arena 7,522 slots, /root/.cache mounted] + C[Qwen3.8-Flash-Next FP8 +1x R9700, experts in pinned host RAM + GPU LRU 10%, +KV pool 299,968 tokens, 2 slots, parsers AUTO, +TP=2 untested on ROCm; second R9700 idle] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] @@ -95,8 +94,22 @@ frank-small, port 7771] target; payload on disk; owns the measured boot-pair table)**: the same runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower — 104.8 tok/s, 3,571 MiB true free -- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since - 2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4 +- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05** + (root systemd unit, port 7770): FreeToken at PR #587 head `77a7201` + (branch rocm714_torch214) built on the host into the local image + `frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned + rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law, + curated --no-deps install per docs/install_amd.md), serving the SAME + orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM + (~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens + (multiple-of-64 law), 2 slots, parsers AUTO (thinking → + `reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s + single-stream, vision "Red" probe passes, native tool_calls; TP=2 does + not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag + frank-primary-vllm-shapeG-20261003 +- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05 + (rollback target; the vLLM trio + env lore live in the tag)**: the + tcclaviger RDNA4 vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3 slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x), diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index c05d0d6..91ee769 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,5 +1,31 @@ # Bundle Change Log +## 2026-10-05 +- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on + dankserver.** Built FreeToken at PR head `77a7201` on the host into the + local image `frank-primary-freetoken:pr587-77a7201` + (digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws + bought with failures: the pip-ROCm wheel needs a `libhiprtc.so` dev + symlink for the PR's `_cpu_moe` link, and a stale `docker login` at + `~/.docker/config.json` poisons podman's ANONYMOUS pulls). Trio rewritten + through git (env/serve/unit + seam + deployer): ONE R9700, experts in + pinned host RAM behind a 0.10 GPU LRU, PLE 47.7 GiB pinned, KV pool + 299,968 tokens (multiple-of-64 — the `qsa_sparse` auto backend overrides + page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking → + `reasoning_content`; models.yml flipped accordingly). Container renamed + to the unit name (Frank's call; `frank-primary-qwen` → legacy list); + deployer image gate → `podman image exists` (local builds carry no + RepoDigests) and health gate reads the BODY — FreeToken 200s with + `{"status":"loading"}` mid-load, which false-greened a deploy at 6 s + (vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK + completion, reasoning split (17×24 → "408"), native `tool_calls`, vision + "Red" probe, boot 2.3 min cold / 92 s warm, decode 17.1 tok/s + single-stream (vLLM was ~30 on two cards; the KV contract caps the + single-card LRU; TP=2 exists but the pynccl JIT can't compile for + gfx1201 — probed, signature in the runbook). `/metrics` is a 404 — the + Dankclaw Prometheus llm-fleet scrape of :7770 reads down until + repointed. Rollback: tag `frank-primary-vllm-shapeG-20261003`. + Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`. ## 2026-10-05 - **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright project.** The `11-admin` walk included the six `*arr`/qBittorrent admin diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index ed7bc58..884403e 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -99,13 +99,17 @@ DOCKERFILE="$REPO_ROOT/scripts/docker/frank-primary-freetoken.Dockerfile" @test "the FreeToken engine shape is in the env file" { # KV pool: TOKENS, not bytes. 24 KiB/token (12 full-attention layers x # 2 KV heads x 256 head_dim x K+V x f16) — the 36 GDN layers use fixed - # per-seq state, not per-token KV. 300,000 tokens = 7.2 GiB and clears + # per-seq state, not per-token KV. 299,968 tokens = 7.0 GiB and clears # the harness worst case (140,492-token prompt + 36,864 max_tokens = # 177,356); one NATIVE 262,144-token conversation + full budget also - # fits. The arithmetic guard keeps future resizes honest. - grep -q '^FRANK_PRIMARY_FT_NUM_TOKENS=300000$' "$ENV_FILE" + # fits. The value must stay a MULTIPLE OF 64: the auto attention + # backend is qsa_sparse, which overrides page size to 64 (measured + # first boot — 300,000 was refused). The arithmetic guard keeps future + # resizes honest. + grep -q '^FRANK_PRIMARY_FT_NUM_TOKENS=299968$' "$ENV_FILE" nt=$(awk -F= '/^FRANK_PRIMARY_FT_NUM_TOKENS=/{print $2}' "$ENV_FILE") [ "$nt" -ge 177356 ] + [ $((nt % 64)) -eq 0 ] # First-boot conservative shape: 2 decode slots, 10% expert LRU # (~12 GiB of the ~120 GiB fp8 expert set), output ceiling = the # harness maxTokens, single R9700 at index 0.