From 57f33e52899e06667cfd6e926c484f2e135b16a8 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 2 Oct 2026 15:05:34 -0700 Subject: [PATCH] frank-primary: record the measured shape-E and compile-cache numbers Live after the 1ac4d48 deploy: 'expert_precise_memory': True, runtime overhead 5.25 GiB, expert cache space 17.22 GiB / 7522 slots, swap region 13.7 GiB / 5986 slots, GPU KV cache 398,658 tokens / 1.99x, pool 3,150,000,000, MNBT 1024 -- the probe's projection is now production. Compile cache proven on consecutive boots of that shape: cold 'Compiling a graph for compile range (1, 1024) takes 179.24 s' / 'torch.compile took 188.26 s'; warm 'Directly load the compiled graph(s) ... from the cache, took 2.143 s' / 'torch.compile took 3.97 s in total' / 'init engine ... took 41.13 s (compilation: 3.98 s)'. Host dir 1.4 GB / 4,707 files. Time to /health stays payload-bound (490 s warm), so the claim is ~184 s of inductor/AOT build per restart, not a proportional boot cut. Tests: 367/367. --- AGENTS.md | 20 ++++++++++++++----- scripts/frank-primary.env | 10 ++++++++++ scripts/serve-frank-primary.sh | 6 +++++- ...server-qwen38-fp8-expert-offload-tuning.md | 13 ++++++++++++ serverAppConfigs-docs/log.md | 16 +++++++++++++++ 5 files changed, 59 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d8b574d..9945ea9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -433,11 +433,13 @@ The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666 digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**, **fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since -2026-10-02 the **shape-D** geometry: `--max-model-len 200704` × 3 slots with +2026-10-02 the **shape-E** geometry: `--max-model-len 200704` × 3 slots with `--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2 full-length conversations** (measured live: `398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`) while the argv keeps the -only slot count that compiles (below). `models.yml` carries +only slot count that compiles (below), plus `--expert-precise-memory` +(`runtime overhead 5.25 GiB`, `expert cache space 17.22 GiB / 7,522 slots`), +`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries `contextWindow: 200704` for this endpoint — the served window, not the model's native 262144, so the harness compacts before vLLM answers a longer prompt with a 400. @@ -485,7 +487,14 @@ torch_compile tree (`VLLM_CACHE_ROOT`, whose default is nothing in the image is hidden because the image has no such directory. The compile RANGE is part of the cache key, so an MNBT change costs one recompilation, not a permanent tax. Same class of fix as the recipe's -`/cache/{triton,vllm,inductor}` volumes. +`/cache/{triton,vllm,inductor}` volumes. **Proven live 2026-10-02 on +consecutive boots:** cold `Compiling a graph for compile range (1, 1024) takes +179.24 s` / `torch.compile took 188.26 s`; warm `Directly load the compiled +graph(s) for compile range (1, 1024) from the cache, took 2.143 s` / +`torch.compile took 3.97 s in total` / `init engine (profile, create kv cache, +warmup model) took 41.13 s (compilation: 3.98 s)`. Time to `/health` is +dominated by the 172.8 GiB payload load, so the visible win is the ~184 s of +inductor/AOT build, not a proportional boot cut. Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode **37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5; @@ -501,8 +510,9 @@ the digest-pinned image, the payload's `config.json` and its exact 131-shard count and the GPU device nodes, clears stale container identities, then blocks on `podman wait`. `deploy_llm_primary` gates the same, waits out :7770, relinks/enables/restarts the unit and health-gates -with the MainPID check. Boot cost ~7 min warm; health returns 503 while -loading. +with the MainPID check. Boot cost ~7 min warm (the 172.8 GiB payload load; the +~184 s inductor/AOT build inside it is what the mounted cache now skips — see +the compile-cache lesson below); health returns 503 while loading. Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`. Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`. diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 6dd151f..3109345 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -127,6 +127,16 @@ FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 # compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a # host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the # comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir). +# PROVEN LIVE 2026-10-02, two boots of this exact shape: +# cold `Compiling a graph for compile range (1, 1024) takes 179.24 s` +# `torch.compile took 188.26 s in total` +# warm `Directly load the compiled graph(s) for compile range (1, 1024) +# from the cache, took 2.143 s` +# `torch.compile took 3.97 s in total` +# `init engine (profile, create kv cache, warmup model) took 41.13 s +# (compilation: 3.98 s)` +# (Total time to /health is dominated by the 172.8 GiB payload load, not the +# compile — the saving is the ~184 s of inductor/AOT build.) # Safe to grow/prune: entries are keyed by version + config hash, so a stale # entry is a miss, never a wrong answer. FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh index 70e0703..8bf4b6e 100755 --- a/scripts/serve-frank-primary.sh +++ b/scripts/serve-frank-primary.sh @@ -32,7 +32,11 @@ # container's ephemeral layer, and `podman rm -f` deletes them — so every # deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s # of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all -# later boots load the compiled artifact instead, and hides nothing. +# later boots load the compiled artifact instead, and hides nothing. PROVEN +# LIVE 2026-10-02 on consecutive boots of this shape: cold `torch.compile took +# 188.26 s`, warm `Directly load the compiled graph(s) for compile range +# (1, 1024) from the cache, took 2.143 s` / `torch.compile took 3.97 s in +# total` / `init engine … took 41.13 s (compilation: 3.98 s)`. # # The root unit (scripts/frank-primary.service, User=dank, # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index f1ab43d..5f3e4e5 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -338,6 +338,19 @@ includes the compile range, so the MNBT choice costs exactly one compilation. Entries are version- and config-hashed: a stale entry is a miss, never a wrong answer. +**Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this +exact shape:** + +| boot | engine log | +|---|---| +| cold (deploy) | `Compiling a graph for compile range (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s in total` | +| warm (restart) | `Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)` | + +The host directory held **1.4 GB in 4,707 files** after the cold boot. Time to +`/health` is dominated by the 172.8 GiB payload load (the warm boot reached +health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a +proportional cut in boot time. + `GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob rather than a no-op - the literal is read out of `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 9da0f25..cad8270 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1187,4 +1187,20 @@ every restart since the cutover. The flag was fine both times. - Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658 tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]]. +- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:** + `'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache + space 17.22 GiB / 7522 slots`, `swap region 13.7 GiB / 5986 slots`, `GPU KV + cache size: 398,658 tokens … 1.99x`, pool 3,150,000,000, MNBT 1024. The + precise overhead and the arena growth are now production numbers, not a + probe's projection. +- **The compile-cache fix was then proven on the same endpoint, two consecutive + boots of the deployed shape:** cold `Compiling a graph for compile range + (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load + the compiled graph(s) for compile range (1, 1024) from the cache, took + 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, + create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)`. The host + dir held 1.4 GB in 4,707 files after the cold boot. **Time to `/health` is + dominated by the 172.8 GiB payload load** (the warm boot still needed 490 s), + so the honest claim is ~184 s of inductor/AOT build recovered per restart — + not a proportional boot cut.