dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: record the measured shape-E and compile-cache numbers

Browse Source 
Live after the 1ac4d48 deploy: 'expert_precise_memory': True, runtime overhead
5.25 GiB, expert cache space 17.22 GiB / 7522 slots, swap region 13.7 GiB /
5986 slots, GPU KV cache 398,658 tokens / 1.99x, pool 3,150,000,000, MNBT 1024
-- the probe's projection is now production.

Compile cache proven on consecutive boots of that shape: cold 'Compiling a
graph for compile range (1, 1024) takes 179.24 s' / 'torch.compile took
188.26 s'; warm 'Directly load the compiled graph(s) ... from the cache, took
2.143 s' / 'torch.compile took 3.97 s in total' / 'init engine ... took 41.13 s
(compilation: 3.98 s)'. Host dir 1.4 GB / 4,707 files. Time to /health stays
payload-bound (490 s warm), so the claim is ~184 s of inductor/AOT build per
restart, not a proportional boot cut.

Tests: 367/367.
This commit is contained in:
Frank
2026-10-02 15:05:34 -07:00
parent 1ac4d48371
commit 57f33e5289
5 changed files with 59 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+13
View File
@@ -338,6 +338,19 @@ includes the compile range, so the MNBT choice costs exactly one compilation.
Entries are version- and config-hashed: a stale entry is a miss, never a wrong
answer.
**Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this
exact shape:**
| boot | engine log |
|---|---|
| cold (deploy) | `Compiling a graph for compile range (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s in total` |
| warm (restart) | `Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)` |
The host directory held **1.4 GB in 4,707 files** after the cold boot. Time to
`/health` is dominated by the 172.8 GiB payload load (the warm boot reached
health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a
proportional cut in boot time.
`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
rather than a no-op - the literal is read out of
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately