frank-primary: record the measured shape-E and compile-cache numbers
Live after the 1ac4d48 deploy: 'expert_precise_memory': True, runtime overhead
5.25 GiB, expert cache space 17.22 GiB / 7522 slots, swap region 13.7 GiB /
5986 slots, GPU KV cache 398,658 tokens / 1.99x, pool 3,150,000,000, MNBT 1024
-- the probe's projection is now production.
Compile cache proven on consecutive boots of that shape: cold 'Compiling a
graph for compile range (1, 1024) takes 179.24 s' / 'torch.compile took
188.26 s'; warm 'Directly load the compiled graph(s) ... from the cache, took
2.143 s' / 'torch.compile took 3.97 s in total' / 'init engine ... took 41.13 s
(compilation: 3.98 s)'. Host dir 1.4 GB / 4,707 files. Time to /health stays
payload-bound (490 s warm), so the claim is ~184 s of inductor/AOT build per
restart, not a proportional boot cut.
Tests: 367/367.
This commit is contained in:
1 parent
1ac4d48371
commit
57f33e5289
5 files changed
+59
-6
No files matched your search
@@ -1187,4 +1187,20 @@ every restart since the cutover.
|
||||
The flag was fine both times.
|
||||
- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
|
||||
tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:**
|
||||
`'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache
|
||||
space 17.22 GiB / 7522 slots`, `swap region 13.7 GiB / 5986 slots`, `GPU KV
|
||||
cache size: 398,658 tokens … 1.99x`, pool 3,150,000,000, MNBT 1024. The
|
||||
precise overhead and the arena growth are now production numbers, not a
|
||||
probe's projection.
|
||||
- **The compile-cache fix was then proven on the same endpoint, two consecutive
|
||||
boots of the deployed shape:** cold `Compiling a graph for compile range
|
||||
(1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load
|
||||
the compiled graph(s) for compile range (1, 1024) from the cache, took
|
||||
2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile,
|
||||
create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)`. The host
|
||||
dir held 1.4 GB in 4,707 files after the cold boot. **Time to `/health` is
|
||||
dominated by the 172.8 GiB payload load** (the warm boot still needed 490 s),
|
||||
so the honest claim is ~184 s of inductor/AOT build recovered per restart —
|
||||
not a proportional boot cut.
|
||||
|
||||
Reference in new issue
Block a user