dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: pin the expert host region to 120 GiB (--expert-offload-mem)

Browse Source 
The host region was the ONE arena-plan term still autofitting per boot: with
no cap the planner binds it to MemAvailable AT BOOT minus PLE/process/api/
margin terms (boot 2026-10-05: 237.97 GiB -> 159.39 GiB usable; the same host
reads 75.4 GiB available while running). Demand is 105.48 GiB
(52.74/rank x 2 ranks, 23,040 swap-target experts); a boot where co-tenant
RAM squeezed the bound near that silently pins next-hottest experts VRAM-only
(pinned_extra warning, smaller swap region, no flag change). 120 GiB =
measured demand + 14% headroom on 246.1 GiB total. Semantics verified in the
shipped image: cap binds as the bound, exceedance only warns.

The GPU-side arena is unchanged (deterministic arithmetic over the same
constants): verify after boot that host reads '--expert-offload-mem' and
slots stay 7,762. Seam: red first (wiring + value), then green; full suite
368+1 green.
This commit is contained in:
Frank
2026-10-04 18:16:38 -07:00
parent 9a144f779e
commit c4603b130c
4 changed files with 40 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+15
View File
@@ -286,6 +286,7 @@ FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see sha
FRANK_PRIMARY_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below)
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120 --expert-offload-mem (host-region cap, explicit)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
@@ -343,6 +344,20 @@ boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
inside the image (the fork's log line says exactly that: *"Update the
registry entry if this drifts"*), not a deploy.
**The host region was the last arena-plan term still autofitting per boot —
now it is a literal (2026-10-04).** With no `--expert-offload-mem` the planner
binds the host region to `MemAvailable` AT BOOT minus `PLE 47.68 - 2 x 7.0
process - 5.0 api/engine - 5% margin`: the 2026-10-05 boot read 237.97 GiB and
offered 159.39 GiB usable, while the same host reads **75.4 GiB available while
running** — the bound tracks whatever the co-tenants hold at that moment. The
demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a
boot where the bound drops near it silently converts next-hottest experts to
VRAM-only (the `pinned_extra` warning is the only trace).
`FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
on a host with 246.1 GiB total. Semantics verified in the shipped image
(`offloader/expert_plan.py`): the cap binds as the bound, an exceedance only
warns, and the plan rewrites the field to the cap it met (+1 MiB/rank).
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25