frank-primary: pin the expert host region to 120 GiB (--expert-offload-mem)
The host region was the ONE arena-plan term still autofitting per boot: with no cap the planner binds it to MemAvailable AT BOOT minus PLE/process/api/ margin terms (boot 2026-10-05: 237.97 GiB -> 159.39 GiB usable; the same host reads 75.4 GiB available while running). Demand is 105.48 GiB (52.74/rank x 2 ranks, 23,040 swap-target experts); a boot where co-tenant RAM squeezed the bound near that silently pins next-hottest experts VRAM-only (pinned_extra warning, smaller swap region, no flag change). 120 GiB = measured demand + 14% headroom on 246.1 GiB total. Semantics verified in the shipped image: cap binds as the bound, exceedance only warns. The GPU-side arena is unchanged (deterministic arithmetic over the same constants): verify after boot that host reads '--expert-offload-mem' and slots stay 7,762. Seam: red first (wiring + value), then green; full suite 368+1 green.
This commit is contained in:
1 parent
9a144f779e
commit
c4603b130c
4 files changed
+40
No files matched your search
@@ -213,6 +213,22 @@ FRANK_PRIMARY_UTIL=0.982
|
||||
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0
|
||||
FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12
|
||||
|
||||
# HOST-region cap — the last arena-plan term that still autofit per boot. With
|
||||
# no cap the planner's host bound is MemAvailable AT BOOT minus PLE 47.68 -
|
||||
# 2 x 7.0 process - 5.0 api/engine - 5% margin (measured boot 2026-10-05:
|
||||
# MemAvailable 237.97 -> 159.39 GiB usable across 2 ranks; the demand is 105.48
|
||||
# = 52.74/rank x 2 ranks, 23,040 swap-target experts). The same host reads 75.4
|
||||
# GiB available while running — the bound tracks whatever the co-tenants hold at
|
||||
# boot, and a boot where that drops near the demand silently pins next-hottest
|
||||
# experts VRAM-only instead (smaller swap region; the pinned_extra warning is
|
||||
# the only trace). Pin 120 GiB: measured demand + 14% headroom, on a host with
|
||||
# 246.1 GiB total. If a boot ever logs "expert-offload-mem … exceeds
|
||||
# MemAvailable at plan time", THAT is the contract being broken — look at the
|
||||
# co-tenant RAM before touching this number. Plan semantics verified in the
|
||||
# shipped image (offloader/expert_plan.py): the cap binds as the bound, an
|
||||
# exceedance only warns.
|
||||
FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120
|
||||
|
||||
# --- optional engine knobs (unset == the flag is absent) ---------------------
|
||||
# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
|
||||
# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
|
||||
|
||||
@@ -156,6 +156,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
||||
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
||||
${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
|
||||
${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"} \
|
||||
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
||||
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
|
||||
Reference in new issue
Block a user