From c4603b130c93a3f7ed18e7a0f94480a7c6356bd4 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 4 Oct 2026 18:16:38 -0700 Subject: [PATCH] frank-primary: pin the expert host region to 120 GiB (--expert-offload-mem) The host region was the ONE arena-plan term still autofitting per boot: with no cap the planner binds it to MemAvailable AT BOOT minus PLE/process/api/ margin terms (boot 2026-10-05: 237.97 GiB -> 159.39 GiB usable; the same host reads 75.4 GiB available while running). Demand is 105.48 GiB (52.74/rank x 2 ranks, 23,040 swap-target experts); a boot where co-tenant RAM squeezed the bound near that silently pins next-hottest experts VRAM-only (pinned_extra warning, smaller swap region, no flag change). 120 GiB = measured demand + 14% headroom on 246.1 GiB total. Semantics verified in the shipped image: cap binds as the bound, exceedance only warns. The GPU-side arena is unchanged (deterministic arithmetic over the same constants): verify after boot that host reads '--expert-offload-mem' and slots stay 7,762. Seam: red first (wiring + value), then green; full suite 368+1 green. --- scripts/frank-primary.env | 16 ++++++++++++++++ scripts/serve-frank-primary.sh | 1 + ...ankserver-qwen38-fp8-expert-offload-tuning.md | 15 +++++++++++++++ tests/frank_primary_unit.bats | 8 ++++++++ 4 files changed, 40 insertions(+) diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 8ea5bc0..deb2475 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -213,6 +213,22 @@ FRANK_PRIMARY_UTIL=0.982 FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12 +# HOST-region cap — the last arena-plan term that still autofit per boot. With +# no cap the planner's host bound is MemAvailable AT BOOT minus PLE 47.68 - +# 2 x 7.0 process - 5.0 api/engine - 5% margin (measured boot 2026-10-05: +# MemAvailable 237.97 -> 159.39 GiB usable across 2 ranks; the demand is 105.48 +# = 52.74/rank x 2 ranks, 23,040 swap-target experts). The same host reads 75.4 +# GiB available while running — the bound tracks whatever the co-tenants hold at +# boot, and a boot where that drops near the demand silently pins next-hottest +# experts VRAM-only instead (smaller swap region; the pinned_extra warning is +# the only trace). Pin 120 GiB: measured demand + 14% headroom, on a host with +# 246.1 GiB total. If a boot ever logs "expert-offload-mem … exceeds +# MemAvailable at plan time", THAT is the contract being broken — look at the +# co-tenant RAM before touching this number. Plan semantics verified in the +# shipped image (offloader/expert_plan.py): the cap binds as the bound, an +# exceedance only warns. +FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120 + # --- optional engine knobs (unset == the flag is absent) --------------------- # FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds # the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh index 9455a0f..8c4d1b6 100755 --- a/scripts/serve-frank-primary.sh +++ b/scripts/serve-frank-primary.sh @@ -156,6 +156,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \ --max-num-seqs "$FRANK_PRIMARY_SEQS" \ ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \ ${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \ + ${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"} \ --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \ --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \ --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \ diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index fb8a12e..c4ca2f1 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -286,6 +286,7 @@ FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see sha FRANK_PRIMARY_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below) FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_* FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB) +FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120 --expert-offload-mem (host-region cap, explicit) FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU) FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env) @@ -343,6 +344,20 @@ boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real inside the image (the fork's log line says exactly that: *"Update the registry entry if this drifts"*), not a deploy. +**The host region was the last arena-plan term still autofitting per boot — +now it is a literal (2026-10-04).** With no `--expert-offload-mem` the planner +binds the host region to `MemAvailable` AT BOOT minus `PLE 47.68 - 2 x 7.0 +process - 5.0 api/engine - 5% margin`: the 2026-10-05 boot read 237.97 GiB and +offered 159.39 GiB usable, while the same host reads **75.4 GiB available while +running** — the bound tracks whatever the co-tenants hold at that moment. The +demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a +boot where the bound drops near it silently converts next-hottest experts to +VRAM-only (the `pinned_extra` warning is the only trace). +`FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom, +on a host with 246.1 GiB total. Semantics verified in the shipped image +(`offloader/expert_plan.py`): the cap binds as the bound, an exceedance only +warns, and the plan rewrites the field to the cap it met (+1 MiB/rank). + **MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25 diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index fc06b46..b0f42a0 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -188,6 +188,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE" grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE" + # The host region is the ONE plan term that autofits per-boot: with no cap + # the bound is MemAvailable at that moment (boot 2026-10-05: 237.97 GiB -> + # 159.39 GiB usable, while the running host later reads 75.4 GiB available). + # An explicit --expert-offload-mem cap makes the plan deterministic whatever + # the co-tenant RAM state is at boot; 120 GiB covers the measured 105.48 + # GiB of swappable experts (52.74/rank x 2) with 14% headroom. + grep -q -- '${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"}' "$SERVE" + grep -q '^FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120$' "$ENV_FILE" } @test "tools and reasoning are wired — the client contract is served" {