diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 7486625..372497e 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -70,12 +70,18 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" # Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55, # non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis: # the fp8 3-window shape used 28.27 and left 3.59). -# PRECISE MEMORY: the registry's default overhead for this architecture is -# 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2 nbt4096/seqs16/len262144, heavier -# rank 5.41) PLUS 2.35 GiB of compile/graph-capture headroom. The flag selects -# the fitted figure, which only moves the plan's `room` and therefore the arena: -# fixed 16.91 -> 14.56 GiB, arena 6,248 -> ~7,278 slots (~2.35 GiB = ~1,025 -# slots at 2.344 MiB/slot). The risk it buys back is boot-time capture slack. +# PRECISE MEMORY: DROPPED 2026-10-01 after it crashed the boot in combination +# with bf16. The flag selects the fitted 5.25 GiB overhead instead of the +# 7.60 GiB default, moving the plan's `room` and therefore the arena +# (6,248 -> ~7,278 slots). But the arena is DEVICE-resident, and the budget +# for this shape does not have 2.35 GiB to spare: +# weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena 16.65 +# + non-torch ~1.45 = 31.73 of 31.86 GiB => ~0.13 GiB free at boot +# measured as a CrashLoop: every attempt reached graph capture, the engine +# exited cleanly, systemd restarted it (NRestarts=3 at ~6.7 min each). +# With the default overhead the arena is 14.31 GiB and the same shape totals +# ~29.4 GiB, leaving ~2.4 GiB for capture. Re-enable the flag only with a +# pool/context reduction that pays for it. # MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so # 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096. FRANK_PRIMARY_MAXLEN=262144 @@ -83,7 +89,6 @@ FRANK_PRIMARY_SEQS=2 FRANK_PRIMARY_KV_BYTES=7300000000 FRANK_PRIMARY_KV_DTYPE=auto FRANK_PRIMARY_MNBT=2048 -FRANK_PRIMARY_PRECISE_MEMORY=1 FRANK_PRIMARY_GATHER_CHUNKS=32 FRANK_PRIMARY_GATHER_LANES=64 FRANK_PRIMARY_UTIL=0.98 diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 5e9c828..da38cbe 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -663,6 +663,38 @@ - `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage. - `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership. +## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop + +**Reverted.** The two changes Frank asked for were applied together and the +result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running +~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains +and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the +new `Restart=always` turned a silent death into a visible loop). + +The budget arithmetic explains it without needing the error text, because the +arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and +that is exactly what `rocm-smi` measured): + +| line | bf16 2 x 262144 + precise | fp8 3 x 262144 | +|---|---|---| +| non-expert weights | 6.28 | 6.28 | +| pinned pool | 6.80 | 5.68 | +| CUDA graphs | 0.55 | 0.55 | +| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) | +| non-torch | ~1.45 | ~1.45 | +| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) | +| free at boot | **~0.13 GiB** | 3.59 | + +`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to +the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they +put graph capture ~0.13 GiB from the ceiling, where every transient allocation +dies. So the flag is dropped and bf16 2 x 262144 ships with the default +overhead: arena 14.31 GiB, ~2.4 GiB free at boot. + +**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by +default — it is the budget for the phase that allocates last. Reclaim it only +by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context). + ## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory Frank's final shape for the night. Two changes at once, both measured in diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 745bb18..35cc95d 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -110,8 +110,11 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_KV_BYTES=7300000000$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE" + # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the boot + # 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and + # crash-looped at graph capture. The serve script keeps the opt-in flag. + ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE" ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE" @@ -139,7 +142,6 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # moves the plan's `room` term and therefore the expert arena: ~1,025 # more slots at 2.344 MiB each. grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE" - grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE" } @test "tools and reasoning are wired — the client contract is served" {