frank-primary: drop --expert-precise-memory (boot crash loop with bf16)

NRestarts climbed to 3, each attempt ~6.7 min then 'Deactivated successfully'
(API server drains after an EngineCore fatal error; Restart=always turned the
silent death into a visible loop). The arena is device-resident, so the budget
decides it: weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena
16.65 (precise overhead 5.25) + non-torch 1.45 = 31.73 of 31.86 GiB -> ~0.13
GiB free at boot, and graph capture is what allocates transiently.

Dropping the flag restores the 7.60 GiB default overhead: arena 14.31 GiB,
total ~29.4 GiB, ~2.4 GiB free. bf16 2 x 262144 (Frank's KV change) stays.
The serve script keeps the opt-in flag; the seam now asserts it is ABSENT.

scripts/frank-primary.env

@@ -70,12 +70,18 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 # Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
 # non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
 # the fp8 3-window shape used 28.27 and left 3.59).
-# PRECISE MEMORY: the registry's default overhead for this architecture is
-# 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2 nbt4096/seqs16/len262144, heavier
-# rank 5.41) PLUS 2.35 GiB of compile/graph-capture headroom. The flag selects
-# the fitted figure, which only moves the plan's `room` and therefore the arena:
-# fixed 16.91 -> 14.56 GiB, arena 6,248 -> ~7,278 slots (~2.35 GiB = ~1,025
-# slots at 2.344 MiB/slot). The risk it buys back is boot-time capture slack.
+# PRECISE MEMORY: DROPPED 2026-10-01 after it crashed the boot in combination
+# with bf16. The flag selects the fitted 5.25 GiB overhead instead of the
+# 7.60 GiB default, moving the plan's `room` and therefore the arena
+# (6,248 -> ~7,278 slots). But the arena is DEVICE-resident, and the budget
+# for this shape does not have 2.35 GiB to spare:
+#   weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena 16.65
+#   + non-torch ~1.45 = 31.73 of 31.86 GiB  =>  ~0.13 GiB free at boot
+# measured as a CrashLoop: every attempt reached graph capture, the engine
+# exited cleanly, systemd restarted it (NRestarts=3 at ~6.7 min each).
+# With the default overhead the arena is 14.31 GiB and the same shape totals
+# ~29.4 GiB, leaving ~2.4 GiB for capture. Re-enable the flag only with a
+# pool/context reduction that pays for it.
 # MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
 # 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
 FRANK_PRIMARY_MAXLEN=262144
@@ -83,7 +89,6 @@ FRANK_PRIMARY_SEQS=2
 FRANK_PRIMARY_KV_BYTES=7300000000
 FRANK_PRIMARY_KV_DTYPE=auto
 FRANK_PRIMARY_MNBT=2048
-FRANK_PRIMARY_PRECISE_MEMORY=1
 FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
 FRANK_PRIMARY_UTIL=0.98

serverAppConfigs-docs/log.md

@@ -663,6 +663,38 @@
 - `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
 - `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
 
+## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop
+
+**Reverted.** The two changes Frank asked for were applied together and the
+result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
+~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
+and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
+new `Restart=always` turned a silent death into a visible loop).
+
+The budget arithmetic explains it without needing the error text, because the
+arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
+that is exactly what `rocm-smi` measured):
+
+| line | bf16 2 x 262144 + precise | fp8 3 x 262144 |
+|---|---|---|
+| non-expert weights | 6.28 | 6.28 |
+| pinned pool | 6.80 | 5.68 |
+| CUDA graphs | 0.55 | 0.55 |
+| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) |
+| non-torch | ~1.45 | ~1.45 |
+| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) |
+| free at boot | **~0.13 GiB** | 3.59 |
+
+`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to
+the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they
+put graph capture ~0.13 GiB from the ceiling, where every transient allocation
+dies. So the flag is dropped and bf16 2 x 262144 ships with the default
+overhead: arena 14.31 GiB, ~2.4 GiB free at boot.
+
+**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by
+default — it is the budget for the phase that allocates last. Reclaim it only
+by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
+
 ## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
 
 Frank's final shape for the night. Two changes at once, both measured in

tests/frank_primary_unit.bats

@@ -110,8 +110,11 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_KV_BYTES=7300000000$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
+    # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the boot
+    # 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
+    # crash-looped at graph capture. The serve script keeps the opt-in flag.
+    ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
     ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
@@ -139,7 +142,6 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     # moves the plan's `room` term and therefore the expert arena: ~1,025
     # more slots at 2.344 MiB each.
     grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
-    grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
 }
 
 @test "tools and reasoning are wired — the client contract is served" {