frank-primary: 2 bf16 windows + --expert-precise-memory

Both of Frank's changes at once.

KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.

--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
 term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.

Seam + docs + AGENTS row + log updated.
This commit is contained in:
Frank committed 2026-10-01 13:52:13 -07:00
1 parent 3f1860aa5f
commit 389b605baf
6 files changed
+82 -41

No files matched your search

@@ -278,10 +278,12 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
```
FRANK_PRIMARY_MAXLEN=262144 x SEQS 3 (786,432 tokens of contract)
FRANK_PRIMARY_KV_BYTES=6100000000 PER RANK (fp8, pinned to the CONTRACT
FLOOR so the remainder feeds the arena)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together)
FRANK_PRIMARY_MAXLEN=262144 x SEQS 2 (524,288 tokens of contract)
FRANK_PRIMARY_KV_BYTES=7300000000 PER RANK (bf16 = auto; 13,839 B/token)
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (fitted overhead
5.25 GiB instead of the 7.60 GiB default,
so ~1,025 more arena slots)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
@@ -300,7 +302,8 @@ memory environment). The night searched two concurrency contracts:
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
| **3 × 262144, fp8, pool at the floor** | **DEPLOYED** (Frank's call): 786,432 tokens of contract; pool pinned at 6,100,000,000 B/rank (the measured 7,698 B/token floor is 6.054e9) so the ~2.2 GiB difference versus the 4-slot shape goes to the expert ARENA instead of unused KV |
| 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured |
| **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots |
Boot is NOT proof of runtime survivability: the five-slot shape passed every
boot gate and still died on the first long-generation request. After any
+25
View File
@@ -663,6 +663,31 @@
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
Frank's final shape for the night. Two changes at once, both measured in
principle before the boot:
- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
- **`--expert-precise-memory`.** The registry's default overhead for this
architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
The flag selects the fitted figure, which moves only the plan's `room` term
— arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
1024 nbt above 4096, so MNBT 2048 pays nothing.
Budget note worth keeping: the arena is `(budget - non_expert_weights -
kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
planned arena; it comes out of free VRAM instead.
## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the