frank-primary: 2 bf16 windows + --expert-precise-memory
Both of Frank's changes at once. KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144). bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280 tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's 6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank. --expert-precise-memory selects the fitted overhead (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so MNBT 2048 pays nothing. Seam + docs + AGENTS row + log updated.
This commit is contained in:
1 parent
3f1860aa5f
commit
389b605baf
6 files changed
+82
-41
No files matched your search
@@ -278,10 +278,12 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 3 (786,432 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=6100000000 PER RANK (fp8, pinned to the CONTRACT
|
||||
FLOOR so the remainder feeds the arena)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together)
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 2 (524,288 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=7300000000 PER RANK (bf16 = auto; 13,839 B/token)
|
||||
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (fitted overhead
|
||||
5.25 GiB instead of the 7.60 GiB default,
|
||||
so ~1,025 more arena slots)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
@@ -300,7 +302,8 @@ memory environment). The night searched two concurrency contracts:
|
||||
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
|
||||
| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
|
||||
| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
|
||||
| **3 × 262144, fp8, pool at the floor** | **DEPLOYED** (Frank's call): 786,432 tokens of contract; pool pinned at 6,100,000,000 B/rank (the measured 7,698 B/token floor is 6.054e9) so the ~2.2 GiB difference versus the 4-slot shape goes to the expert ARENA instead of unused KV |
|
||||
| 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured |
|
||||
| **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots |
|
||||
|
||||
Boot is NOT proof of runtime survivability: the five-slot shape passed every
|
||||
boot gate and still died on the first long-generation request. After any
|
||||
|
||||
@@ -663,6 +663,31 @@
|
||||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|
||||
|
||||
## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
|
||||
|
||||
Frank's final shape for the night. Two changes at once, both measured in
|
||||
principle before the boot:
|
||||
|
||||
- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
|
||||
bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
|
||||
11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
|
||||
1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
|
||||
524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
|
||||
3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
|
||||
- **`--expert-precise-memory`.** The registry's default overhead for this
|
||||
architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
|
||||
nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
|
||||
The flag selects the fitted figure, which moves only the plan's `room` term
|
||||
— arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
|
||||
cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
|
||||
1024 nbt above 4096, so MNBT 2048 pays nothing.
|
||||
|
||||
Budget note worth keeping: the arena is `(budget - non_expert_weights -
|
||||
kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
|
||||
planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
|
||||
with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
|
||||
planned arena; it comes out of free VRAM instead.
|
||||
|
||||
## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
|
||||
|
||||
`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
|
||||
|
||||
Reference in new issue
Block a user