dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: 2 bf16 windows + --expert-precise-memory

Browse Source 
Both of Frank's changes at once.

KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.

--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
 term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.

Seam + docs + AGENTS row + log updated.
This commit is contained in:
Frank
2026-10-01 13:52:13 -07:00
parent 3f1860aa5f
commit 389b605baf
6 changed files with 82 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/log.md
+25
View File
@@ -663,6 +663,31 @@
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
Frank's final shape for the night. Two changes at once, both measured in
principle before the boot:
- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
- **`--expert-precise-memory`.** The registry's default overhead for this
architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
The flag selects the fitted figure, which moves only the plan's `room` term
— arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
1024 nbt above 4096, so MNBT 2048 pays nothing.
Budget note worth keeping: the arena is `(budget - non_expert_weights -
kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
planned arena; it comes out of free VRAM instead.
## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the