frank-primary: 2 bf16 windows + --expert-precise-memory
Both of Frank's changes at once. KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144). bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280 tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's 6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank. --expert-precise-memory selects the fitted overhead (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so MNBT 2048 pays nothing. Seam + docs + AGENTS row + log updated.
This commit is contained in:
Frank
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **3 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 6,100,000,000/rank — the pool pinned to the contract FLOOR so everything else feeds the expert arena — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Five full-length contexts BOOT but leave ~0 bytes free at runtime — a real client request (995-token prompt, `max_tokens=261149`) killed the worker with `torch.OutOfMemoryError ... 0 bytes free`; six never fit at all (6×241664 asked 10.70 GiB vs 9.18 GiB free). Three slots with a floor-sized pool is the deployed shape. DCP cannot help here: it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
|
||||
| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 4 slots of 151,552, bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |
|
||||
|
||||
|
||||
+23
-20
@@ -59,28 +59,31 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# hoc experiment identity (both hold the GPUs / name registrations).
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape (2026-10-01, final: 3 windows + max arena) --------
|
||||
# SEQS 3 x MAXLEN 262144 at fp8 KV (Frank's call, 2026-10-01): three contexts
|
||||
# of the model's FULL native window, with the KV pool pinned to the FLOOR the
|
||||
# contract needs so that everything else goes to the expert ARENA — the
|
||||
# remainder is what the fork's planner gives the decode-critical expert cache.
|
||||
# 3 x 262144 = 786,432 tokens; measured fp8 rate 7,698 B/token (from the
|
||||
# 4-slot boot: 8,311,013,376 B -> 1,079,582 tokens, and the 5-slot boot:
|
||||
# 10,388,766,720 B -> 1,350,182 tokens) => 6.054e9 B is the floor. Pinned at
|
||||
# 6,100,000,000 => ~792k tokens, a ~5.9k-token margin, and ~2.2 GiB returned
|
||||
# to the arena versus the 4-slot shape.
|
||||
# History in this shape: 5 x 262144 booted (KV 1,350,182 / 5.15x) but left 0
|
||||
# bytes free and a real request (995-token prompt, max_tokens=261149) killed
|
||||
# the worker; 4 x 262144 (KV 1,079,582 / 4.12x) was the runtime-survivable
|
||||
# stopgap. Six full-length contexts never fit (6 x 241664 asked 10.70 GiB
|
||||
# against 9.18 GiB free).
|
||||
# MNBT 2048 (Frank's call): the arena sweep measured it at the same 6,499-slot
|
||||
# ceiling as <=4096, so the bigger batch costs the arena nothing.
|
||||
# --- qualified engine shape (2026-10-01, final: 2 bf16 windows + precise mem) --
|
||||
# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
|
||||
# the model's FULL native window with UNQUANTIZED KV, plus
|
||||
# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
|
||||
# Pool: bf16 costs 13,839 B/token (measured: 11,047,010,304 B -> 798,280 tokens
|
||||
# on the 3x241664 bf16 boot; decomposes as 12,276 attention + 1,563 mamba/side
|
||||
# caches, against fp8's 6,138 + 1,563 = 7,701). 2 x 262144 = 524,288 tokens
|
||||
# -> 7.256e9 B floor, pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin).
|
||||
# Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
|
||||
# non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
|
||||
# the fp8 3-window shape used 28.27 and left 3.59).
|
||||
# PRECISE MEMORY: the registry's default overhead for this architecture is
|
||||
# 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2 nbt4096/seqs16/len262144, heavier
|
||||
# rank 5.41) PLUS 2.35 GiB of compile/graph-capture headroom. The flag selects
|
||||
# the fitted figure, which only moves the plan's `room` and therefore the arena:
|
||||
# fixed 16.91 -> 14.56 GiB, arena 6,248 -> ~7,278 slots (~2.35 GiB = ~1,025
|
||||
# slots at 2.344 MiB/slot). The risk it buys back is boot-time capture slack.
|
||||
# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
|
||||
# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
|
||||
FRANK_PRIMARY_MAXLEN=262144
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
FRANK_PRIMARY_KV_BYTES=6100000000
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_SEQS=2
|
||||
FRANK_PRIMARY_KV_BYTES=7300000000
|
||||
FRANK_PRIMARY_KV_DTYPE=auto
|
||||
FRANK_PRIMARY_MNBT=2048
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1
|
||||
FRANK_PRIMARY_GATHER_CHUNKS=32
|
||||
FRANK_PRIMARY_GATHER_LANES=64
|
||||
FRANK_PRIMARY_UTIL=0.98
|
||||
|
||||
@@ -115,6 +115,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
||||
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
||||
${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
|
||||
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
||||
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
|
||||
@@ -278,10 +278,12 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 3 (786,432 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=6100000000 PER RANK (fp8, pinned to the CONTRACT
|
||||
FLOOR so the remainder feeds the arena)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together)
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 2 (524,288 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=7300000000 PER RANK (bf16 = auto; 13,839 B/token)
|
||||
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (fitted overhead
|
||||
5.25 GiB instead of the 7.60 GiB default,
|
||||
so ~1,025 more arena slots)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
@@ -300,7 +302,8 @@ memory environment). The night searched two concurrency contracts:
|
||||
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
|
||||
| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
|
||||
| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
|
||||
| **3 × 262144, fp8, pool at the floor** | **DEPLOYED** (Frank's call): 786,432 tokens of contract; pool pinned at 6,100,000,000 B/rank (the measured 7,698 B/token floor is 6.054e9) so the ~2.2 GiB difference versus the 4-slot shape goes to the expert ARENA instead of unused KV |
|
||||
| 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured |
|
||||
| **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots |
|
||||
|
||||
Boot is NOT proof of runtime survivability: the five-slot shape passed every
|
||||
boot gate and still died on the first long-generation request. After any
|
||||
|
||||
@@ -663,6 +663,31 @@
|
||||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|
||||
|
||||
## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
|
||||
|
||||
Frank's final shape for the night. Two changes at once, both measured in
|
||||
principle before the boot:
|
||||
|
||||
- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
|
||||
bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
|
||||
11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
|
||||
1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
|
||||
524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
|
||||
3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
|
||||
- **`--expert-precise-memory`.** The registry's default overhead for this
|
||||
architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
|
||||
nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
|
||||
The flag selects the fitted figure, which moves only the plan's `room` term
|
||||
— arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
|
||||
cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
|
||||
1024 nbt above 4096, so MNBT 2048 pays nothing.
|
||||
|
||||
Budget note worth keeping: the arena is `(budget - non_expert_weights -
|
||||
kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
|
||||
planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
|
||||
with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
|
||||
planned arena; it comes out of free VRAM instead.
|
||||
|
||||
## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
|
||||
|
||||
`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
|
||||
|
||||
@@ -100,16 +100,18 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# deployed shape: ~1.93 GiB more headroom per rank for
|
||||
# activations/graphs. UTIL stays 0.98 (lowering it shrinks the engine's
|
||||
# own budget instead of buying slack).
|
||||
# 3 x 262144 = 786,432 tokens; at the measured 7,698 B/token the pool
|
||||
# floor is 6.054e9 B, pinned at 6.1e9 so the margin is ~5.9k tokens and
|
||||
# EVERYTHING ELSE goes to the expert arena (the decode-critical expert
|
||||
# cache is the remainder after weights, pool and workspace).
|
||||
# FINAL (2026-10-01, Frank): TWO full-size windows with BF16 KV plus
|
||||
# --expert-precise-memory. bf16 costs 13,839 B/token (measured), so
|
||||
# 2 x 262144 = 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527.5k
|
||||
# tokens, 3,206 margin). The precise flag drops the plan's overhead
|
||||
# 7.60 -> 5.25 GiB, moving the arena from 6,248 to ~7,278 slots.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=7300000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
|
||||
@@ -132,6 +134,12 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
|
||||
grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
|
||||
! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
|
||||
# --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
|
||||
# default (7.60 GiB = fitted + compile/graph-capture headroom), which only
|
||||
# moves the plan's `room` term and therefore the expert arena: ~1,025
|
||||
# more slots at 2.344 MiB each.
|
||||
grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "tools and reasoning are wired — the client contract is served" {
|
||||
@@ -175,14 +183,15 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
! grep -q -- '--speculative-config' "$SERVE"
|
||||
}
|
||||
|
||||
@test "fp8 KV is PRESENT — capacity won over the bf16 decode edge" {
|
||||
# --kv-cache-dtype fp8 (the stock flag quantizes K and V together).
|
||||
# Capacity is what buys the five full 262,144-token contexts: fp8 costs
|
||||
# 7,926 B/token against bf16's ~13,900-14,047, so 5x262144 fits in
|
||||
# 10,388,766,720 B/rank (KV 1,350,182 tokens / 5.15x) where bf16 topped
|
||||
# out at three 241,664-token contexts. The bf16 ~4% decode edge was
|
||||
# traded away for it (Frank's call, 2026-10-01).
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
@test "KV dtype is bf16 (auto) — exactness, paid for in pool bytes" {
|
||||
# --kv-cache-dtype auto = bf16 (unquantized). Frank's call 2026-10-01.
|
||||
# bf16 costs 13,839 B/token against fp8's 7,701 — decomposed as 12,276
|
||||
# attention + 1,563 mamba/side (dtype-independent) versus fp8's 6,138 +
|
||||
# 1,563. That halves the pool capacity per byte, which is why the contract
|
||||
# is two full-size windows instead of three; in exchange the KV is exact
|
||||
# (no quantization error) and measured +4% decode at shallow depth, where
|
||||
# fp8's dequant cost dominates over the extra bytes read.
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user