frank-primary: 5 x full 262144 contexts at fp8 (final shape)
Frank's call after the night ladder: five contexts at the model's full native 262,144 window beat six shortened ones. Six full-length contexts do NOT fit at any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the 6-seq graph/activation footprint leaves too little for the pool); 5 x 262144 at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured 'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens per request: 5.15x' (39,462 tokens of margin). Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144) re-pinned.
This commit is contained in:
Frank
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **241664 ctx × 3 seqs** at KV_BYTES 10,184,000,000/rank (measured pool 732,359 tokens / 3.03× concurrency), util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. 241,664 is **max possible** on this box (Frank's subtract-16k ladder: 262144/245760 OOM at graph capture, 237568 passed, +4096 bonus step won). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **5 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 10,388,766,720/rank (measured pool 1,350,182 tokens / 5.15× concurrency), util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Six full-length contexts do **not** fit (6×241664 asked 10.70 GiB vs 9.18 GiB free); 5×262144 was the first rung that passed (Frank's subtract-16k ladder). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
|
||||
@@ -60,14 +60,19 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape (2026-10-01 night ladder, Frank's method) ---------
|
||||
# MAXLEN ladder per Frank: 262144 OOMed in graph capture (76 MiB short),
|
||||
# 245760 OOMed (0 bytes free at graph capture), 237568 PASSED (718,606
|
||||
# tokens / 3.0x), then the +4096 bonus attempt 241664 PASSED and is
|
||||
# PRODUCTION (KV 732,359 tokens / 3.03x at 241,664-token requests).
|
||||
FRANK_PRIMARY_MAXLEN=241664
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
FRANK_PRIMARY_KV_BYTES=10184000000
|
||||
FRANK_PRIMARY_KV_DTYPE=auto
|
||||
# SEQS 5 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape (Frank's call: five
|
||||
# contexts of the model's FULL native window beats six shortened ones).
|
||||
# Measured: "GPU KV cache size: 1,350,182 tokens, Maximum concurrency for
|
||||
# 262,144 tokens per request: 5.15x" — 39,462 tokens of margin over the
|
||||
# 1,310,720-token contract.
|
||||
# The night's ladder, both contracts: at SEQS=3 bf16, 262144/245760 OOMed at
|
||||
# graph capture and 241664 was max possible (KV 732,359 / 3.03x). At SEQS=6
|
||||
# fp8, 6 x 241664 OOMed (tried 10.70 GiB with 9.18 GiB free) — six
|
||||
# full-length contexts do NOT fit on this box; five do.
|
||||
FRANK_PRIMARY_MAXLEN=262144
|
||||
FRANK_PRIMARY_SEQS=5
|
||||
FRANK_PRIMARY_KV_BYTES=10388766720
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_MNBT=1024
|
||||
FRANK_PRIMARY_GATHER_CHUNKS=32
|
||||
FRANK_PRIMARY_GATHER_LANES=64
|
||||
|
||||
@@ -275,39 +275,46 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
|
||||
behavior (the gather win growing with depth, bf16's decode cost at depth)
|
||||
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
|
||||
### The bf16 no-spec incumbent (production since 2026-10-01)
|
||||
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=241664 x SEQS 3 (724,992 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=10184000000 PER RANK (bf16, pool scales with window)
|
||||
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 5 (1,310,720 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=10388766720 PER RANK (fp8, pool scales with windows)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
```
|
||||
|
||||
Boot-probed 2026-10-01 (transient systemd units, full-log capture): 262144
|
||||
and 245760 both died at graph capture (`Tried to allocate 76.00 MiB ...
|
||||
0 bytes free` — the pool+profile+graph budget exceeds by a hair at util
|
||||
0.98); 237568 passed (KV 718,606 tokens) and the +4096 bonus attempt
|
||||
**241664 PASSED and is production: KV 732,359 tokens / Maximum concurrency
|
||||
for 241,664 tokens per request: 3.03x** — all three full windows fit with
|
||||
a 7,367-token margin. Decode measured at the 262k-era shape before the
|
||||
ladder: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
|
||||
The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
|
||||
reservation, not a VRAM wall — no-spec bf16 takes ~9.5-10.2 GB/rank and
|
||||
boots clean. The failed 11.3 GB attempt OOMed at the engine's free-memory
|
||||
check (`RuntimeError: Worker failed with error 'CUDA out of memory'`,
|
||||
captured from the journald json stream — the fork's giant ERROR lines are
|
||||
split by journald, so grep the `-o json` output, not `-o cat`).
|
||||
Boot-probed 2026-10-01 (transient systemd units at the production uid +
|
||||
memlock, full-log capture; the rootless bench probe had masked the production
|
||||
memory environment). The night searched two concurrency contracts:
|
||||
|
||||
Rollback to the fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`
|
||||
(env trio + seam re-pinned there). The bf16 trade is deliberate: ~14%
|
||||
decode cost at 4064 ctx vs the fp8+gather arm (49.25 -> 36.42) bought
|
||||
clean cache + the largest window this box boots. The model's native
|
||||
262,144 exceeds what graph capture grants at util 0.98 here: the ladder
|
||||
measured 262144/245760 OOM and 241,664 as max possible (Frank's
|
||||
subtract-16k method, then one +4k bonus step).
|
||||
| rung | result |
|
||||
|---|---|
|
||||
| 3 × 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
|
||||
| 3 × 245760, bf16 | **OOM identically** |
|
||||
| 3 × 237568, bf16 | PASS — KV 718,606 tokens |
|
||||
| **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) |
|
||||
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
|
||||
| **5 × 262144, fp8** | **PASS on the first rung — KV 1,350,182 tokens, 5.15x; DEPLOYED** (39,462-token margin over the 1,310,720-token contract) |
|
||||
|
||||
The final shape is Frank's call: **five contexts of the model's full native
|
||||
262,144 window** beat six shortened ones. fp8 KV is what buys the capacity —
|
||||
7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
|
||||
decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
|
||||
graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144
|
||||
needs 12.47 GB even at fp8.
|
||||
|
||||
The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
|
||||
reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
|
||||
reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
|
||||
which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
|
||||
read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
|
||||
only available while the container lives).
|
||||
|
||||
Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
|
||||
`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -4,9 +4,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-01):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs
|
||||
(max possible — 262144 does not fit at a 3-seq contract on this box),
|
||||
bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **5 seqs x 262144 ctx
|
||||
(the model's full native window)**, fp8 KV (6 full-length contexts do not
|
||||
fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
@@ -32,8 +32,8 @@ gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
|
||||
TP=2, expert offload automatic, 241664 ctx x 3,
|
||||
bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
TP=2, expert offload automatic, 5 x 262144 ctx,
|
||||
fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
|
||||
@@ -709,8 +709,17 @@
|
||||
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
|
||||
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
|
||||
"NO tool dialect" test is flipped to a PRESENT test.
|
||||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 ->
|
||||
241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and
|
||||
- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
|
||||
fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
|
||||
free) — six full-length contexts do not fit at any KV dtype — so the
|
||||
contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
|
||||
rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
|
||||
5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
|
||||
three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
|
||||
same night) is the documented max at 3 contexts.
|
||||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
|
||||
with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
|
||||
<= 262,144) and
|
||||
`preserve_thinking` moved from a bare top-level body field to
|
||||
`chat_template_kwargs` (the form vLLM actually reads).
|
||||
- Production bring-up: the winner (241,664) boots and serves on :7770 —
|
||||
|
||||
@@ -75,20 +75,20 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
}
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-10-01 night ladder (Frank's method: subtract until it boots, then
|
||||
# one bonus step): 262144 OOMed at graph capture (76 MiB short, 0 bytes
|
||||
# free), 245760 OOMed identically, 237568 PASSED (KV 718,606 tokens /
|
||||
# 3.0x), and the +4096 bonus attempt 241664 PASSED and IS production
|
||||
# (KV 732,359 tokens / 3.03x at 241,664-token requests, 7,367-token
|
||||
# margin over the 3-window contract). UTIL stays 0.98 — the production
|
||||
# unit's pinned-expert path needs the workspace the bench probe hid;
|
||||
# 0.98 with the smaller window fits. KV_BYTES 10,184,000,000 scales the
|
||||
# pool with the window.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=241664$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
# 2026-10-01 night ladder (Frank's method). Two contracts were searched:
|
||||
# SEQS=3 bf16 topped out at 241664 (262144/245760 OOMed at graph capture;
|
||||
# KV 732,359 tokens / 3.03x). SEQS=6 fp8 did NOT fit — 6 x 241664 tried
|
||||
# 10.70 GiB against 9.18 GiB free, so six full-length contexts are
|
||||
# impossible here. SEQS=5 fp8 at the model's FULL native 262144 window
|
||||
# PASSED on the first rung and is DEPLOYED: KV 1,350,182 tokens,
|
||||
# Maximum concurrency 5.15x, 39,462 tokens of margin over the
|
||||
# 1,310,720-token contract. UTIL stays 0.98 (lowering it shrinks the
|
||||
# engine's own budget instead of buying slack).
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=5$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=10184000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=10388766720$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
@@ -151,13 +151,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
! grep -q -- '--speculative-config' "$SERVE"
|
||||
}
|
||||
|
||||
@test "bf16 KV is PRESENT — clean cache won the trade" {
|
||||
# --kv-cache-dtype auto = bf16 attention KV. Doubles attention-KV
|
||||
# bytes/token (measured 14,047 vs 7,926) and costs ~14% decode vs fp8
|
||||
# at 4064 ctx (49.25 -> 36.42 with the gather win) — traded deliberately
|
||||
# for clean cache and 262k windows (Frank's call). The fp8 shape is
|
||||
# restorable from git tag frank-primary-fp8-mtp-20261001.
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
@test "fp8 KV is PRESENT — capacity won over the bf16 decode edge" {
|
||||
# --kv-cache-dtype fp8 (the stock flag quantizes K and V together).
|
||||
# Capacity is what buys the five full 262,144-token contexts: fp8 costs
|
||||
# 7,926 B/token against bf16's ~13,900-14,047, so 5x262144 fits in
|
||||
# 10,388,766,720 B/rank (KV 1,350,182 tokens / 5.15x) where bf16 topped
|
||||
# out at three 241,664-token contexts. The bf16 ~4% decode edge was
|
||||
# traded away for it (Frank's call, 2026-10-01).
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user