frank-primary: max-possible window (241664) + reasoning/tool contract
Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.
Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.
Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
Frank
@@ -244,13 +244,78 @@ scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
|
||||
are `podman cp`'d into the container after every recreate (the serve script
|
||||
re-copies them; a recreate wipes container-local files).
|
||||
|
||||
## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)
|
||||
|
||||
Two same-day cutovers moved this page's findings into production:
|
||||
|
||||
1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
|
||||
window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
|
||||
`frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
|
||||
production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
|
||||
measured 37.4-40.6 tok/s at ~54% acceptance.
|
||||
2. **The overnight redo campaign re-measured EVERY lever at matched
|
||||
geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
|
||||
and concluded the day: the incumbent cutover to **bf16 KV, no
|
||||
speculative decode** (Frank's call).
|
||||
|
||||
### What the redo campaign measured (all at shallow depth, 166/1087/4064)
|
||||
|
||||
| lever | verdict |
|
||||
|---|---|
|
||||
| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
|
||||
| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
|
||||
| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
|
||||
| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
|
||||
| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
|
||||
| UVA | 2048 POISON (-39%); 8192 ~parity |
|
||||
| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
|
||||
| six slots | every SEQS-6 KV sizing refused at engine init |
|
||||
|
||||
Depth caveat: every number above is at a near-empty KV. Deep-context
|
||||
behavior (the gather win growing with depth, bf16's decode cost at depth)
|
||||
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
|
||||
### The bf16 no-spec incumbent (production since 2026-10-01)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=241664 x SEQS 3 (724,992 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=10184000000 PER RANK (bf16, pool scales with window)
|
||||
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
```
|
||||
|
||||
Boot-probed 2026-10-01 (transient systemd units, full-log capture): 262144
|
||||
and 245760 both died at graph capture (`Tried to allocate 76.00 MiB ...
|
||||
0 bytes free` — the pool+profile+graph budget exceeds by a hair at util
|
||||
0.98); 237568 passed (KV 718,606 tokens) and the +4096 bonus attempt
|
||||
**241664 PASSED and is production: KV 732,359 tokens / Maximum concurrency
|
||||
for 241,664 tokens per request: 3.03x** — all three full windows fit with
|
||||
a 7,367-token margin. Decode measured at the 262k-era shape before the
|
||||
ladder: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
|
||||
The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
|
||||
reservation, not a VRAM wall — no-spec bf16 takes ~9.5-10.2 GB/rank and
|
||||
boots clean. The failed 11.3 GB attempt OOMed at the engine's free-memory
|
||||
check (`RuntimeError: Worker failed with error 'CUDA out of memory'`,
|
||||
captured from the journald json stream — the fork's giant ERROR lines are
|
||||
split by journald, so grep the `-o json` output, not `-o cat`).
|
||||
|
||||
Rollback to the fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`
|
||||
(env trio + seam re-pinned there). The bf16 trade is deliberate: ~14%
|
||||
decode cost at 4064 ctx vs the fp8+gather arm (49.25 -> 36.42) bought
|
||||
clean cache + the largest window this box boots. The model's native
|
||||
262,144 exceeds what graph capture grants at util 0.98 here: the ladder
|
||||
measured 262144/245760 OOM and 241,664 as max possible (Frank's
|
||||
subtract-16k method, then one +4k bonus step).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
|
||||
(which frontier MoE fits this box; the same PCIe-expert-stream wall at
|
||||
V4.1 scale)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
|
||||
(production is the R9V fork; this page's container is the ad hoc detour)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
|
||||
fork era; production has been the tcclaviger fork since)
|
||||
- [[x4-pair-experiment]] — the same residency law measured on a narrower
|
||||
link: streaming over PCIe loses, computing in RAM does not care
|
||||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|
||||
@@ -1,17 +1,21 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-30):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-01):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both
|
||||
Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB
|
||||
of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx,
|
||||
bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and
|
||||
Dankclaw runs two llama.cpp endpoints —
|
||||
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
|
||||
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
|
||||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
|
||||
QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs
|
||||
(max possible — 262144 does not fit at a 3-seq contract on this box),
|
||||
bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints —
|
||||
gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
|
||||
Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M
|
||||
drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||||
HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
|
||||
port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -27,9 +31,9 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
podman run, frank-primary, port 7770]
|
||||
C[GLM-5.3-Flash W4A16 int4
|
||||
TP=2, expert offload 73.64 GiB/rank pinned,
|
||||
14.51 GiB LRU arena, 131072 ctx, vision, tools]
|
||||
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
|
||||
TP=2, expert offload automatic, 241664 ctx x 3,
|
||||
bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
|
||||
@@ -662,3 +662,59 @@
|
||||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
|
||||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|
||||
|
||||
## 2026-10-01 — frank-primary bf16 no-spec cutover
|
||||
|
||||
- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
|
||||
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
|
||||
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
|
||||
FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
|
||||
"KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
|
||||
request: 3.03x".
|
||||
- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
|
||||
(`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
|
||||
allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
|
||||
(KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
|
||||
PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
|
||||
model's native 262,144 does not fit with a 3-seq contract. Every rung was
|
||||
booted under a transient systemd unit at uid dank with
|
||||
`LimitMEMLOCK=infinity` (the rootless bench probe masked the production
|
||||
memory environment) with full engine-log capture.
|
||||
- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
|
||||
`--gpu-memory-utilization` shrinks the engine's own budget rather than
|
||||
buying slack; Frank forbade it and the winning ladder ran at 0.98.
|
||||
- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
|
||||
`--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
|
||||
container-env passthrough added; shape header rewritten.
|
||||
- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
|
||||
spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
|
||||
qualified-shape and flags tests flipped to the new env keys.
|
||||
- Overnight redo campaign (prefetch OFF everywhere) tabled in
|
||||
`/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
|
||||
with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
|
||||
1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
|
||||
six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
|
||||
KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
|
||||
- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
|
||||
- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
|
||||
updated out of the GLM era.
|
||||
- **Client contract fixed (thinking + tools).** The live endpoint returned a
|
||||
null `reasoning` field and dumped the thinking text INLINE in `content`
|
||||
with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
|
||||
— the malformed thinking block clients choke on — because the deployed argv
|
||||
had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
|
||||
three contract flags and the qwen rollback inherited the gap. Restored from
|
||||
`ecaad36`'s values, confirmed against this fork's registries:
|
||||
`--reasoning-parser qwen3`, `--enable-auto-tool-choice
|
||||
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
|
||||
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
|
||||
"NO tool dialect" test is flipped to a PRESENT test.
|
||||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 ->
|
||||
241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and
|
||||
`preserve_thinking` moved from a bare top-level body field to
|
||||
`chat_template_kwargs` (the form vLLM actually reads).
|
||||
- Production bring-up: the winner (241,664) boots and serves on :7770 —
|
||||
engine log verified (KV 732,359 tokens, 3.03x concurrency, MNBT 1024,
|
||||
gather 32x64). Rollback target for the old fp8+MTP4 shape: git tag
|
||||
`frank-primary-fp8-mtp-20261001`; the 237,568 fallback is documented in
|
||||
the runbook.
|
||||
|
||||
Reference in New Issue
Block a user