diff --git a/AGENTS.md b/AGENTS.md index f194b5e..12e093f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 | +| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **241664 ctx × 3 seqs** at KV_BYTES 10,184,000,000/rank (measured pool 732,359 tokens / 3.03× concurrency), util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. 241,664 is **max possible** on this box (Frank's subtract-16k ladder: 262144/245760 OOM at graph capture, 237568 passed, +4096 bonus step won). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 | | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index a8a14f8..5189427 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -8,12 +8,12 @@ # restorable from the tag. # # Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited): -# * MAXLEN 262144 x SEQS 3 (786,432 tokens) at bf16 KV boots at -# KV_BYTES 11,047,010,304 PER RANK: the engine measured "KV cache size: -# 798,280 tokens; Maximum concurrency for 262,144 tokens per request: -# 3.05x". The pool holds all three full windows with an 11,848-token -# margin. Measured decode at the winner shape: 39.38/38.79/36.42 tok/s -# (166/1087/4064 ctx). +# * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED +# shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK, +# engine measured "KV cache size: 732,359 tokens; Maximum concurrency +# for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed +# at graph capture (76 MiB short / 0 bytes free); 237568 passed at +# 718,606 tokens and the +4096 bonus step passed and won. # * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was # the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because # the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes @@ -59,12 +59,32 @@ FRANK_PRIMARY_PUBLISH=7770:7770 # hoc experiment identity (both hold the GPUs / name registrations). FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" -# --- qualified engine shape --------------------------------------------------- -FRANK_PRIMARY_MAXLEN=262144 +# --- qualified engine shape (2026-10-01 night ladder, Frank's method) --------- +# MAXLEN ladder per Frank: 262144 OOMed in graph capture (76 MiB short), +# 245760 OOMed (0 bytes free at graph capture), 237568 PASSED (718,606 +# tokens / 3.0x), then the +4096 bonus attempt 241664 PASSED and is +# PRODUCTION (KV 732,359 tokens / 3.03x at 241,664-token requests). +FRANK_PRIMARY_MAXLEN=241664 FRANK_PRIMARY_SEQS=3 -FRANK_PRIMARY_KV_BYTES=11047010304 +FRANK_PRIMARY_KV_BYTES=10184000000 FRANK_PRIMARY_KV_DTYPE=auto FRANK_PRIMARY_MNBT=1024 FRANK_PRIMARY_GATHER_CHUNKS=32 FRANK_PRIMARY_GATHER_LANES=64 -FRANK_PRIMARY_UTIL=0.90 +FRANK_PRIMARY_UTIL=0.98 + +# --- client contract: reasoning split, tools, preserve_thinking ---------------- +# Without --reasoning-parser the engine leaves the thinking text INLINE in +# `content` with a dangling `` and a null `reasoning` field (measured +# 2026-10-01: content = 'We need answer...Brief.\n\n\n391') — the +# malformed thinking block clients choke on. The qwen3 parser splits it +# properly. The GLM rollback (d91379d) had dropped all three of these; the +# working values are the ones ecaad36 shipped for this model. +FRANK_PRIMARY_REASONING_PARSER=qwen3 +# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes +# tool_calls work here (without it, tool_choice auto returns 400). +FRANK_PRIMARY_TOOL_PARSER=qwen3_xml +# preserve_thinking keeps prior thinking blocks in the rendered prompt across +# turns; enable_thinking makes thinking the server-side default. Verified the +# payload's tokenizer_config template consumes both keys. +FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}' diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh index aea16ad..073f7af 100755 --- a/scripts/serve-frank-primary.sh +++ b/scripts/serve-frank-primary.sh @@ -47,7 +47,10 @@ set +a : "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}" : "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}" : "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}" -: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}" +: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}" +: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}" +: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}" +: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}" # Capture podman's output BEFORE grepping: under `set -o pipefail`, the # classic `podman images | grep -q` pattern races — grep exits on its first # match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail @@ -115,6 +118,9 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \ --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \ --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \ --enable-prefix-caching \ + --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \ + --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \ + --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \ --trust-remote-code \ > "$LOG" 2>&1 diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index a9ab7db..283330b 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -244,13 +244,78 @@ scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh` are `podman cp`'d into the container after every recreate (the serve script re-copies them; a recreate wipes container-local files). +## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec) + +Two same-day cutovers moved this page's findings into production: + +1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM + window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag + `frank-primary-fp8-mtp-20261001` is the last commit of that shape). The + production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off — + measured 37.4-40.6 tok/s at ~54% acceptance. +2. **The overnight redo campaign re-measured EVERY lever at matched + geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline) + and concluded the day: the incumbent cutover to **bf16 KV, no + speculative decode** (Frank's call). + +### What the redo campaign measured (all at shallow depth, 166/1087/4064) + +| lever | verdict | +|---|---| +| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed | +| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak | +| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) | +| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed | +| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** | +| UVA | 2048 POISON (-39%); 8192 ~parity | +| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below | +| six slots | every SEQS-6 KV sizing refused at engine init | + +Depth caveat: every number above is at a near-empty KV. Deep-context +behavior (the gather win growing with depth, bf16's decode cost at depth) +is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap. + +### The bf16 no-spec incumbent (production since 2026-10-01) + +``` +FRANK_PRIMARY_MAXLEN=241664 x SEQS 3 (724,992 tokens of contract) +FRANK_PRIMARY_KV_BYTES=10184000000 PER RANK (bf16, pool scales with window) +FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized) +FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024 +R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env) +NO --speculative-config, NO --expert-prefetch +``` + +Boot-probed 2026-10-01 (transient systemd units, full-log capture): 262144 +and 245760 both died at graph capture (`Tried to allocate 76.00 MiB ... +0 bytes free` — the pool+profile+graph budget exceeds by a hair at util +0.98); 237568 passed (KV 718,606 tokens) and the +4096 bonus attempt +**241664 PASSED and is production: KV 732,359 tokens / Maximum concurrency +for 241,664 tokens per request: 3.03x** — all three full windows fit with +a 7,367-token margin. Decode measured at the 262k-era shape before the +ladder: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx). +The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE +reservation, not a VRAM wall — no-spec bf16 takes ~9.5-10.2 GB/rank and +boots clean. The failed 11.3 GB attempt OOMed at the engine's free-memory +check (`RuntimeError: Worker failed with error 'CUDA out of memory'`, +captured from the journald json stream — the fork's giant ERROR lines are +split by journald, so grep the `-o json` output, not `-o cat`). + +Rollback to the fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001` +(env trio + seam re-pinned there). The bf16 trade is deliberate: ~14% +decode cost at 4064 ctx vs the fp8+gather arm (49.25 -> 36.42) bought +clean cache + the largest window this box boots. The model's native +262,144 exceeds what graph capture grants at util 0.98 here: the ladder +measured 262144/245760 OOM and 241,664 as max possible (Frank's +subtract-16k method, then one +4k bonus step). + ## See Also - [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page (which frontier MoE fits this box; the same PCIe-expert-stream wall at V4.1 scale) -- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary - (production is the R9V fork; this page's container is the ad hoc detour) +- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V + fork era; production has been the tcclaviger fork since) - [[x4-pair-experiment]] — the same residency law measured on a narrower link: streaming over PCIe loses, computing in RAM does not care - [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index cc26170..ed6cf68 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,17 +1,21 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-09-30): +Documentation for the LLM inference fleet on the two hosts (2026-10-01): **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving -Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both -Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB -of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx, -bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and -Dankclaw runs two llama.cpp endpoints — -stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth -Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151 -iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B -QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771). +orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s +on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs +(max possible — 262144 does not fit at a 3-seq contract on this box), +bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98, +reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on, +measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus +the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw +runs two endpoints — +gufo v0.1.1 (the Strix Halo engine, built from source; frank-large, +Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M +drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, +HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, +port 7771). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -27,9 +31,9 @@ graph LR gfx1201, 32 GiB each] B2[tcclaviger RDNA4 vLLM fork, digest-pinned podman run, frank-primary, port 7770] - C[GLM-5.3-Flash W4A16 int4 -TP=2, expert offload 73.64 GiB/rank pinned, -14.51 GiB LRU arena, 131072 ctx, vision, tools] + C[Qwen3.8-Flash-Next FP8 bf16 no-spec +TP=2, expert offload automatic, 241664 ctx x 3, +bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 8ff6c7f..7a49cef 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -662,3 +662,59 @@ - `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path). - `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage. - `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership. + +## 2026-10-01 — frank-primary bf16 no-spec cutover + +- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the + night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank, + KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98; + FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner: + "KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per + request: 3.03x". +- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture + (`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB + allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED** + (KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is + PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the + model's native 262,144 does not fit with a 3-seq contract. Every rung was + booted under a transient systemd unit at uid dank with + `LimitMEMLOCK=infinity` (the rootless bench probe masked the production + memory environment) with full engine-log capture. +- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering + `--gpu-memory-utilization` shrinks the engine's own budget rather than + buying slack; Frank forbade it and the winning ladder ran at 0.98. +- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed; + `--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES` + container-env passthrough added; shape header rewritten. +- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present, + spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the + qualified-shape and flags tests flipped to the new env keys. +- Overnight redo campaign (prefetch OFF everywhere) tabled in + `/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` + with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat + 1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone, + six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at + KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`). +- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape. +- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node + updated out of the GLM era. +- **Client contract fixed (thinking + tools).** The live endpoint returned a + null `reasoning` field and dumped the thinking text INLINE in `content` + with a dangling `` (measured: `'We need answer…Brief.\n\n\n391'`) + — the malformed thinking block clients choke on — because the deployed argv + had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all + three contract flags and the qwen rollback inherited the gap. Restored from + `ecaad36`'s values, confirmed against this fork's registries: + `--reasoning-parser qwen3`, `--enable-auto-tool-choice + --tool-call-parser qwen3_xml`, `--default-chat-template-kwargs + '{"enable_thinking": true, "preserve_thinking": true}'`. The seam's + "NO tool dialect" test is flipped to a PRESENT test. +- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 -> + 241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and + `preserve_thinking` moved from a bare top-level body field to + `chat_template_kwargs` (the form vLLM actually reads). +- Production bring-up: the winner (241,664) boots and serves on :7770 — + engine log verified (KV 732,359 tokens, 3.03x concurrency, MNBT 1024, + gather 32x64). Rollback target for the old fp8+MTP4 shape: git tag + `frank-primary-fp8-mtp-20261001`; the 237,568 fallback is documented in + the runbook. diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index a19c22e..4d2e158 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -6,8 +6,8 @@ # script drives `podman run` DIRECTLY (the image entrypoint execs # `vllm serve`), with every engine flag qualified by boot probe and carried # in the repo env file. The seam pins what the flags alone do not say: the -# KV qualification law (11.05 GiB/rank boots bf16 at 798,280 tokens / 3.05x -# concurrency; the old 7.2 GiB ceiling was the MTP4 workspace, not VRAM), +# KV qualification law (the night ladder: 262144/245760 OOM at graph +# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98; # the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost, # clean cache + 262k windows), the automatic-offload law (the fork plans # the tiered-expert offload itself; no flag exists to set), the self-healing @@ -18,7 +18,8 @@ # laws: expert prefetch ABSENT (36% precision), NO speculative decode (the # no-spec family measured fastest once the prefetch tax was removed; MTP # re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode), -# MNBT 1024 (Frank's lock), and NO tool dialect. +# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3, +# tool parser qwen3_xml, preserve_thinking on). load helpers @@ -74,22 +75,19 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" } @test "the qualified engine shape is in the env file" { - # 2026-10-01 boot probe: 262,144 x 3 seqs at bf16 KV, KV_BYTES - # 11,047,010,304 PER RANK boots at "KV cache size: 798,280 tokens" and - # reports max concurrency 3.05x for 262,144-token requests — all three - # full windows fit with an 11,848-token margin. The old "KV_BYTES is - # PER RANK, 9/16 GiB OOM" law was the MTP4 drafter's workspace - # reservation, not a VRAM wall: no-spec takes 11.05 GiB/rank and boots - # clean. - # UTIL 0.90 is a PRODUCTION-corrected value: the first production boot - # at 0.98 died at KV allocation (torch OOM, tried 10.28 GiB with 9.46 - # free — 0.82 GiB short) while the rootless bench probe at the same - # numbers had booted; util 0.90 returns ~2.4 GiB/rank of workspace and - # is the deployed value. - grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE" + # 2026-10-01 night ladder (Frank's method: subtract until it boots, then + # one bonus step): 262144 OOMed at graph capture (76 MiB short, 0 bytes + # free), 245760 OOMed identically, 237568 PASSED (KV 718,606 tokens / + # 3.0x), and the +4096 bonus attempt 241664 PASSED and IS production + # (KV 732,359 tokens / 3.03x at 241,664-token requests, 7,367-token + # margin over the 3-window contract). UTIL stays 0.98 — the production + # unit's pinned-expert path needs the workspace the bench probe hid; + # 0.98 with the smaller window fits. KV_BYTES 10,184,000,000 scales the + # pool with the window. + grep -q '^FRANK_PRIMARY_MAXLEN=241664$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_UTIL=0.90$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_KV_BYTES=11047010304$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_KV_BYTES=10184000000$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE" @@ -112,14 +110,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q -- '--trust-remote-code' "$SERVE" } -@test "the endpoint serves NO tool dialect — tool agents route elsewhere" { - # The 2339fae trio does NOT wire --enable-auto-tool-choice / - # --tool-call-parser (the GLM era added glm47 dialect flags; the rollback - # dropped them). tool_choice "auto" gets a 400 here — tool-using agents - # must route at frank-small or frank-large until a parser is qualified - # for this arm (a behavior change beyond the rollback, not a silent fix). - ! grep -q -- '--enable-auto-tool-choice' "$SERVE" - ! grep -q -- '--tool-call-parser' "$SERVE" +@test "tools and reasoning are wired — the client contract is served" { + # 2026-10-01: the rollback-era trio served NO tool dialect and left + # thinking inline in `content` with a dangling `` and a null + # `reasoning` field (measured: content 'We need answer...Brief.\n + # \n\n391') — the malformed thinking block clients choke on. The working + # values are the ones ecaad36 shipped for this model and this fork's + # registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`. + # preserve_thinking keeps prior thinking blocks across turns. + grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE" + grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE" + grep -q -- '--enable-auto-tool-choice' "$SERVE" + grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE" + grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE" + grep -q 'preserve_thinking' "$ENV_FILE" + grep -q 'enable_thinking' "$ENV_FILE" } @test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {