frank-primary: max-possible window (241664) + reasoning/tool contract
Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.
Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.
Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
Frank
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 |
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **241664 ctx × 3 seqs** at KV_BYTES 10,184,000,000/rank (measured pool 732,359 tokens / 3.03× concurrency), util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. 241,664 is **max possible** on this box (Frank's subtract-16k ladder: 262144/245760 OOM at graph capture, 237568 passed, +4096 bonus step won). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
|
||||
+30
-10
@@ -8,12 +8,12 @@
|
||||
# restorable from the tag.
|
||||
#
|
||||
# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
|
||||
# * MAXLEN 262144 x SEQS 3 (786,432 tokens) at bf16 KV boots at
|
||||
# KV_BYTES 11,047,010,304 PER RANK: the engine measured "KV cache size:
|
||||
# 798,280 tokens; Maximum concurrency for 262,144 tokens per request:
|
||||
# 3.05x". The pool holds all three full windows with an 11,848-token
|
||||
# margin. Measured decode at the winner shape: 39.38/38.79/36.42 tok/s
|
||||
# (166/1087/4064 ctx).
|
||||
# * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
|
||||
# shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
|
||||
# engine measured "KV cache size: 732,359 tokens; Maximum concurrency
|
||||
# for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
|
||||
# at graph capture (76 MiB short / 0 bytes free); 237568 passed at
|
||||
# 718,606 tokens and the +4096 bonus step passed and won.
|
||||
# * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
|
||||
# the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
|
||||
# the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
|
||||
@@ -59,12 +59,32 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# hoc experiment identity (both hold the GPUs / name registrations).
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape ---------------------------------------------------
|
||||
FRANK_PRIMARY_MAXLEN=262144
|
||||
# --- qualified engine shape (2026-10-01 night ladder, Frank's method) ---------
|
||||
# MAXLEN ladder per Frank: 262144 OOMed in graph capture (76 MiB short),
|
||||
# 245760 OOMed (0 bytes free at graph capture), 237568 PASSED (718,606
|
||||
# tokens / 3.0x), then the +4096 bonus attempt 241664 PASSED and is
|
||||
# PRODUCTION (KV 732,359 tokens / 3.03x at 241,664-token requests).
|
||||
FRANK_PRIMARY_MAXLEN=241664
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
FRANK_PRIMARY_KV_BYTES=11047010304
|
||||
FRANK_PRIMARY_KV_BYTES=10184000000
|
||||
FRANK_PRIMARY_KV_DTYPE=auto
|
||||
FRANK_PRIMARY_MNBT=1024
|
||||
FRANK_PRIMARY_GATHER_CHUNKS=32
|
||||
FRANK_PRIMARY_GATHER_LANES=64
|
||||
FRANK_PRIMARY_UTIL=0.90
|
||||
FRANK_PRIMARY_UTIL=0.98
|
||||
|
||||
# --- client contract: reasoning split, tools, preserve_thinking ----------------
|
||||
# Without --reasoning-parser the engine leaves the thinking text INLINE in
|
||||
# `content` with a dangling `</think>` and a null `reasoning` field (measured
|
||||
# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
|
||||
# malformed thinking block clients choke on. The qwen3 parser splits it
|
||||
# properly. The GLM rollback (d91379d) had dropped all three of these; the
|
||||
# working values are the ones ecaad36 shipped for this model.
|
||||
FRANK_PRIMARY_REASONING_PARSER=qwen3
|
||||
# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
|
||||
# tool_calls work here (without it, tool_choice auto returns 400).
|
||||
FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
|
||||
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
|
||||
# turns; enable_thinking makes thinking the server-side default. Verified the
|
||||
# payload's tokenizer_config template consumes both keys.
|
||||
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
|
||||
|
||||
@@ -47,7 +47,10 @@ set +a
|
||||
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
|
||||
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
|
||||
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
|
||||
: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
|
||||
: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
|
||||
: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
|
||||
# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
|
||||
# classic `podman images | grep -q` pattern races — grep exits on its first
|
||||
# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
|
||||
@@ -115,6 +118,9 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
|
||||
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
|
||||
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
|
||||
--trust-remote-code \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
|
||||
@@ -244,13 +244,78 @@ scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
|
||||
are `podman cp`'d into the container after every recreate (the serve script
|
||||
re-copies them; a recreate wipes container-local files).
|
||||
|
||||
## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)
|
||||
|
||||
Two same-day cutovers moved this page's findings into production:
|
||||
|
||||
1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
|
||||
window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
|
||||
`frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
|
||||
production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
|
||||
measured 37.4-40.6 tok/s at ~54% acceptance.
|
||||
2. **The overnight redo campaign re-measured EVERY lever at matched
|
||||
geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
|
||||
and concluded the day: the incumbent cutover to **bf16 KV, no
|
||||
speculative decode** (Frank's call).
|
||||
|
||||
### What the redo campaign measured (all at shallow depth, 166/1087/4064)
|
||||
|
||||
| lever | verdict |
|
||||
|---|---|
|
||||
| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
|
||||
| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
|
||||
| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
|
||||
| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
|
||||
| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
|
||||
| UVA | 2048 POISON (-39%); 8192 ~parity |
|
||||
| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
|
||||
| six slots | every SEQS-6 KV sizing refused at engine init |
|
||||
|
||||
Depth caveat: every number above is at a near-empty KV. Deep-context
|
||||
behavior (the gather win growing with depth, bf16's decode cost at depth)
|
||||
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
|
||||
### The bf16 no-spec incumbent (production since 2026-10-01)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=241664 x SEQS 3 (724,992 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=10184000000 PER RANK (bf16, pool scales with window)
|
||||
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
```
|
||||
|
||||
Boot-probed 2026-10-01 (transient systemd units, full-log capture): 262144
|
||||
and 245760 both died at graph capture (`Tried to allocate 76.00 MiB ...
|
||||
0 bytes free` — the pool+profile+graph budget exceeds by a hair at util
|
||||
0.98); 237568 passed (KV 718,606 tokens) and the +4096 bonus attempt
|
||||
**241664 PASSED and is production: KV 732,359 tokens / Maximum concurrency
|
||||
for 241,664 tokens per request: 3.03x** — all three full windows fit with
|
||||
a 7,367-token margin. Decode measured at the 262k-era shape before the
|
||||
ladder: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
|
||||
The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
|
||||
reservation, not a VRAM wall — no-spec bf16 takes ~9.5-10.2 GB/rank and
|
||||
boots clean. The failed 11.3 GB attempt OOMed at the engine's free-memory
|
||||
check (`RuntimeError: Worker failed with error 'CUDA out of memory'`,
|
||||
captured from the journald json stream — the fork's giant ERROR lines are
|
||||
split by journald, so grep the `-o json` output, not `-o cat`).
|
||||
|
||||
Rollback to the fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`
|
||||
(env trio + seam re-pinned there). The bf16 trade is deliberate: ~14%
|
||||
decode cost at 4064 ctx vs the fp8+gather arm (49.25 -> 36.42) bought
|
||||
clean cache + the largest window this box boots. The model's native
|
||||
262,144 exceeds what graph capture grants at util 0.98 here: the ladder
|
||||
measured 262144/245760 OOM and 241,664 as max possible (Frank's
|
||||
subtract-16k method, then one +4k bonus step).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
|
||||
(which frontier MoE fits this box; the same PCIe-expert-stream wall at
|
||||
V4.1 scale)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
|
||||
(production is the R9V fork; this page's container is the ad hoc detour)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
|
||||
fork era; production has been the tcclaviger fork since)
|
||||
- [[x4-pair-experiment]] — the same residency law measured on a narrower
|
||||
link: streaming over PCIe loses, computing in RAM does not care
|
||||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|
||||
@@ -1,17 +1,21 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-30):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-01):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both
|
||||
Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB
|
||||
of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx,
|
||||
bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and
|
||||
Dankclaw runs two llama.cpp endpoints —
|
||||
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
|
||||
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
|
||||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
|
||||
QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs
|
||||
(max possible — 262144 does not fit at a 3-seq contract on this box),
|
||||
bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints —
|
||||
gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
|
||||
Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M
|
||||
drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||||
HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
|
||||
port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -27,9 +31,9 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
podman run, frank-primary, port 7770]
|
||||
C[GLM-5.3-Flash W4A16 int4
|
||||
TP=2, expert offload 73.64 GiB/rank pinned,
|
||||
14.51 GiB LRU arena, 131072 ctx, vision, tools]
|
||||
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
|
||||
TP=2, expert offload automatic, 241664 ctx x 3,
|
||||
bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
|
||||
@@ -662,3 +662,59 @@
|
||||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
|
||||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|
||||
|
||||
## 2026-10-01 — frank-primary bf16 no-spec cutover
|
||||
|
||||
- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
|
||||
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
|
||||
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
|
||||
FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
|
||||
"KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
|
||||
request: 3.03x".
|
||||
- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
|
||||
(`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
|
||||
allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
|
||||
(KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
|
||||
PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
|
||||
model's native 262,144 does not fit with a 3-seq contract. Every rung was
|
||||
booted under a transient systemd unit at uid dank with
|
||||
`LimitMEMLOCK=infinity` (the rootless bench probe masked the production
|
||||
memory environment) with full engine-log capture.
|
||||
- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
|
||||
`--gpu-memory-utilization` shrinks the engine's own budget rather than
|
||||
buying slack; Frank forbade it and the winning ladder ran at 0.98.
|
||||
- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
|
||||
`--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
|
||||
container-env passthrough added; shape header rewritten.
|
||||
- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
|
||||
spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
|
||||
qualified-shape and flags tests flipped to the new env keys.
|
||||
- Overnight redo campaign (prefetch OFF everywhere) tabled in
|
||||
`/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
|
||||
with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
|
||||
1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
|
||||
six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
|
||||
KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
|
||||
- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
|
||||
- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
|
||||
updated out of the GLM era.
|
||||
- **Client contract fixed (thinking + tools).** The live endpoint returned a
|
||||
null `reasoning` field and dumped the thinking text INLINE in `content`
|
||||
with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
|
||||
— the malformed thinking block clients choke on — because the deployed argv
|
||||
had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
|
||||
three contract flags and the qwen rollback inherited the gap. Restored from
|
||||
`ecaad36`'s values, confirmed against this fork's registries:
|
||||
`--reasoning-parser qwen3`, `--enable-auto-tool-choice
|
||||
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
|
||||
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
|
||||
"NO tool dialect" test is flipped to a PRESENT test.
|
||||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 ->
|
||||
241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and
|
||||
`preserve_thinking` moved from a bare top-level body field to
|
||||
`chat_template_kwargs` (the form vLLM actually reads).
|
||||
- Production bring-up: the winner (241,664) boots and serves on :7770 —
|
||||
engine log verified (KV 732,359 tokens, 3.03x concurrency, MNBT 1024,
|
||||
gather 32x64). Rollback target for the old fp8+MTP4 shape: git tag
|
||||
`frank-primary-fp8-mtp-20261001`; the 237,568 fallback is documented in
|
||||
the runbook.
|
||||
|
||||
@@ -6,8 +6,8 @@
|
||||
# script drives `podman run` DIRECTLY (the image entrypoint execs
|
||||
# `vllm serve`), with every engine flag qualified by boot probe and carried
|
||||
# in the repo env file. The seam pins what the flags alone do not say: the
|
||||
# KV qualification law (11.05 GiB/rank boots bf16 at 798,280 tokens / 3.05x
|
||||
# concurrency; the old 7.2 GiB ceiling was the MTP4 workspace, not VRAM),
|
||||
# KV qualification law (the night ladder: 262144/245760 OOM at graph
|
||||
# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
|
||||
# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
|
||||
# clean cache + 262k windows), the automatic-offload law (the fork plans
|
||||
# the tiered-expert offload itself; no flag exists to set), the self-healing
|
||||
@@ -18,7 +18,8 @@
|
||||
# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
|
||||
# no-spec family measured fastest once the prefetch tax was removed; MTP
|
||||
# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
|
||||
# MNBT 1024 (Frank's lock), and NO tool dialect.
|
||||
# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3,
|
||||
# tool parser qwen3_xml, preserve_thinking on).
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -74,22 +75,19 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
}
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-10-01 boot probe: 262,144 x 3 seqs at bf16 KV, KV_BYTES
|
||||
# 11,047,010,304 PER RANK boots at "KV cache size: 798,280 tokens" and
|
||||
# reports max concurrency 3.05x for 262,144-token requests — all three
|
||||
# full windows fit with an 11,848-token margin. The old "KV_BYTES is
|
||||
# PER RANK, 9/16 GiB OOM" law was the MTP4 drafter's workspace
|
||||
# reservation, not a VRAM wall: no-spec takes 11.05 GiB/rank and boots
|
||||
# clean.
|
||||
# UTIL 0.90 is a PRODUCTION-corrected value: the first production boot
|
||||
# at 0.98 died at KV allocation (torch OOM, tried 10.28 GiB with 9.46
|
||||
# free — 0.82 GiB short) while the rootless bench probe at the same
|
||||
# numbers had booted; util 0.90 returns ~2.4 GiB/rank of workspace and
|
||||
# is the deployed value.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
|
||||
# 2026-10-01 night ladder (Frank's method: subtract until it boots, then
|
||||
# one bonus step): 262144 OOMed at graph capture (76 MiB short, 0 bytes
|
||||
# free), 245760 OOMed identically, 237568 PASSED (KV 718,606 tokens /
|
||||
# 3.0x), and the +4096 bonus attempt 241664 PASSED and IS production
|
||||
# (KV 732,359 tokens / 3.03x at 241,664-token requests, 7,367-token
|
||||
# margin over the 3-window contract). UTIL stays 0.98 — the production
|
||||
# unit's pinned-expert path needs the workspace the bench probe hid;
|
||||
# 0.98 with the smaller window fits. KV_BYTES 10,184,000,000 scales the
|
||||
# pool with the window.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=241664$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.90$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=11047010304$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=10184000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
@@ -112,14 +110,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q -- '--trust-remote-code' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the endpoint serves NO tool dialect — tool agents route elsewhere" {
|
||||
# The 2339fae trio does NOT wire --enable-auto-tool-choice /
|
||||
# --tool-call-parser (the GLM era added glm47 dialect flags; the rollback
|
||||
# dropped them). tool_choice "auto" gets a 400 here — tool-using agents
|
||||
# must route at frank-small or frank-large until a parser is qualified
|
||||
# for this arm (a behavior change beyond the rollback, not a silent fix).
|
||||
! grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
! grep -q -- '--tool-call-parser' "$SERVE"
|
||||
@test "tools and reasoning are wired — the client contract is served" {
|
||||
# 2026-10-01: the rollback-era trio served NO tool dialect and left
|
||||
# thinking inline in `content` with a dangling `</think>` and a null
|
||||
# `reasoning` field (measured: content 'We need answer...Brief.\n</think>
|
||||
# \n\n391') — the malformed thinking block clients choke on. The working
|
||||
# values are the ones ecaad36 shipped for this model and this fork's
|
||||
# registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
|
||||
# preserve_thinking keeps prior thinking blocks across turns.
|
||||
grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
|
||||
grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
|
||||
grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
|
||||
grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
|
||||
grep -q 'preserve_thinking' "$ENV_FILE"
|
||||
grep -q 'enable_thinking' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
|
||||
|
||||
Reference in New Issue
Block a user