frank-primary: max-possible window (241664) + reasoning/tool contract

Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.

Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.

Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
Frank
2026-10-01 03:08:52 -07:00
parent e8524d4046
commit 7cab15ac39
7 changed files with 210 additions and 53 deletions
+56
View File
@@ -662,3 +662,59 @@
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
## 2026-10-01 — frank-primary bf16 no-spec cutover
- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
"KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
request: 3.03x".
- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
(`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
(KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
model's native 262,144 does not fit with a 3-seq contract. Every rung was
booted under a transient systemd unit at uid dank with
`LimitMEMLOCK=infinity` (the rootless bench probe masked the production
memory environment) with full engine-log capture.
- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
`--gpu-memory-utilization` shrinks the engine's own budget rather than
buying slack; Frank forbade it and the winning ladder ran at 0.98.
- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
`--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
container-env passthrough added; shape header rewritten.
- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
qualified-shape and flags tests flipped to the new env keys.
- Overnight redo campaign (prefetch OFF everywhere) tabled in
`/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
updated out of the GLM era.
- **Client contract fixed (thinking + tools).** The live endpoint returned a
null `reasoning` field and dumped the thinking text INLINE in `content`
with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
— the malformed thinking block clients choke on — because the deployed argv
had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
three contract flags and the qwen rollback inherited the gap. Restored from
`ecaad36`'s values, confirmed against this fork's registries:
`--reasoning-parser qwen3`, `--enable-auto-tool-choice
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
"NO tool dialect" test is flipped to a PRESENT test.
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 ->
241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and
`preserve_thinking` moved from a bare top-level body field to
`chat_template_kwargs` (the form vLLM actually reads).
- Production bring-up: the winner (241,664) boots and serves on :7770 —
engine log verified (KV 732,359 tokens, 3.03x concurrency, MNBT 1024,
gather 32x64). Rollback target for the old fp8+MTP4 shape: git tag
`frank-primary-fp8-mtp-20261001`; the 237,568 fallback is documented in
the runbook.