frank-primary: max-possible window (241664) + reasoning/tool contract
Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.
Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.
Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
@@ -662,3 +662,59 @@
|
||||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
|
||||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|
||||
|
||||
## 2026-10-01 — frank-primary bf16 no-spec cutover
|
||||
|
||||
- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
|
||||
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
|
||||
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
|
||||
FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
|
||||
"KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
|
||||
request: 3.03x".
|
||||
- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
|
||||
(`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
|
||||
allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
|
||||
(KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
|
||||
PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
|
||||
model's native 262,144 does not fit with a 3-seq contract. Every rung was
|
||||
booted under a transient systemd unit at uid dank with
|
||||
`LimitMEMLOCK=infinity` (the rootless bench probe masked the production
|
||||
memory environment) with full engine-log capture.
|
||||
- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
|
||||
`--gpu-memory-utilization` shrinks the engine's own budget rather than
|
||||
buying slack; Frank forbade it and the winning ladder ran at 0.98.
|
||||
- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
|
||||
`--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
|
||||
container-env passthrough added; shape header rewritten.
|
||||
- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
|
||||
spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
|
||||
qualified-shape and flags tests flipped to the new env keys.
|
||||
- Overnight redo campaign (prefetch OFF everywhere) tabled in
|
||||
`/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
|
||||
with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
|
||||
1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
|
||||
six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
|
||||
KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
|
||||
- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
|
||||
- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
|
||||
updated out of the GLM era.
|
||||
- **Client contract fixed (thinking + tools).** The live endpoint returned a
|
||||
null `reasoning` field and dumped the thinking text INLINE in `content`
|
||||
with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
|
||||
— the malformed thinking block clients choke on — because the deployed argv
|
||||
had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
|
||||
three contract flags and the qwen rollback inherited the gap. Restored from
|
||||
`ecaad36`'s values, confirmed against this fork's registries:
|
||||
`--reasoning-parser qwen3`, `--enable-auto-tool-choice
|
||||
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
|
||||
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
|
||||
"NO tool dialect" test is flipped to a PRESENT test.
|
||||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 ->
|
||||
241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and
|
||||
`preserve_thinking` moved from a bare top-level body field to
|
||||
`chat_template_kwargs` (the form vLLM actually reads).
|
||||
- Production bring-up: the winner (241,664) boots and serves on :7770 —
|
||||
engine log verified (KV 732,359 tokens, 3.03x concurrency, MNBT 1024,
|
||||
gather 32x64). Rollback target for the old fp8+MTP4 shape: git tag
|
||||
`frank-primary-fp8-mtp-20261001`; the 237,568 fallback is documented in
|
||||
the runbook.
|
||||
|
||||
Reference in New Issue
Block a user