fix(frank-zealot): key the ledger to the production boot; record the arm drop-in hijack

The cutover deploy's FIRST boot was not the shipped payload: a stale
frank-zealot.service.d/ptq1.conf from the bench window still pointed
FRANK_ZEALOT_ENV_FILE at /tmp/zealot-arm-current.env, so the unit booted the
unmeasured 27B PTQ1_0 arm (3 slots of 133,888, 21,188 MiB) while /health
answered 200, /v1/models echoed the alias and the e2e gate passed 33/33.
Drop-in retired to ~/.AGENTTRASH/20261018-zealot-ptq1-arm/; the 13:19 restart
is the shape the docs now quote (n_ctx_slot 262144, KV 13,056.00 MiB,
settled 17,922/6,203).

Corrections the production ledger forced:
- RS buffer is 150.75 MiB (3 cells) on the shipped spec-off shape; 452.25 MiB
  belongs to the SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot.
  The drafter triples the recurrent state on top of its 3,072 MiB of f16 KV.
- Decode measured 168.4-168.7 t/s single-stream and ~300 t/s aggregate over
  three streams (102.8 each). Wall-clock timing of a generation that stops on
  its own reads 4x low (72 t/s for a 24-token countdown) - measure with
  ignore_eos and read the engine's timings.
- OpenWebUI stores both decoration values inside JSON columns (params/meta),
  and frank-dragoon's context.length had drifted to 262,144 against its own
  200,704 ceiling; both re-keyed in the live DB.

New law in AGENTS.md: after deploying a recently-benched LLM service, check
DropInPaths, then /proc/<pid>/cmdline, then the ledger. Health checks cannot
see the payload.
This commit is contained in:
Frank
2026-10-18 13:27:27 -07:00
parent e7f0d93c86
commit 695a5f3281
5 changed files with 145 additions and 36 deletions
+33
View File
@@ -2156,4 +2156,37 @@ every restart since the cutover.
picker row (DB `description` + context length 262,144) re-keyed. Rollback:
revert the swap commit — the 27B payload and the build-10754 fork tree are
both still on disk.
- Deployed 13:16 PDT (`deploy.sh dankclaw --service frank-zealot`, e2e 33/33)
— and the first boot was **not the shipped payload**: a stale
`frank-zealot.service.d/ptq1.conf` from the bench window (written 03:29 for
an unfinished 27B **PTQ1_0** experiment) pointed
`FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit booted
that arm: 3 slots of 133,888, 21,188 MiB used, batch 3072/ubatch 1536.
`/health` 200, `/v1/models` echoed the alias, Playwright 33/33 — nothing in
the gate reads the payload. Drop-in moved to
`~/.AGENTTRASH/20261018-zealot-ptq1-arm/`, `daemon-reload`, restart at 13:19:
`n_slots = 3, n_ctx_slot = 262144`, KV 13,056.00 MiB, settled 17,922/6,203.
- Production measures (engine `timings`, greedy, `ignore_eos`): **168.7 t/s**
decode (168.4/168.7/168.7) and **~300 t/s** aggregate across three 512-token
streams (102.8 each). My first battery reported 117 and 72 t/s: it timed
natural generations that stopped at 24 tokens — wall-clock aggregate on an
early EOS reads 4× low. `ignore_eos` + the engine's own numbers is the only
honest way to time this.
- Ledger correction: the shipped shape prints **RS 150.75 MiB** (3 cells), not
the 452.25 MiB my first-pass notes carried — that figure belongs to the
SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot (9 cells). The
drafter triples the recurrent state as well as billing 3,072 MiB of f16 KV.
Corrected in AGENTS.md (topology row + architecture section), the env header
and the runbook, with the production boot ledger replacing the trial's.
- Verified live: chat `27*59` → 1593; `where_am_i` tool call fired; 512 px
solid-red PNG → **Red** (278 prompt tokens, `process_mtmd` in the journal);
`/props` `n_ctx 262144`, `presence_penalty 1.0`; `/metrics` 200 with 15
`llamacpp:*` families and `up{job="frank-zealot"} = 1` — the first non-zero
sample in that job's retained history.
- OpenWebUI `model` rows re-keyed in the live DB (backup
`webui.db.bak-20261018`): `frank-zealot` → 262,144 + the 9B description, and
`frank-dragoon`'s `context.length` corrected 262,144 → **200,704** (it had
kept the pre-lowering window from 2026-10-17 while its own description said
200,704). Note the columns are `params`/`meta` JSON — the table has no
`context_length`/`description` columns.