fix(frank-zealot): key the ledger to the production boot; record the arm drop-in hijack
The cutover deploy's FIRST boot was not the shipped payload: a stale frank-zealot.service.d/ptq1.conf from the bench window still pointed FRANK_ZEALOT_ENV_FILE at /tmp/zealot-arm-current.env, so the unit booted the unmeasured 27B PTQ1_0 arm (3 slots of 133,888, 21,188 MiB) while /health answered 200, /v1/models echoed the alias and the e2e gate passed 33/33. Drop-in retired to ~/.AGENTTRASH/20261018-zealot-ptq1-arm/; the 13:19 restart is the shape the docs now quote (n_ctx_slot 262144, KV 13,056.00 MiB, settled 17,922/6,203). Corrections the production ledger forced: - RS buffer is 150.75 MiB (3 cells) on the shipped spec-off shape; 452.25 MiB belongs to the SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot. The drafter triples the recurrent state on top of its 3,072 MiB of f16 KV. - Decode measured 168.4-168.7 t/s single-stream and ~300 t/s aggregate over three streams (102.8 each). Wall-clock timing of a generation that stops on its own reads 4x low (72 t/s for a 24-token countdown) - measure with ignore_eos and read the engine's timings. - OpenWebUI stores both decoration values inside JSON columns (params/meta), and frank-dragoon's context.length had drifted to 262,144 against its own 200,704 ceiling; both re-keyed in the live DB. New law in AGENTS.md: after deploying a recently-benched LLM service, check DropInPaths, then /proc/<pid>/cmdline, then the ledger. Health checks cannot see the payload.
This commit is contained in:
Frank
@@ -76,8 +76,10 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
|
||||
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
|
||||
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||||
instead — `RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB
|
||||
each — so pool arithmetic alone under-prices the shape.
|
||||
instead: the PRODUCTION ledger prints `RS buffer size = 150.75 MiB (3 cells,
|
||||
32 layers, 3 seqs)` ≈ 50 MiB a cell, while the SPEC-ARMED arms printed
|
||||
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples
|
||||
even the recurrent state. Pool arithmetic alone under-prices the shape.
|
||||
|
||||
## The shape (`scripts/frank-zealot.env`)
|
||||
|
||||
@@ -123,30 +125,36 @@ inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
|
||||
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
|
||||
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
|
||||
|
||||
## Boot ledger (deployed arm, verbosity 4)
|
||||
## Boot ledger (production boot 2026-10-18 13:19, under the unit, verbosity 4)
|
||||
|
||||
```
|
||||
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
|
||||
CUDA0 RS buffer size = 452.25 MiB # 3 recurrent-state cells, 32 layers, 3 seqs
|
||||
compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host
|
||||
mmproj 595 MiB + 248 MiB # charged AFTER the fit check
|
||||
nvidia-smi settled: 17,876 MiB used / 6,249 MiB free of 24,123
|
||||
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
|
||||
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
|
||||
CUDA0 RS buffer size = 150.75 MiB # 3 cells, 32 layers, 3 seqs
|
||||
CUDA0 compute buffer = 1409.02 MiB # + 1088.87 MiB CUDA_Host, 2.84 output
|
||||
mmproj worst-case (est.) = 868.32 MiB # 248.10 CUDA + 24.93 CPU, AFTER fit
|
||||
nvidia-smi settled: 17,922 MiB used / 6,203 MiB free of 24,123
|
||||
n_slots = 3, n_ctx_slot = 262144, kv_unified = false
|
||||
```
|
||||
|
||||
No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
|
||||
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
|
||||
never the projection**.
|
||||
|
||||
## Measured (deployed arm)
|
||||
## Measured (production boot, `/completion` `timings`, greedy, `ignore_eos`)
|
||||
|
||||
- **167.2 t/s** greedy decode (512 tokens, `/completion` `timings`); 158.9 t/s
|
||||
single-stream chat with the card's sampler.
|
||||
- **280.2 t/s aggregate** across three concurrent streams (≈93.8 each).
|
||||
- **2,877 t/s prefill** at 106,418 prompt tokens.
|
||||
- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
|
||||
(the trial arm printed 167.2; same shape, same noise band).
|
||||
- **~300 t/s aggregate** across three concurrent 512-token streams
|
||||
(102.8 t/s per stream, round 1 and 2 both) — the trial arm's 280.2.
|
||||
- **2,877 t/s prefill at 106,418 tokens** — the trial arm; the production boot
|
||||
has not re-run that depth.
|
||||
- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams,
|
||||
691.7 prefill at 166k. Beaten on every axis.
|
||||
- The **first request after a boot is a warmup artifact** (37.7 t/s class) —
|
||||
never report it.
|
||||
never report it. And never time a wall-clock aggregate on a generation that
|
||||
stops on its own: a 24-token countdown measured 72 t/s across three streams
|
||||
while the same shape with `ignore_eos` gave 300.
|
||||
|
||||
## Contract traps
|
||||
|
||||
@@ -186,6 +194,20 @@ never the projection**.
|
||||
7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the
|
||||
unit (llama.cpp needs the drain time to release slots and the card); the
|
||||
hand-booted arm must be killed by PID, never `pkill -f llama-server`.
|
||||
- **A leftover arm drop-in hijacks the production boot, and no health check
|
||||
sees it.** The sanctioned bench method is a transient
|
||||
`Environment=FRANK_ZEALOT_ENV_FILE=/tmp/<arm>.env` drop-in, and an arm env
|
||||
`source`s the repo env and then overrides it. On 2026-10-18 a stale
|
||||
`frank-zealot.service.d/ptq1.conf` — written 03:29 for an unfinished 27B
|
||||
**PTQ1_0** experiment — was still installed, so the cutover deploy restarted
|
||||
the unit and booted THAT: `--model …/Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf
|
||||
--ctx-size 401408 --parallel 3 --batch-size 3072 --ubatch-size 1536`, three
|
||||
slots of 133,888, 21,188 MiB used. `/health` was 200, `/v1/models` said
|
||||
`frank-zealot` (the alias is a flag, not a fact), and the Playwright gate
|
||||
passed 33/33 — none of those read the payload. **After deploying an endpoint
|
||||
that was recently benched: `systemctl show -p DropInPaths`, then
|
||||
`tr '\0' ' ' </proc/$pid/cmdline`, then the ledger — in that order.** Retire
|
||||
the drop-in to `~/.AGENTTRASH/<date>-<topic>/`, `daemon-reload`, restart.
|
||||
|
||||
## Compaction hub
|
||||
|
||||
@@ -197,6 +219,33 @@ the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--paralle
|
||||
a compaction request no longer serializes behind a live generation (the Gemma4
|
||||
era's `--parallel 1` law).
|
||||
|
||||
## Deploy ledger (2026-10-18)
|
||||
|
||||
- `./scripts/deploy.sh dankclaw --service frank-zealot` at 13:16 PDT; the e2e
|
||||
gate ran 33/33. The first boot was the hijacked arm above; the drop-in went
|
||||
to `~/.AGENTTRASH/20261018-zealot-ptq1-arm/` and the 13:19 restart is the
|
||||
ledger recorded here.
|
||||
- Live probes on the shipped boot: chat `27*59` → **1593**; `where_am_i` tool
|
||||
call fired with schema-correct empty arguments; 512 px solid-red PNG →
|
||||
**Red** at 278 prompt tokens with `process_mtmd` chunk encoding in the
|
||||
journal; `/props` reports `n_ctx 262144` and `presence_penalty 1.0`;
|
||||
`/metrics` answers 200 with 15 `llamacpp:*` families and
|
||||
`up{job="frank-zealot"}` is **1** in Prometheus for the first time in its
|
||||
retained history.
|
||||
- OpenWebUI picker rows re-keyed in the live DB (`Connection.backup()` snapshot
|
||||
`webui.db.bak-20261018` taken first; `frank-dragoon`'s `context.length` also
|
||||
corrected 262,144 → 200,704 — see [[OpenWebUI]]).
|
||||
- What the bench window left behind, and it is NOT garbage: the **PTQ1_0
|
||||
candidate was fetched and armed but never measured** —
|
||||
`scripts/download-frank-zealot-bonsai2-ptq1.sh` is committed, and its header
|
||||
states the case (1,201 MiB of freed weights against the publisher's own table
|
||||
showing dense-ternary unpacking LOSING on Ampere, with no GA102 row to settle
|
||||
the 3090). `/home/dank/models/Bonsai2-PTQ1_0/` (5,946,648,928 B + a hardlinked
|
||||
mmproj, 03:02), the arm env `/tmp/zealot-arm-current.env` and the
|
||||
`/tmp/arm-*.jsonl` 9B results are disk/`/tmp` state a reboot erases. Either
|
||||
measure the PTQ1_0 arm in a real window or record it as declined — leaving it
|
||||
armed and unmeasured is exactly what let it hijack a production boot.
|
||||
|
||||
## Fallback rungs
|
||||
|
||||
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
|
||||
|
||||
Reference in New Issue
Block a user