dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(frank-zealot): key the ledger to the production boot; record the arm drop-in hijack

Browse Source 
The cutover deploy's FIRST boot was not the shipped payload: a stale
frank-zealot.service.d/ptq1.conf from the bench window still pointed
FRANK_ZEALOT_ENV_FILE at /tmp/zealot-arm-current.env, so the unit booted the
unmeasured 27B PTQ1_0 arm (3 slots of 133,888, 21,188 MiB) while /health
answered 200, /v1/models echoed the alias and the e2e gate passed 33/33.
Drop-in retired to ~/.AGENTTRASH/20261018-zealot-ptq1-arm/; the 13:19 restart
is the shape the docs now quote (n_ctx_slot 262144, KV 13,056.00 MiB,
settled 17,922/6,203).

Corrections the production ledger forced:
- RS buffer is 150.75 MiB (3 cells) on the shipped spec-off shape; 452.25 MiB
  belongs to the SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot.
  The drafter triples the recurrent state on top of its 3,072 MiB of f16 KV.
- Decode measured 168.4-168.7 t/s single-stream and ~300 t/s aggregate over
  three streams (102.8 each). Wall-clock timing of a generation that stops on
  its own reads 4x low (72 t/s for a 24-token countdown) - measure with
  ignore_eos and read the engine's timings.
- OpenWebUI stores both decoration values inside JSON columns (params/meta),
  and frank-dragoon's context.length had drifted to 262,144 against its own
  200,704 ceiling; both re-keyed in the live DB.

New law in AGENTS.md: after deploying a recently-benched LLM service, check
DropInPaths, then /proc/<pid>/cmdline, then the ledger. Health checks cannot
see the payload.
This commit is contained in:
Frank
2026-10-18 13:27:27 -07:00
parent e7f0d93c86
commit 695a5f3281
5 changed files with 145 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Dankclaw/OpenWebUI.md
+12 -5
View File
@@ -53,11 +53,18 @@ Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-*
PQ2_0 with the MTP head, 3 × 262,144 — was **deployed as production**, so the
picker row below was re-keyed to it (see the next paragraph).
The `model` table carries exactly one decoration row, `frank-zealot`
(context length 262,144 = the per-slot window since 2026-10-18; description
re-keyed the same day to Ternary-Bonsai-2-9B PQ2_0 + MTP + Q8_0 vision mmproj —
the row's `description` is live DB state, so a payload swap must re-key it in
the database too, not only in the repo).
The `model` table carries exactly two decoration rows, `frank-zealot` and
`frank-dragoon`. Both the context length and the description live **inside
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
`{"description": …}`; there is no `context_length` or `description` column, so
`sqlite3` one-liners against those names fail. Re-keyed 2026-10-18:
`frank-zealot` → 262,144 with the Bonsai-2-9B description, and `frank-dragoon`
→ 200,704 (its `params` had kept the pre-lowering 262,144 window from
2026-10-17 while its own description already said 200,704). These rows are
live DB state, so a payload swap must re-key the database too, not only the
repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the
stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the
DB is in WAL and the app is live).
A third gate sits *after* the table, and it is the one that bites family
accounts (2026-10-07): a model row is only visible to a non-admin user when
serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md
+63 -14
View File
@@ -76,8 +76,10 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
instead — `RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB
each — so pool arithmetic alone under-prices the shape.
instead: the PRODUCTION ledger prints `RS buffer size = 150.75 MiB (3 cells,
32 layers, 3 seqs)` ≈ 50 MiB a cell, while the SPEC-ARMED arms printed
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples
even the recurrent state. Pool arithmetic alone under-prices the shape.
## The shape (`scripts/frank-zealot.env`)
@@ -123,30 +125,36 @@ inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
## Boot ledger (deployed arm, verbosity 4)
## Boot ledger (production boot 2026-10-18 13:19, under the unit, verbosity 4)
```
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
CUDA0 RS buffer size = 452.25 MiB # 3 recurrent-state cells, 32 layers, 3 seqs
compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host
mmproj 595 MiB + 248 MiB # charged AFTER the fit check
nvidia-smi settled: 17,876 MiB used / 6,249 MiB free of 24,123
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
CUDA0 RS buffer size = 150.75 MiB # 3 cells, 32 layers, 3 seqs
CUDA0 compute buffer = 1409.02 MiB # + 1088.87 MiB CUDA_Host, 2.84 output
mmproj worst-case (est.) = 868.32 MiB # 248.10 CUDA + 24.93 CPU, AFTER fit
nvidia-smi settled: 17,922 MiB used / 6,203 MiB free of 24,123
n_slots = 3, n_ctx_slot = 262144, kv_unified = false
```
No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
never the projection**.
## Measured (deployed arm)
## Measured (production boot, `/completion` `timings`, greedy, `ignore_eos`)
- **167.2 t/s** greedy decode (512 tokens, `/completion` `timings`); 158.9 t/s
single-stream chat with the card's sampler.
- **280.2 t/s aggregate** across three concurrent streams (≈93.8 each).
- **2,877 t/s prefill** at 106,418 prompt tokens.
- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
(the trial arm printed 167.2; same shape, same noise band).
- **~300 t/s aggregate** across three concurrent 512-token streams
(102.8 t/s per stream, round 1 and 2 both) — the trial arm's 280.2.
- **2,877 t/s prefill at 106,418 tokens** — the trial arm; the production boot
has not re-run that depth.
- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams,
691.7 prefill at 166k. Beaten on every axis.
- The **first request after a boot is a warmup artifact** (37.7 t/s class) —
never report it.
never report it. And never time a wall-clock aggregate on a generation that
stops on its own: a 24-token countdown measured 72 t/s across three streams
while the same shape with `ignore_eos` gave 300.
## Contract traps
@@ -186,6 +194,20 @@ never the projection**.
7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the
unit (llama.cpp needs the drain time to release slots and the card); the
hand-booted arm must be killed by PID, never `pkill -f llama-server`.
- **A leftover arm drop-in hijacks the production boot, and no health check
sees it.** The sanctioned bench method is a transient
`Environment=FRANK_ZEALOT_ENV_FILE=/tmp/<arm>.env` drop-in, and an arm env
`source`s the repo env and then overrides it. On 2026-10-18 a stale
`frank-zealot.service.d/ptq1.conf` — written 03:29 for an unfinished 27B
**PTQ1_0** experiment — was still installed, so the cutover deploy restarted
the unit and booted THAT: `--model …/Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf
--ctx-size 401408 --parallel 3 --batch-size 3072 --ubatch-size 1536`, three
slots of 133,888, 21,188 MiB used. `/health` was 200, `/v1/models` said
`frank-zealot` (the alias is a flag, not a fact), and the Playwright gate
passed 33/33 — none of those read the payload. **After deploying an endpoint
that was recently benched: `systemctl show -p DropInPaths`, then
`tr '\0' ' ' </proc/$pid/cmdline`, then the ledger — in that order.** Retire
the drop-in to `~/.AGENTTRASH/<date>-<topic>/`, `daemon-reload`, restart.
## Compaction hub
@@ -197,6 +219,33 @@ the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--paralle
a compaction request no longer serializes behind a live generation (the Gemma4
era's `--parallel 1` law).
## Deploy ledger (2026-10-18)
- `./scripts/deploy.sh dankclaw --service frank-zealot` at 13:16 PDT; the e2e
gate ran 33/33. The first boot was the hijacked arm above; the drop-in went
to `~/.AGENTTRASH/20261018-zealot-ptq1-arm/` and the 13:19 restart is the
ledger recorded here.
- Live probes on the shipped boot: chat `27*59` → **1593**; `where_am_i` tool
call fired with schema-correct empty arguments; 512 px solid-red PNG →
**Red** at 278 prompt tokens with `process_mtmd` chunk encoding in the
journal; `/props` reports `n_ctx 262144` and `presence_penalty 1.0`;
`/metrics` answers 200 with 15 `llamacpp:*` families and
`up{job="frank-zealot"}` is **1** in Prometheus for the first time in its
retained history.
- OpenWebUI picker rows re-keyed in the live DB (`Connection.backup()` snapshot
`webui.db.bak-20261018` taken first; `frank-dragoon`'s `context.length` also
corrected 262,144 → 200,704 — see [[OpenWebUI]]).
- What the bench window left behind, and it is NOT garbage: the **PTQ1_0
candidate was fetched and armed but never measured** —
`scripts/download-frank-zealot-bonsai2-ptq1.sh` is committed, and its header
states the case (1,201 MiB of freed weights against the publisher's own table
showing dense-ternary unpacking LOSING on Ampere, with no GA102 row to settle
the 3090). `/home/dank/models/Bonsai2-PTQ1_0/` (5,946,648,928 B + a hardlinked
mmproj, 03:02), the arm env `/tmp/zealot-arm-current.env` and the
`/tmp/arm-*.jsonl` 9B results are disk/`/tmp` state a reboot erases. Either
measure the PTQ1_0 arm in a real window or record it as declined — leaving it
armed and unmeasured is exactly what let it hijack a production boot.
## Fallback rungs
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
serverAppConfigs-docs/log.md
+33
View File
@@ -2156,4 +2156,37 @@ every restart since the cutover.
picker row (DB `description` + context length 262,144) re-keyed. Rollback:
revert the swap commit — the 27B payload and the build-10754 fork tree are
both still on disk.
- Deployed 13:16 PDT (`deploy.sh dankclaw --service frank-zealot`, e2e 33/33)
— and the first boot was **not the shipped payload**: a stale
`frank-zealot.service.d/ptq1.conf` from the bench window (written 03:29 for
an unfinished 27B **PTQ1_0** experiment) pointed
`FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit booted
that arm: 3 slots of 133,888, 21,188 MiB used, batch 3072/ubatch 1536.
`/health` 200, `/v1/models` echoed the alias, Playwright 33/33 — nothing in
the gate reads the payload. Drop-in moved to
`~/.AGENTTRASH/20261018-zealot-ptq1-arm/`, `daemon-reload`, restart at 13:19:
`n_slots = 3, n_ctx_slot = 262144`, KV 13,056.00 MiB, settled 17,922/6,203.
- Production measures (engine `timings`, greedy, `ignore_eos`): **168.7 t/s**
decode (168.4/168.7/168.7) and **~300 t/s** aggregate across three 512-token
streams (102.8 each). My first battery reported 117 and 72 t/s: it timed
natural generations that stopped at 24 tokens — wall-clock aggregate on an
early EOS reads 4× low. `ignore_eos` + the engine's own numbers is the only
honest way to time this.
- Ledger correction: the shipped shape prints **RS 150.75 MiB** (3 cells), not
the 452.25 MiB my first-pass notes carried — that figure belongs to the
SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot (9 cells). The
drafter triples the recurrent state as well as billing 3,072 MiB of f16 KV.
Corrected in AGENTS.md (topology row + architecture section), the env header
and the runbook, with the production boot ledger replacing the trial's.
- Verified live: chat `27*59` → 1593; `where_am_i` tool call fired; 512 px
solid-red PNG → **Red** (278 prompt tokens, `process_mtmd` in the journal);
`/props` `n_ctx 262144`, `presence_penalty 1.0`; `/metrics` 200 with 15
`llamacpp:*` families and `up{job="frank-zealot"} = 1` — the first non-zero
sample in that job's retained history.
- OpenWebUI `model` rows re-keyed in the live DB (backup
`webui.db.bak-20261018`): `frank-zealot` → 262,144 + the 9B description, and
`frank-dragoon`'s `context.length` corrected 262,144 → **200,704** (it had
kept the pre-lowering window from 2026-10-17 while its own description said
200,704). Note the columns are `params`/`meta` JSON — the table has no
`context_length`/`description` columns.