fix(frank-zealot): key the ledger to the production boot; record the arm drop-in hijack
The cutover deploy's FIRST boot was not the shipped payload: a stale frank-zealot.service.d/ptq1.conf from the bench window still pointed FRANK_ZEALOT_ENV_FILE at /tmp/zealot-arm-current.env, so the unit booted the unmeasured 27B PTQ1_0 arm (3 slots of 133,888, 21,188 MiB) while /health answered 200, /v1/models echoed the alias and the e2e gate passed 33/33. Drop-in retired to ~/.AGENTTRASH/20261018-zealot-ptq1-arm/; the 13:19 restart is the shape the docs now quote (n_ctx_slot 262144, KV 13,056.00 MiB, settled 17,922/6,203). Corrections the production ledger forced: - RS buffer is 150.75 MiB (3 cells) on the shipped spec-off shape; 452.25 MiB belongs to the SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot. The drafter triples the recurrent state on top of its 3,072 MiB of f16 KV. - Decode measured 168.4-168.7 t/s single-stream and ~300 t/s aggregate over three streams (102.8 each). Wall-clock timing of a generation that stops on its own reads 4x low (72 t/s for a 24-token countdown) - measure with ignore_eos and read the engine's timings. - OpenWebUI stores both decoration values inside JSON columns (params/meta), and frank-dragoon's context.length had drifted to 262,144 against its own 200,704 ceiling; both re-keyed in the live DB. New law in AGENTS.md: after deploying a recently-benched LLM service, check DropInPaths, then /proc/<pid>/cmdline, then the ledger. Health checks cannot see the payload.
This commit is contained in:
Frank
@@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for **vision**, on the RTX 3090, **THREE 262,144-token slots** (`--ctx-size 786432 --parallel 3` = every conversation at the trained window), **q8_0 K + q8_0 V** under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; **f16 KV needs 26,112 MiB and cannot boot this card**; +452 MiB of recurrent state for the three DeltaNet cells), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, not because the head is absent**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Boot 17,876 used / 6,249 free; 167.2 t/s single-stream decode, 280.2 t/s aggregate across three concurrent streams, 2,877 t/s prefill at 106,418. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
|
||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for **vision**, on the RTX 3090, **THREE 262,144-token slots** (`--ctx-size 786432 --parallel 3` = every conversation at the trained window), **q8_0 K + q8_0 V** under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; **f16 KV needs 26,112 MiB and cannot boot this card**; +151 MiB of recurrent state for the three DeltaNet cells — the SPEC-ARMED arms print 452 MiB, because MTP depth 2 bills three seqs per slot), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, not because the head is absent**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Production boot 17,922 used / 6,203 free; **168.7 t/s** single-stream greedy decode (three runs, 168.4–168.7) and **~300 t/s** aggregate across three concurrent streams (102.8 each); 2,877 t/s prefill at 106,418 was the trial arm. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -494,9 +494,10 @@ so **q8_0 K + q8_0 V bills 17,408 B/token ≈ 17 KiB** = **13,056.00 MiB** at th
|
||||
786,432 pool (ledger-exact) — exactly HALF the 27B's 34,816 B/token, which is
|
||||
the whole reason three native windows fit where the 27B afforded only two
|
||||
shortened ones. **f16 K would bill 26,112 MiB and cannot boot on a 24,123 MiB
|
||||
card.** The 24 DeltaNet layers bill a fixed recurrent state per SLOT —
|
||||
`RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB each — so
|
||||
pool arithmetic alone under-prices the shape.
|
||||
card.** The 24 DeltaNet layers bill a fixed recurrent state per SLOT — the
|
||||
production ledger prints `RS buffer size = 150.75 MiB (3 cells, 32 layers,
|
||||
3 seqs)`, while the SPEC-ARMED arms print 452.25 MiB (MTP depth 2 bills three
|
||||
seqs per slot, so 9 cells) — so pool arithmetic alone under-prices the shape.
|
||||
|
||||
**The shape (2026-10-18):** `--ctx-size 786432 --parallel 3` = THREE
|
||||
conversations of 262,144, every one at the model's TRAINED window (Frank: "if
|
||||
@@ -1168,6 +1169,18 @@ linked runbooks.
|
||||
is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list,
|
||||
and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an
|
||||
alert that has been firing for weeks.
|
||||
- **A leftover bench drop-in silently hijacks the next deploy, and no health
|
||||
gate sees it.** The sanctioned bench-window method — a transient
|
||||
`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env` drop-in, the arm env
|
||||
`source`ing the repo env and then overriding it — outlives the bench unless
|
||||
it is retired. On 2026-10-18 a stale `frank-zealot.service.d/ptq1.conf` from
|
||||
an unfinished experiment made the cutover deploy boot a **different model at
|
||||
a different shape** (3 slots of 133,888, not 3 × 262,144) while `/health`
|
||||
answered 200, `/v1/models` echoed the alias (a flag, not a fact) and the
|
||||
Playwright gate passed 33/33. After deploying any recently-benched LLM
|
||||
service: `systemctl show <svc> -p DropInPaths` (must be empty), then
|
||||
`tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot ledger —
|
||||
and retire drop-ins to `~/.AGENTTRASH/<date>-<topic>/` + `daemon-reload`.
|
||||
|
||||
**Ops**
|
||||
|
||||
|
||||
+20
-13
@@ -65,9 +65,11 @@ UBATCH_SIZE=2048
|
||||
# the ledger said exactly that. HALF the 27B's 34,816 B/token, which is the
|
||||
# whole reason three native windows fit. f16 would bill 26,112 MiB at this pool
|
||||
# (against a 24,123 MiB card) and dies at cudaMalloc; the 24 DeltaNet layers
|
||||
# bill a fixed recurrent state per SLOT instead of per token — the ledger's
|
||||
# "RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)" = 150.75 MiB a
|
||||
# slot — which is why pool arithmetic alone under-prices the shape.
|
||||
# bill a fixed recurrent state per SLOT instead of per token — the PRODUCTION
|
||||
# ledger prints "RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)"
|
||||
# (~50 MiB a cell), while the SPEC-ARMED arms print 452.25 MiB, because MTP
|
||||
# depth 2 bills three seqs per slot (9 cells) — which is why pool arithmetic
|
||||
# alone under-prices the shape.
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
|
||||
@@ -103,18 +105,23 @@ SPEC_DRAFT_P_MIN=0
|
||||
# PRESENCE_PENALTY="" is the documented opt-out.
|
||||
PRESENCE_PENALTY=1.0
|
||||
NGL=99
|
||||
# Boot ledger (2026-10-18, this exact shape, nvidia-smi settled):
|
||||
# 17,876 MiB used / 6,249 MiB free of 24,123
|
||||
# Boot ledger (2026-10-18 PRODUCTION boot under the unit, verbosity 4):
|
||||
# 17,922 MiB used / 6,203 MiB free of 24,123 (nvidia-smi settled)
|
||||
# CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB
|
||||
# CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
|
||||
# CUDA0 RS 452.25 MiB (3 recurrent-state cells)
|
||||
# compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB
|
||||
# CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs)
|
||||
# compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf)
|
||||
# mmproj worst-case estimate 868.32 MiB (248.10 CUDA + 24.93 CPU compute
|
||||
# meta) — charged AFTER the engine's own fit check, as always
|
||||
# no clamp line, no cudaMalloc error
|
||||
# Measured on the same boot: 167.2 t/s greedy decode (/completion, 512 tokens),
|
||||
# 158.9 t/s single-stream chat, 280.2 t/s aggregate across THREE concurrent
|
||||
# streams (93.8 each), 2,877 t/s prefill at 106,418 tokens. The 27B record it
|
||||
# replaces: 76.6 decode, 109.4 aggregate across two streams, 691.7 prefill at
|
||||
# 166k — beaten on every axis. The FIRST request after a boot is a warmup
|
||||
# artifact (37.7 t/s class); never report it.
|
||||
# Measured on this boot (/completion timings, greedy, ignore_eos, 512 tokens):
|
||||
# 168.7 t/s decode (three runs 168.4-168.7) and ~300 t/s aggregate across
|
||||
# THREE concurrent streams (102.8 each). The trial arm's record (167.2 decode,
|
||||
# 280.2 aggregate, 2,877 t/s prefill at 106,418) stands; the 27B it replaces
|
||||
# gave 76.6 / 109.4 / 691.7 at 166k. The FIRST request after a boot is a
|
||||
# warmup artifact (37.7 t/s class) and a natural generation that stops early
|
||||
# makes a wall-clock aggregate read 4x low - measure with ignore_eos and read
|
||||
# the engine's own timings.
|
||||
# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV on the
|
||||
# 27B era measured 16,392 used / 7,733 free at 2 × 200,704; the same drop here
|
||||
# frees ~6.5 GiB at three windows. UBATCH_SIZE=1024 is the cheaper lever
|
||||
|
||||
@@ -53,11 +53,18 @@ Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-*
|
||||
PQ2_0 with the MTP head, 3 × 262,144 — was **deployed as production**, so the
|
||||
picker row below was re-keyed to it (see the next paragraph).
|
||||
|
||||
The `model` table carries exactly one decoration row, `frank-zealot`
|
||||
(context length 262,144 = the per-slot window since 2026-10-18; description
|
||||
re-keyed the same day to Ternary-Bonsai-2-9B PQ2_0 + MTP + Q8_0 vision mmproj —
|
||||
the row's `description` is live DB state, so a payload swap must re-key it in
|
||||
the database too, not only in the repo).
|
||||
The `model` table carries exactly two decoration rows, `frank-zealot` and
|
||||
`frank-dragoon`. Both the context length and the description live **inside
|
||||
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
|
||||
`{"description": …}`; there is no `context_length` or `description` column, so
|
||||
`sqlite3` one-liners against those names fail. Re-keyed 2026-10-18:
|
||||
`frank-zealot` → 262,144 with the Bonsai-2-9B description, and `frank-dragoon`
|
||||
→ 200,704 (its `params` had kept the pre-lowering 262,144 window from
|
||||
2026-10-17 while its own description already said 200,704). These rows are
|
||||
live DB state, so a payload swap must re-key the database too, not only the
|
||||
repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the
|
||||
stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the
|
||||
DB is in WAL and the app is live).
|
||||
|
||||
A third gate sits *after* the table, and it is the one that bites family
|
||||
accounts (2026-10-07): a model row is only visible to a non-admin user when
|
||||
|
||||
@@ -76,8 +76,10 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
|
||||
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
|
||||
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||||
instead — `RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB
|
||||
each — so pool arithmetic alone under-prices the shape.
|
||||
instead: the PRODUCTION ledger prints `RS buffer size = 150.75 MiB (3 cells,
|
||||
32 layers, 3 seqs)` ≈ 50 MiB a cell, while the SPEC-ARMED arms printed
|
||||
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples
|
||||
even the recurrent state. Pool arithmetic alone under-prices the shape.
|
||||
|
||||
## The shape (`scripts/frank-zealot.env`)
|
||||
|
||||
@@ -123,30 +125,36 @@ inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
|
||||
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
|
||||
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
|
||||
|
||||
## Boot ledger (deployed arm, verbosity 4)
|
||||
## Boot ledger (production boot 2026-10-18 13:19, under the unit, verbosity 4)
|
||||
|
||||
```
|
||||
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
|
||||
CUDA0 RS buffer size = 452.25 MiB # 3 recurrent-state cells, 32 layers, 3 seqs
|
||||
compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host
|
||||
mmproj 595 MiB + 248 MiB # charged AFTER the fit check
|
||||
nvidia-smi settled: 17,876 MiB used / 6,249 MiB free of 24,123
|
||||
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
|
||||
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
|
||||
CUDA0 RS buffer size = 150.75 MiB # 3 cells, 32 layers, 3 seqs
|
||||
CUDA0 compute buffer = 1409.02 MiB # + 1088.87 MiB CUDA_Host, 2.84 output
|
||||
mmproj worst-case (est.) = 868.32 MiB # 248.10 CUDA + 24.93 CPU, AFTER fit
|
||||
nvidia-smi settled: 17,922 MiB used / 6,203 MiB free of 24,123
|
||||
n_slots = 3, n_ctx_slot = 262144, kv_unified = false
|
||||
```
|
||||
|
||||
No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
|
||||
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
|
||||
never the projection**.
|
||||
|
||||
## Measured (deployed arm)
|
||||
## Measured (production boot, `/completion` `timings`, greedy, `ignore_eos`)
|
||||
|
||||
- **167.2 t/s** greedy decode (512 tokens, `/completion` `timings`); 158.9 t/s
|
||||
single-stream chat with the card's sampler.
|
||||
- **280.2 t/s aggregate** across three concurrent streams (≈93.8 each).
|
||||
- **2,877 t/s prefill** at 106,418 prompt tokens.
|
||||
- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
|
||||
(the trial arm printed 167.2; same shape, same noise band).
|
||||
- **~300 t/s aggregate** across three concurrent 512-token streams
|
||||
(102.8 t/s per stream, round 1 and 2 both) — the trial arm's 280.2.
|
||||
- **2,877 t/s prefill at 106,418 tokens** — the trial arm; the production boot
|
||||
has not re-run that depth.
|
||||
- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams,
|
||||
691.7 prefill at 166k. Beaten on every axis.
|
||||
- The **first request after a boot is a warmup artifact** (37.7 t/s class) —
|
||||
never report it.
|
||||
never report it. And never time a wall-clock aggregate on a generation that
|
||||
stops on its own: a 24-token countdown measured 72 t/s across three streams
|
||||
while the same shape with `ignore_eos` gave 300.
|
||||
|
||||
## Contract traps
|
||||
|
||||
@@ -186,6 +194,20 @@ never the projection**.
|
||||
7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the
|
||||
unit (llama.cpp needs the drain time to release slots and the card); the
|
||||
hand-booted arm must be killed by PID, never `pkill -f llama-server`.
|
||||
- **A leftover arm drop-in hijacks the production boot, and no health check
|
||||
sees it.** The sanctioned bench method is a transient
|
||||
`Environment=FRANK_ZEALOT_ENV_FILE=/tmp/<arm>.env` drop-in, and an arm env
|
||||
`source`s the repo env and then overrides it. On 2026-10-18 a stale
|
||||
`frank-zealot.service.d/ptq1.conf` — written 03:29 for an unfinished 27B
|
||||
**PTQ1_0** experiment — was still installed, so the cutover deploy restarted
|
||||
the unit and booted THAT: `--model …/Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf
|
||||
--ctx-size 401408 --parallel 3 --batch-size 3072 --ubatch-size 1536`, three
|
||||
slots of 133,888, 21,188 MiB used. `/health` was 200, `/v1/models` said
|
||||
`frank-zealot` (the alias is a flag, not a fact), and the Playwright gate
|
||||
passed 33/33 — none of those read the payload. **After deploying an endpoint
|
||||
that was recently benched: `systemctl show -p DropInPaths`, then
|
||||
`tr '\0' ' ' </proc/$pid/cmdline`, then the ledger — in that order.** Retire
|
||||
the drop-in to `~/.AGENTTRASH/<date>-<topic>/`, `daemon-reload`, restart.
|
||||
|
||||
## Compaction hub
|
||||
|
||||
@@ -197,6 +219,33 @@ the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--paralle
|
||||
a compaction request no longer serializes behind a live generation (the Gemma4
|
||||
era's `--parallel 1` law).
|
||||
|
||||
## Deploy ledger (2026-10-18)
|
||||
|
||||
- `./scripts/deploy.sh dankclaw --service frank-zealot` at 13:16 PDT; the e2e
|
||||
gate ran 33/33. The first boot was the hijacked arm above; the drop-in went
|
||||
to `~/.AGENTTRASH/20261018-zealot-ptq1-arm/` and the 13:19 restart is the
|
||||
ledger recorded here.
|
||||
- Live probes on the shipped boot: chat `27*59` → **1593**; `where_am_i` tool
|
||||
call fired with schema-correct empty arguments; 512 px solid-red PNG →
|
||||
**Red** at 278 prompt tokens with `process_mtmd` chunk encoding in the
|
||||
journal; `/props` reports `n_ctx 262144` and `presence_penalty 1.0`;
|
||||
`/metrics` answers 200 with 15 `llamacpp:*` families and
|
||||
`up{job="frank-zealot"}` is **1** in Prometheus for the first time in its
|
||||
retained history.
|
||||
- OpenWebUI picker rows re-keyed in the live DB (`Connection.backup()` snapshot
|
||||
`webui.db.bak-20261018` taken first; `frank-dragoon`'s `context.length` also
|
||||
corrected 262,144 → 200,704 — see [[OpenWebUI]]).
|
||||
- What the bench window left behind, and it is NOT garbage: the **PTQ1_0
|
||||
candidate was fetched and armed but never measured** —
|
||||
`scripts/download-frank-zealot-bonsai2-ptq1.sh` is committed, and its header
|
||||
states the case (1,201 MiB of freed weights against the publisher's own table
|
||||
showing dense-ternary unpacking LOSING on Ampere, with no GA102 row to settle
|
||||
the 3090). `/home/dank/models/Bonsai2-PTQ1_0/` (5,946,648,928 B + a hardlinked
|
||||
mmproj, 03:02), the arm env `/tmp/zealot-arm-current.env` and the
|
||||
`/tmp/arm-*.jsonl` 9B results are disk/`/tmp` state a reboot erases. Either
|
||||
measure the PTQ1_0 arm in a real window or record it as declined — leaving it
|
||||
armed and unmeasured is exactly what let it hijack a production boot.
|
||||
|
||||
## Fallback rungs
|
||||
|
||||
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
|
||||
|
||||
@@ -2156,4 +2156,37 @@ every restart since the cutover.
|
||||
picker row (DB `description` + context length 262,144) re-keyed. Rollback:
|
||||
revert the swap commit — the 27B payload and the build-10754 fork tree are
|
||||
both still on disk.
|
||||
- Deployed 13:16 PDT (`deploy.sh dankclaw --service frank-zealot`, e2e 33/33)
|
||||
— and the first boot was **not the shipped payload**: a stale
|
||||
`frank-zealot.service.d/ptq1.conf` from the bench window (written 03:29 for
|
||||
an unfinished 27B **PTQ1_0** experiment) pointed
|
||||
`FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit booted
|
||||
that arm: 3 slots of 133,888, 21,188 MiB used, batch 3072/ubatch 1536.
|
||||
`/health` 200, `/v1/models` echoed the alias, Playwright 33/33 — nothing in
|
||||
the gate reads the payload. Drop-in moved to
|
||||
`~/.AGENTTRASH/20261018-zealot-ptq1-arm/`, `daemon-reload`, restart at 13:19:
|
||||
`n_slots = 3, n_ctx_slot = 262144`, KV 13,056.00 MiB, settled 17,922/6,203.
|
||||
- Production measures (engine `timings`, greedy, `ignore_eos`): **168.7 t/s**
|
||||
decode (168.4/168.7/168.7) and **~300 t/s** aggregate across three 512-token
|
||||
streams (102.8 each). My first battery reported 117 and 72 t/s: it timed
|
||||
natural generations that stopped at 24 tokens — wall-clock aggregate on an
|
||||
early EOS reads 4× low. `ignore_eos` + the engine's own numbers is the only
|
||||
honest way to time this.
|
||||
- Ledger correction: the shipped shape prints **RS 150.75 MiB** (3 cells), not
|
||||
the 452.25 MiB my first-pass notes carried — that figure belongs to the
|
||||
SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot (9 cells). The
|
||||
drafter triples the recurrent state as well as billing 3,072 MiB of f16 KV.
|
||||
Corrected in AGENTS.md (topology row + architecture section), the env header
|
||||
and the runbook, with the production boot ledger replacing the trial's.
|
||||
- Verified live: chat `27*59` → 1593; `where_am_i` tool call fired; 512 px
|
||||
solid-red PNG → **Red** (278 prompt tokens, `process_mtmd` in the journal);
|
||||
`/props` `n_ctx 262144`, `presence_penalty 1.0`; `/metrics` 200 with 15
|
||||
`llamacpp:*` families and `up{job="frank-zealot"} = 1` — the first non-zero
|
||||
sample in that job's retained history.
|
||||
- OpenWebUI `model` rows re-keyed in the live DB (backup
|
||||
`webui.db.bak-20261018`): `frank-zealot` → 262,144 + the 9B description, and
|
||||
`frank-dragoon`'s `context.length` corrected 262,144 → **200,704** (it had
|
||||
kept the pre-lowering window from 2026-10-17 while its own description said
|
||||
200,704). Note the columns are `params`/`meta` JSON — the table has no
|
||||
`context_length`/`description` columns.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user