fix(frank-zealot): key the ledger to the production boot; record the arm drop-in hijack

The cutover deploy's FIRST boot was not the shipped payload: a stale
frank-zealot.service.d/ptq1.conf from the bench window still pointed
FRANK_ZEALOT_ENV_FILE at /tmp/zealot-arm-current.env, so the unit booted the
unmeasured 27B PTQ1_0 arm (3 slots of 133,888, 21,188 MiB) while /health
answered 200, /v1/models echoed the alias and the e2e gate passed 33/33.
Drop-in retired to ~/.AGENTTRASH/20261018-zealot-ptq1-arm/; the 13:19 restart
is the shape the docs now quote (n_ctx_slot 262144, KV 13,056.00 MiB,
settled 17,922/6,203).

Corrections the production ledger forced:
- RS buffer is 150.75 MiB (3 cells) on the shipped spec-off shape; 452.25 MiB
  belongs to the SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot.
  The drafter triples the recurrent state on top of its 3,072 MiB of f16 KV.
- Decode measured 168.4-168.7 t/s single-stream and ~300 t/s aggregate over
  three streams (102.8 each). Wall-clock timing of a generation that stops on
  its own reads 4x low (72 t/s for a 24-token countdown) - measure with
  ignore_eos and read the engine's timings.
- OpenWebUI stores both decoration values inside JSON columns (params/meta),
  and frank-dragoon's context.length had drifted to 262,144 against its own
  200,704 ceiling; both re-keyed in the live DB.

New law in AGENTS.md: after deploying a recently-benched LLM service, check
DropInPaths, then /proc/<pid>/cmdline, then the ledger. Health checks cannot
see the payload.
This commit is contained in:
Frank
2026-10-18 13:27:27 -07:00
parent e7f0d93c86
commit 695a5f3281
5 changed files with 145 additions and 36 deletions
+20 -13
View File
@@ -65,9 +65,11 @@ UBATCH_SIZE=2048
# the ledger said exactly that. HALF the 27B's 34,816 B/token, which is the
# whole reason three native windows fit. f16 would bill 26,112 MiB at this pool
# (against a 24,123 MiB card) and dies at cudaMalloc; the 24 DeltaNet layers
# bill a fixed recurrent state per SLOT instead of per token — the ledger's
# "RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)" = 150.75 MiB a
# slot — which is why pool arithmetic alone under-prices the shape.
# bill a fixed recurrent state per SLOT instead of per token — the PRODUCTION
# ledger prints "RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)"
# (~50 MiB a cell), while the SPEC-ARMED arms print 452.25 MiB, because MTP
# depth 2 bills three seqs per slot (9 cells) — which is why pool arithmetic
# alone under-prices the shape.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0
# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
@@ -103,18 +105,23 @@ SPEC_DRAFT_P_MIN=0
# PRESENCE_PENALTY="" is the documented opt-out.
PRESENCE_PENALTY=1.0
NGL=99
# Boot ledger (2026-10-18, this exact shape, nvidia-smi settled):
# 17,876 MiB used / 6,249 MiB free of 24,123
# Boot ledger (2026-10-18 PRODUCTION boot under the unit, verbosity 4):
# 17,922 MiB used / 6,203 MiB free of 24,123 (nvidia-smi settled)
# CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB
# CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
# CUDA0 RS 452.25 MiB (3 recurrent-state cells)
# compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB
# CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs)
# compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf)
# mmproj worst-case estimate 868.32 MiB (248.10 CUDA + 24.93 CPU compute
# meta) — charged AFTER the engine's own fit check, as always
# no clamp line, no cudaMalloc error
# Measured on the same boot: 167.2 t/s greedy decode (/completion, 512 tokens),
# 158.9 t/s single-stream chat, 280.2 t/s aggregate across THREE concurrent
# streams (93.8 each), 2,877 t/s prefill at 106,418 tokens. The 27B record it
# replaces: 76.6 decode, 109.4 aggregate across two streams, 691.7 prefill at
# 166k — beaten on every axis. The FIRST request after a boot is a warmup
# artifact (37.7 t/s class); never report it.
# Measured on this boot (/completion timings, greedy, ignore_eos, 512 tokens):
# 168.7 t/s decode (three runs 168.4-168.7) and ~300 t/s aggregate across
# THREE concurrent streams (102.8 each). The trial arm's record (167.2 decode,
# 280.2 aggregate, 2,877 t/s prefill at 106,418) stands; the 27B it replaces
# gave 76.6 / 109.4 / 691.7 at 166k. The FIRST request after a boot is a
# warmup artifact (37.7 t/s class) and a natural generation that stops early
# makes a wall-clock aggregate read 4x low - measure with ignore_eos and read
# the engine's own timings.
# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV on the
# 27B era measured 16,392 used / 7,733 free at 2 × 200,704; the same drop here
# frees ~6.5 GiB at three windows. UBATCH_SIZE=1024 is the cheaper lever