Compare commits

...
Author SHA1 Message Date
Frank 71c6b5e758 docs(openwebui): frank-dragoon enabled in the picker beside frank-zealot
Live DB trio applied on Dankclaw (enable idx 0, model row + global
access_grant, restart); verified /api/models as admin and role=user,
chat answers pong. Battery rewritten to gate on /api/models —
/api/v1/models/list lists only workspace models on 0.6.x.
2026-10-09 20:43:28 -07:00
Frank 5ced474eb9 docs(log): wire-rename wave — three served ids, OpenWebUI + Hermes re-key, bench drop-in retired 2026-10-09 17:10:25 -07:00
Frank 2992a7ddd6 docs: record the wire rename, the OpenWebUI re-key, and the frank-primary bench drop-in trap
- AGENTS.md Model Config: the Protoss names are now the ids on the wire
  (per-endpoint knob named); verify a rename on /v1/models, never on a chat
  response — radiance echoes the requested name
- OMP Harness.md: harness id == served id; one id per model everywhere
- Homepage.md: the docs-page "api id" annotations and the switchover promise
  are retired
- OpenWebUI.md: the id-keyed trio (model row, access_grant, api_configs
  model_ids) is a foreign-key set — re-key it in the same deploy; plus the
  contenteditable-composer probe note
- frank-secondary/frank-large/frank-primary runbooks: served ids
  frank-zealot/frank-dragoon/frank-archon, and the bench.conf drop-in that
  made every frank-primary deploy since 2026-10-08 04:09 restart the staging
  worktree behind a green health gate
2026-10-09 17:06:44 -07:00
Frank 7c77326c33 llm: complete the Protoss rename on the wire — served ids frank-archon/dragoon/zealot
The 2026-10-08 rename moved the harness model ids and the homepage cards;
the served aliases on the wire stayed frank-primary/frank-large/
frank-secondary. This closes the gap: the alias each endpoint announces is
now the family name, while the services, units, scripts, env files and
runbooks keep the old names (container identity stays stable).

- scripts/frank-secondary.env ALIAS, scripts/frank-large.env
  HALOGEN_MODEL_ID, scripts/frank-primary.env FRANK_PRIMARY_SERVED_NAME
  (+ the two serve-script defaults/gate messages)
- seams re-pinned: frank_secondary/frank_large/frank_primary unit bats
- fleet harnesses retargeted: bench-probe, deep-probe, calib-push,
  fleet-probes-swift15, autoresearch(+measure), bench-threads-batch-http,
  bench-decode
- shared/homepage/docs.html: api-id annotations and the "switchover will
  rename the ids" footnote removed — the page's names are now the ids;
  opencode/Pi/curl examples send frank-zealot
- dankclaw/hermes/config.yaml.example: frank-large -> frank-dragoon and the
  live 262144 context (Hermes routes by served id; see Hermes.md)

Consumers with live state re-keyed out-of-band in the same window:
OpenWebUI (model row, access_grant, openai.api_configs model_ids) and the
Hermes config.yaml.
2026-10-09 16:47:19 -07:00
Frank 65dec6a80e docs(authentik): provision kirby/ry/nic/debo/freya (pk 17-21); OWU pending-role first-login flip law 2026-10-09 11:49:58 -07:00
Frank f99b698d25 docs(LLM): 3090 worked example — archon w4dn8 methodology ports to GGUF-native quants; 24 GiB size law for 35B-A3B 2026-10-08 23:44:13 -07:00
Frank 757a90057f docs(LLM): recipe portability audit — qwen35moe plugin carries the calibration tap; 35B-A3B w4 fits one R9700 resident 2026-10-08 23:42:58 -07:00
Frank 009e8a885d docs(LLM): Quantization Production — 3090 as the unlocked card (runtime menu, Ampere ceilings, Playbook B safetensors ladder) 2026-10-08 23:38:49 -07:00
Frank 6b677139ad docs(LLM): Quantization Production reference — per-card self-quant playbooks, REAP/Heretic/abliteration/K_P prior art 2026-10-08 22:39:18 -07:00
Frank 184a5c665a homepage+docs: frank-archon runs our own GPTQ-w4 build, link our HF publication 2026-10-08 22:15:33 -07:00
Frank 958b10545e homepage+docs: bottom-pinned privacy disclosure chips, live OWU api-keys screenshot, docs steps for 0.6 UI, drop house-DNS/Pi-hole bullet 2026-10-08 18:07:31 -07:00
Frank c4dca68a03 homepage: link Model Information panel to the docs AI section
The panel title carries a right-aligned 'AI guide ->' link to
/docs#models (the running-models section), at the point of use.
Bare mode recolors the link; layout is flex baseline, doctrine-minimal.

Seam: homepage_config.bats 6/6; headless smoke census + screenshot.
2026-10-08 18:05:50 -07:00
Frank f29dbdf7c7 homepage: rounded corners, aligned title glyphs, greyed closed items, spliced dragoon tagline, inline privacy notes 2026-10-08 17:29:45 -07:00
Frank 98f11a19c1 docs+AGENTS: model ids officially frank-archon/frank-dragoon/frank-zealot (services keep old names) 2026-10-08 17:16:21 -07:00
Frank a30ac9457a merge: swift15 runbook LIVE 2026-10-08 05:02:10 -07:00
Frank 75695c312a swift15: AGENTS.md aligned to deployed shape (fp8 KV, spec 2, seqs 3, PLE fp8 n-gram) 2026-10-08 05:02:10 -07:00
Frank c507f91438 swift15: runbook LIVE + deployed verification + vision probe law 2026-10-08 05:02:10 -07:00
Frank b175f4afd1 swift15: vision probe reads reasoning_content fallback, 256-token budget 2026-10-08 04:57:31 -07:00
Frank 971e23a25c swift15: fix fleet-probes vision heredoc (quoted 'PY' ate $B); probe header 2026-10-08 04:57:10 -07:00
Frank 3ca64e14fc merge: swift15 KLD re-PASS at production shape 2026-10-08 04:50:28 -07:00
Frank f3dfea13ad swift15: KLD re-PASS at production MNBT 2048 (KL 0.0131, top-1 97.92%) 2026-10-08 04:50:23 -07:00
Frank 02fa1e22ae merge: swift15-radiance-w4dn8 — frank-primary GPTQ w4 (Swift1.5)
Conflict resolved: frank-primary row from the branch (w4 chain),
frank-secondary row from main (Qwen3.5-9B swap of 2026-10-07).
2026-10-08 04:46:11 -07:00
Frank 1aef7418ad swift15 cutover: AGENTS.md + runbook + log + fleet probes + KLD gate shape fix
- AGENTS.md: fleet table row + frank-primary section rewritten for the
  GPTQ w4 chain (1.0.13 serve/convert, 1.2.0 calib, int8 KV, vram pair).
- runbook: throughput gate PASS (136.0 t/s median @ 44.5k, 4.5x bar),
  fleet probes PASS, control baselines recorded.
- log.md: 2026-10-08 campaign entry with the four measured engine laws.
- kld-gate: shape matched to the reference recording, then reshaped for
  the w4 MNBT<=2048 file-tier law (pool 40960).
- fleet-probes-swift15.sh: tools/reasoning/response_format/vision walk.
2026-10-08 04:41:07 -07:00
Frank fa5a444b64 homepage: simplified console — flat navy, static down-state, no scanlines/lattice/frames/glows 2026-10-08 04:31:30 -07:00
Frank 90f678582e homepage: navy Protoss console — maximal octagon cornering, Share Tech Mono + Rajdhani, pictograms for CW/RomM/Copyparty/Marreta, dragoon teal 2026-10-08 04:17:33 -07:00
Frank cc0fdd15a1 swift15: env -> candidate payload, RAD floor 110000 (measured 116,416 MiB) 2026-10-08 04:11:02 -07:00
Frank f148ecab4a homepage: HF source links per model; docs reorder (AI last), arr stack out of family docs 2026-10-08 04:00:26 -07:00
Frank c2f3411342 swift15: campaign decode probe (43.6k greedy, timings-based) + runbook gate results 2026-10-08 03:58:17 -07:00
Frank 8c5bde7034 homepage: Protoss theme, brand-color icons, dragoon portrait, GameBoy RomM, OWU white art + search warn 2026-10-08 03:51:50 -07:00
Frank 22a7331100 homepage: icon swaps, clickable tiles, down pulse, header nav, compaction callout
- Icons: Copyparty -> improvised amber folder, Marreta -> crumbling
  wall (bricks + red crack), Calibre-Web -> book, OpenWebUI -> official
  logo.png from openwebui.com, Seerr -> official seerr.dev
  os_logo_filled.svg.
- Tiles with an open link are clickable whole (.panel.clickable: hover
  lift + glow; inner link clicks pass through).
- Any DOWN panel (app tile or model card) gets a red gradient wash and a
  slow pulsing glow (.panel.down, .mi-card:has(.badge.down)).
- docs / admin / account-settings buttons moved from the footer to the
  header (nav.top-links); footer keeps the deployment stamp.
- Zealot compaction advice promoted from the Use paragraph to a bordered
  glowing model.compaction callout (.mi-comp); component seam pinned to
  the new field.
2026-10-08 03:37:13 -07:00
Frank c1e192df00 homepage: app-card art, zealot-first order, common uncensored warning, admin identity gate
- Every app card carries hex-clipped art: 19 amber-tinted brand marks
  from the Simple Icons CDN plus 19 improvised monogram SVGs, wired via
  a per-service "art" field; homepage_config.bats pins art presence for
  all 32 apps.
- Model cards render zealot -> dragoon -> archon (registry order pinned
  in the seam).
- Dragoon and zealot share one common uncensored legal warning (adult
  content, harmful/hateful/illegal descriptions, sensitive offensive
  cybersecurity topics, severe legal risk, "don't ruin this server for
  everybody"), hover label on card and docs.
- Archon gets a bordered red permission callout: ask Frank before use.
- SearXNG obfuscation-not-anonymity note moved to a card-level hover
  warning (.svc-warn) and a docs hover note.
- Gitea: self-service signup flow; no need to ask Frank.
- Footer docs/admin/account-settings become glowing bordered buttons;
  the admin button is shown only to homepage-admins members via a new
  Caddy /whoami endpoint echoing the outpost-injected X-Authentik-*
  headers (spoof-proof; the /admin* 403 stays the real gate). e2e pins
  both sides.
- Docs naming swept: Protoss names primary, bracketed api ids, explicit
  note that a future switchover renames the ids themselves.
2026-10-08 03:21:26 -07:00
Frank 8375567de3 swift15: restore expert_tiered + disk-tier flags dropped in the gate rewrite 2026-10-08 03:18:04 -07:00
Frank 79c2e64c2c swift15: gate MNBT 3072 per Chief (w4 arena scales hard with MNBT; 32k step is absurd here) 2026-10-08 03:17:33 -07:00
Frank f20da6a6ff swift15: gate shape MNBT 16384 (w4 arena at 32768 costs 33 GiB), kv 2048, headroom 512 2026-10-08 03:16:45 -07:00
Frank 058ea6cca8 swift15: gate boots at the production pool (w4 gate_up experts are RAM-resident; disk tier refuses non-serving stored forms) 2026-10-08 03:14:20 -07:00
Frank 3f8531179e swift15: gate flags per engine (--kld-ref takes no corpus; --kld-out JSON parsed) 2026-10-08 03:12:18 -07:00
Frank 3c5d77d1fa e2e: count fused Model Information names in the user homepage spec
The Protoss model cards render as .mi-name inside one full-width panel;
the panel-name census now counts both render styles, keeping services.json
the single source of truth.
2026-10-08 02:51:09 -07:00
Frank 203b0be145 homepage: Protoss rename + fused Model Information component
The model cards become the StarCraft trio: frank-large -> frank-dragoon,
frank-primary -> frank-archon, frank-secondary -> frank-zealot (family
names only; endpoint/API ids unchanged). The three cards fuse into one
Model Information component: per-unit Protoss accent colours and hex-
clipped unit art, 15px type, clickable StarCraft unit-voice quotations
(original .ogg files from the fandom wiki, served from /sfx/), supervision
notes (dragoon/archon safe for autonomous use, zealot tight supervision),
compaction-model recommendation for frank-zealot, and uncensored-model
legal notes rendered as hover tooltips instead of wall text. docs.html
carries the same renames plus api-id mapping.

App-card updates: Seerr, Calibre-Web, RomM, Jellyfin set to WIP: DOWN
with Open links removed (new wip "down" badge state); Copyparty notes it
serves all other media apps content including WIP apps; Gitea and Plex
noted as needing logins separate from SSO; SearXNG noted as obfuscated,
not anonymous (upstream API keys tie to Frank).

Plumbing: Caddy @data allowlist grew /sfx/*; deploy_homepage copies sfx/;
homepage_config.bats pins the new probe names, the component fields,
art/voice-file presence, and the wip-down schema.
2026-10-08 02:40:45 -07:00
Frank 74da746c51 docs(homepage): record the capability equivalence notes 2026-10-08 01:32:35 -07:00
Frank 7066accc25 homepage: capability equivalence on the AI cards
Every fleet model card (and the #models docs section) now states, in
plain family-facing terms, what the model is equivalent to:
frank-large ~ Claude Sonnet 5 / OpenAI-Luna, frank-primary ~ Claude
Opus 4.8 / OpenAI-Terra (both Summer 2026 frontier), frank-secondary ~
last year (2025) frontier. Rendered as a new details.equiv line
(desc-equiv, styled in the head color, bare-mode clean); pinned in
homepage_config.bats.
2026-10-08 01:30:50 -07:00
Frank 76aaa2543b frank-large: raise ctx to 262144 to close the harness 400-zone
The harness clamps max_tokens to min(45056, contextWindow - est - 4096),
its chars/4 est under-counts the rendered prompt by ~3 % (measured:
152,938 -> 157,388 and 158,906 -> 165,301), and compaction fires only at
85 % -- so with window == engine ceiling 200192, every request in the
est ~146k-184k zone was a guaranteed 400. HALOGEN_CTX=262144 (native) +
KV pool 524288; harness window 217088 = ceiling - 45k absorbs the
estimate error (worst-case request ~219k <= 262144). models.yml updated
out-of-repo: contextWindow 217088, maxTokens 45056.
2026-10-08 00:40:12 -07:00
Frank 8220030ba1 swift15: canonical recipe = host copy the convert loaded (all 24 protected experts; committed copy was a partial splice) 2026-10-07 23:45:45 -07:00
Frank 5191c49967 swift15: KLD gate script (candidate vs bf16 ref, KL<=0.03 / top-1>=95%) 2026-10-07 23:43:06 -07:00
Frank f9330fa201 docs(homepage): record the @data allowlist law for new assets 2026-10-07 23:41:58 -07:00
Frank 4b71c014e9 caddy: serve /img/* from the homepage root
The @data path allowlist on the post-auth listener did not include /img/*,
so docs screenshots fell through to the index.html catch-all and the
browser got 200 text/html for every PNG (docs images broken live
2026-10-08). Pinned by the homepage_caddy seam.
2026-10-07 23:38:18 -07:00
Frank cee90eea52 homepage: amber CRT theme, VT323, current-fleet docs + API access guide
- style.css: green phosphor -> amber CRT-AMBIOS palette; docs styles for
  screenshots, code blocks, key warning (full + bare mode)
- index/admin/docs: VT323 from the Google Fonts CDN
- docs.html: rewrite #chat/#models for the live fleet (flash-server v2,
  radiance native FP8, Qwen3.5-9B Q6_K); new #api-access with OpenWebUI
  key walkthrough + Opencode and Pi configs; three OpenWebUI screenshots
- services.json: AI card details match the current fleet
- deploy-remote.sh: ship shared/homepage/img/ to /srv/homepage/
- docs: Homepage.md theme wave + OpenWebUI.md picker access-control law
2026-10-07 23:27:23 -07:00
Frank 73c37f4301 swift15: all experts through GPTQ w4 (plugin one-table-one-encoding law)
rad_arch_declare rejects the split tier (down block-fp8 + gate_up w4): the
qwen4exp_fp8 kernel serves one encoding per expert table (w4nl64a8h) with only
whole bf16 experts beside it. Recipe now sends gate_up AND down through the
GPTQ w4 rule (down calibrated by the Grams, gate_up uncalibrated), drops the
down-fp8 rule, and lowers the size floor 126000->108000 MiB for the ~19 GiB
lighter container. Convert script: checkpoint lives under models/ukisai/, and
the in-container path must keep the subdir (basename stripped it).
2026-10-07 21:31:10 -07:00
Frank 6632d0a2f6 docs(openwebui,authentik): frank-secondary-only picker, API gating, artem account
OpenWebUI model picker narrowed to exactly frank-secondary (connections
frank-large/frank-primary disabled in openai.api_configs, model table rebuilt,
stale frank-small model_id retargeted to the served alias). Documented the
auth-gated API surface (anonymous probes all 401; keys per account).
Authentik roster gains artem (pk 16) in the human-user template.
2026-10-07 18:49:22 -07:00
Frankandomp caf5e1e0ca docs(authentik): record the human-user template and alex's recreation
kassie (pk 14) and alex (pk 15, deleted pk 7 then recreated) now share the
ridley shape: <user>@franksdank.cloud + Non-Admin Users + jellyfin_users +
openwebui_users. Audit before destroy: the old alex row carried no MFA
device, token, entitlement, or proxy session - checked across the MFA
device tables, authentik_core_token, and providers_proxy_proxysession
(the fleet's column names differ from the ORM names: events.user is
denormalized JSON, certificatekeypair has no user_id). Events survive the
delete, so the audit trail keeps its shape.

Co-Authored-By: omp <omp@franksdank.cloud>
2026-10-07 18:20:11 -07:00
Frankandomp 5278f5be26 frank-secondary: record the deployed boot, keep the unit description honest
Deploy verified: nvidia-smi 21,436 MiB used / 2,689 free at ubatch 2048 -
the MiMo-priced projection (~21,430/~2,700) landed within 11 MiB, and the
verbosity-4 ledger confirms the KV law to the MiB (CUDA0 KV buffer
12,544.00 MiB = K(f16) 6,272 + V(f16) 6,272 = 32 KiB/token over the
401,408 pool), compute buffers 1,136 MiB CUDA0 + 848 MiB host, mmproj
879 + 248 MiB. Playwright gate: 33 passed.

Two lessons recorded: the 203/EXEC boot loop my shebang-less rewrite
caused (journal signature + the head -1 check), and that llama.cpp's
"n_ctx_seq < n_ctx_train" line is informational when the context is BELOW
the trained window - the Gemma4 drafter warning was the opposite case.

The unit's Description named the Swift-1.5 era (and said 2x131072/q8_0/
draft-mtp); it now names the Q6_K shape. Cosmetic, so it is NOT deployed
as its own restart: the next frank-secondary deploy relinks it.

Co-Authored-By: omp <omp@franksdank.cloud>
2026-10-07 17:33:21 -07:00
Frankandomp ec6184de9a frank-secondary: restore the shebang my rewrite dropped (unit failed 203/EXEC)
Co-Authored-By: omp <omp@franksdank.cloud>
2026-10-07 14:50:49 -07:00
Frankandomp 1d062f06a3 frank-secondary: swap Gemma4-26B-A4B Q4_K_M -> Qwen3.5-9B-Uncensored Q6_K 2x200,704
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention
+ 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less,
natively multimodal) costs ~7 GiB of weights where the 26B MoE cost
~16.6 GiB. The freed room buys back the second conversation slot and
exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K +
f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable
is impossible), vision via the publisher's BF16 mmproj, ubatch 1024
(the CUDA decode-optimal value), verbosity 4, publisher sampler recipe
(0.6/0.95/20/0) in models.yml.

Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB
used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch
2048. Compaction stays routed here; window math is unchanged (maxTokens
16,384 keeps the worst-case harness request inside the slot).

Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the
rollback; its runbook is retired in place, keeping the attention_k_eq_v
pair-billing law.

Co-Authored-By: omp <omp@franksdank.cloud>
2026-10-07 14:47:32 -07:00
Frank b51e078e36 frank-secondary: record the deployed f16-K boot ledger, correct the k_eq_v billing law
Verbosity 4 paid for itself: the engine's own llama_kv_cache lines give the
exact accounting — the cache bills the K+V pair TWICE under attention_k_eq_v
(6.125 B/elem at f16 K + q8_0 V), reproducing 3,001.25 MiB (5 full layers at
n_head_kv 2) + 306.25 MiB (25 SWA rings at n_head_kv 8, window 1024) to the
byte. The earlier "bill K-only" note was wrong; full layers have 2 KV heads,
not 8.

Deployed boot: 21,868 used / 2,257 free — the f16-K raise (+2,161 MiB KV) is
funded by the ubatch 2048 -> 1024 cut (~616 MiB of compute buffer) plus
variance, so exact K came essentially free. Measured decode 139.9 t/s temp 0;
vision probe returns Red at prompt_tokens 151.
2026-10-07 13:58:30 -07:00
Frank 68a525740d frank-secondary: swap Swift-1.5 RCO-IQ3_XXS -> Gemma4-26B-A4B QAT Q4_K_M
Payload: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored Q4_K_M (16,796,015,520 B)
+ the publisher's BF16 mmproj, /home/dank/models/Gemma4-26B-A4B/.

Shape: ONE 200,704-token slot (--parallel 1; 2x200,704 does not fit the
24 GiB card), f16 K + q8_0 V under flash attention, batch 4096 / ubatch
1024, --log-verbosity 4, MTP OFF (the gemma4-assistant file is on disk,
not loaded: its draft KV is built against the target's full window,
measured +1.9 GiB -> 23,806/319 free, no margin beside the f16-K raise).

Measured boots (nvidia-smi): q8_0 K + q8_0 V / ubatch 2048 / MTP off =
21,890 used / 2,235 free; with the MTP assistant = 23,806 / 319.

Engine facts recorded: gemma4 hybrid attention (25 sliding-window + 5
full-attention layers) so KV scales with 5 layers only, and
attention_k_eq_v is honoured by build 5fc4f3c (pool bills K-only; the
measured q8_0 pool = 2.05 GiB = the K-only figure).

Serve script gained VERBOSITY (+ --log-verbosity in the argv), the
SPEC_DRAFT_FILE drafter path, and defaults/lore retargeted to Gemma4.
deploy-remote.sh payload gates follow the env. The frank-secondary bats
seam is rewritten for the new contract (argv-based) and fixes a
pre-existing vacuity: serve_argv regenerates the test env, so
mutations made before it were silently undone - tests that mutate now
exec through run_stage. frank-secondary is the fleet compaction hub
(200,704 is the only window that fits a worst-case compaction input).
2026-10-07 13:48:54 -07:00
Frank 5b7b3ccb90 frank-secondary: swap to Gemma4-26B-A4B Q4_K_M, 1x200704, q8_0 KV, MTP via separate assistant file
- MODEL_FILE: Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS -> Gemma4-26B-A4B Q4_K_M
- CTX_SIZE: 229376 (2x114,688) -> 200704 (1x200,704)
- PARALLEL: 2 -> 1
- CACHE_TYPE_K/V: q8_0 (unchanged, FP8 class)
- SPEC_TYPE: draft-mtp (unchanged)
- SPEC_DRAFT_FILE: new — separate MTP assistant file (mtp-gemma-4-26B-A4B-it.gguf)
- serve script: add --spec-draft-model support for separate draft files
2026-10-07 12:41:07 -07:00
Frank 2137080d1c docs(index): top-level link for the staged Swift15 runbook (seam 135) 2026-10-07 04:42:36 -07:00
Frank f747cb0dab docs(runbook): staged Swift15 w4dn8 runbook skeleton + index link
Contract facts frozen (payload chain, recipe tiers, shape, gates);
MEASURED@cutover slots for bench/KLD numbers.
2026-10-07 04:40:31 -07:00
Frank db8faa68ff docs(worklog): KLD reference complete (rows 50235), script-swap law third occurrence 2026-10-07 02:49:14 -07:00
Frank cf6c62d44a fix(kld-then-calib): restore S= path clobbered by the SKIP_KLD edit 2026-10-07 02:45:28 -07:00
Frank 0aafad5f2c fix(kld-then-calib): SKIP_KLD resume + rows-50235 receipt gate
The KLD engine exited clean with the record complete (kld.json rows:
50235, 24 GiB, 5 files), but the wrapper died rc=127: a mid-run scp of
the calib-shape edit shifted bash's lazy-read offset — the fleet's own
script-swap law, biting on the same class as the autoresearch collision.
Resume path: SKIP_KLD=1 verifies the receipt (rows + size + files) and
goes straight to calib. Never scp a script a running process owns.
2026-10-07 02:44:48 -07:00
Frank a3f45ce028 docs(campaign): interim work log for the Swift1.5 w4dn8 campaign
Execution facts, blocker trail (plugin-tap probe law, weights-slab
formula, KLD_MOUNT), engine verdict, and the post-chain order. Linked
from the docs index per the completeness seam; the cutover runbook
replaces it.
2026-10-06 23:42:32 -07:00
Frank 45de2fd47f perf(run-swift15-bf16): calib at the vendor big-step shape
MNBT 65536 / seqs 8: MOE-W4.md:244 measures 640 tok/s at a 32k step
vs 930 at 55k — the KLD record's own rate confirmed step size drives
disk-tier prefill. 131k resident tokens fits the 3072 pool (~237k
tokens/rank at fp8). Cuts the 10M-token calibration from ~4.3 h to
~3 h.
2026-10-06 23:39:10 -07:00
Frank fa858f4c10 feat(campaign): protected-cast applier, recipe markers, KLD gate runner
apply-protected-cast.sh splices derive-protected-experts.py output into
the recipe's PROTECTED-BEGIN/END section (refuses to re-derive, trashes
the pre-edit recipe). kld-gate-swift15.sh boots the candidate on the
serving pin in --kld-ref mode (pool fully resident, no disk tier) and
enforces mean KL<=0.03 / top-1>=95% from the JSON report.
2026-10-06 22:41:18 -07:00
Frank 07af1a9366 feat(kld-then-calib): chain KLD record -> 1.2.0 calib serve -> 10M push
One-GPU serialization: the chain waits for the kld driver log to close
rc=0, sanity-checks kld-ref, boots the calibration server, pushes the
corpus at parallel 16 (4 engine slots; queue absorbs the rest), then
kills the server so its trap restores production.
2026-10-06 22:39:14 -07:00
Frank f0d0614ce6 fix(run-swift15-bf16): define KLD mounts (were unbound-empty)
kld died: 'cannot read --kld-corpus /data/kld.jsonl' — KLD_MOUNT was
referenced but never defined, so the corpus/ref-dir mounts silently
expanded to nothing. Add them (nested order: after the /data mount).

feat(convert-swift15-w4dn8): final recipe convert on 1.0.13

Convert with the SERVING pin so the .rad crosses no version boundary;
only the Gram files cross (RADGRAM2 magic, gram.r%d.%s.bin naming and
calib_gram/calib_resolve/calib_stem are byte-identical between the
1.0.13 and 1.2.0 libquant). Host-backend run: prod can stay up.
Gates: grams present (>=48), bf16 container deleted, disk floor,
rad-info gptq receipt, size floor 126000 MiB.
2026-10-06 22:37:16 -07:00
Frank 9aed9987b6 fix(run-swift15-bf16): kv pool 3072 both offline arms
The weights slab derives as card - headroom - kv, and the disk tier
refuses under 9.57 GiB (4.21 layer buffers + 5.36 fixed). 4608 starved
it (8.68 budget, DID NOT FIT). 3072 leaves 10.2 for weights; 4x16k fp8
slots fit the pool, recording is prefill-bound so slot count only
queues pushes.
2026-10-06 22:33:19 -07:00
Frank 4048dc8630 fix(run-swift15-bf16): per-mode engine pins + explicit KV pools
kld stays on 1.0.13 (record on the scoring engine) with --vram-kv-mib
4608 + 16 slots: the ratio-derived pool came up SHORT by 217 MiB at
32x16k. calib moves to 1.2.0: the RADIANCE_CALIB_TOKENS tap is a 1.2.0
qwen4exp plugin feature (1.0.13 plugin has the DIR half only), and
Grams are bf16 activations - engine-invariant. --vram-kv-mib 9984
covers 64x16k fp8 slots.
2026-10-06 22:31:17 -07:00
Frank 9bc40fd899 fix(run-swift15-bf16): drop duplicated radiance arg (image entrypoint already runs radiance) 2026-10-06 22:25:01 -07:00
Frank bb60e8524f run-swift15-bf16: boot the bf16 container for KLD record / calibration (vendor disk-tier shape) 2026-10-06 21:46:05 -07:00
Frank 319dd0f7da derive: restore the header read 2026-10-06 21:42:16 -07:00
Frank 650dfff5a2 derive: RADGRAM2 header is magic,k,rank,rows,domain (28 B) 2026-10-06 21:42:00 -07:00
Frank de70320cde derive-protected-experts: RADGRAM2 scan + 3%/3x law + parity fill from calib recording 2026-10-06 21:41:16 -07:00
Frank 910c54b6ff frank-primary: keep 1.0.13 engine pin — 1.2.0 bench rejected (accept 0.7123 vs 0.7549, decode -2.9%; vram pair exonerated, ledger-exact) 2026-10-06 21:35:39 -07:00
Frank 8c887e6fc9 calib-push: drive calib.jsonl through scratch prefill until the tap writes Hessians 2026-10-06 21:32:56 -07:00
Frank 49dd2be4cd corpus: handle codeparrot content column + per-class caps
- load_parquet_texts: read codeparrot-clean 'content' column (schema-robust)
- per-class HF caps (prose 60k for short ~66-tok rows, code 40k for long files)
- pyarrow 18: ParquetFile.iter_row_groups removed -> read_table().to_pydict()

Produces 10,014,932 calib tokens (4.5M code/3.0M prose/2.5M tools) and
1,800,816 KLD tokens (197 prompts, all >=8192, score_from=n-256) from real
HF sources (codeparrot-clean + wikitext-103).
2026-10-06 21:16:16 -07:00
Frank a11f703c49 swift15: corpus generator v2 — HF code/prose ingest, richer tool JSON, 8k KLD prompts 2026-10-06 20:58:32 -07:00
Frank 6cdf88f611 swift15: restore the numpy import the census edit clobbered 2026-10-06 20:41:33 -07:00
Frank 259fc5505d swift15: PLE audit adds over-cap census + next-bf16 scale suggestion 2026-10-06 20:41:07 -07:00
Frank 3e46d51df8 swift15: BF16 absmax bits belong in the HIGH half of the float32 2026-10-06 20:39:21 -07:00
Frank d062a376b2 swift15: corpus generator (10M calib + 1.8M KLD, Froggeric-rendered, disjoint pools) 2026-10-06 20:38:26 -07:00
Frank 95219cf4c1 swift15: release the numpy view before closing the PLE audit mmap 2026-10-06 20:35:51 -07:00
Frank fee66187a1 swift15: PLE fixed-scale audit (streaming BF16 absmax vs 448*scale cap) 2026-10-06 20:33:55 -07:00
Frank 0258ae0897 frank-primary: pin radiance v1.2.0, translate expert-vs-cache-ratio to vram knobs
Engine v1.2.0 (Codeberg e95d397, Docker Hub tag 1.2.0, index digest
665775ed...) removes --expert-vs-cache-ratio. The explicit
--vram-weights-mib 26071 / --vram-kv-mib 3543 pair transcribes the live
1.0.13 boot ledger at ratio 0.82 (weights 25.46 GiB, KV 3.46 GiB per
card); bracket arms scale KV linearly in (1-r): 0.90 -> 1968, 0.95 -> 984.
Serve script, autoresearch harness, and bats gates follow; the bats
shape asserts are realigned to the deployed seqs=3/spec=2 sweep result
(95923d3, df6b5bb) they failed to track.
2026-10-06 20:31:46 -07:00
Frank 9aee00f2b3 swift15: fetch script + w4dn8 expert recipe for the frank-primary campaign
- download-frank-primary-swift15.sh: ukisai/Swift1.5-Qwen3.8-Flash-Next
  (BF16, 131 shards + index + configs, 360037355038 B = 335.3 GiB, ungated)
  via the vendored hfget.sh; verify = listed sizes + 131-shard weight-map
  count + every shard's sha256 against the Hub LFS oid. Pattern cloned
  from scripts/download-frank-primary.sh.
- qwen4exp-swift15-w4dn8-i8-hc8m.recipe: routed experts with DOWN at
  block-fp8 (rtn codes=fp8_e4m3 block=128x128 scale=bf16, the served
  FP8-native tile form; moe_gemm_fp8a8 reads it; minicpm5-2b is the
  shipped bf16->fp8a8 precedent) and gate_up at w4nl64 GPTQ cd=3;
  hc/MTP-fc fp8, trunk i8, lm_head i8, PLE fixed scale, draft_head u2
  (inert). Protected-expert cast lines carry the BASE checkpoint's names
  as a warm start; the derivation step rewrites them from Swift's own
  calibration recording before the recipe is ever run (law documented
  in the file).
2026-10-06 19:19:40 -07:00
Frank 41d2f56269 docs: frank-secondary at ubatch 2048 — Measured section, rollback ladder, fleet docs
23,120 used / 1,005 free at boot, 763 settled after a vision probe
(encoder buffers resident). Rung cost 746 MiB, not the transferable
616 prediction. Prefill baseline 1,209 t/s @ 3,760 recorded; decode
unchanged as the law says. 2048 = ceiling for this shape.
2026-10-06 18:40:26 -07:00
Frank d8da7a9eb7 frank-secondary: ubatch 2048 (the second rung inside the 112k headroom)
Frank's order the same night. The 1024->2048 rung measured +4.6 %
prefill for 616 MiB on the 3090 (2026-10-05 boot pair); the 112k cut
left 1,751 MiB free, so it rides with ~1.1 GiB predicted margin.
Decode is ubatch-invariant; batch 4096 >= 2048 keeps the seam happy.
2026-10-06 18:30:35 -07:00
Frank 44e7a8dfaf docs: frank-secondary at 2x114688/ubatch 1024 — runbook renamed + rewritten, fleet docs updated
- runbook git mv ...-mtp-131072.md -> ...-mtp-114688.md; new Measured
  section (22,374 used / 1,751 MiB free, decode 58.2/66.9 t/s, confounded
  vs 2x131072 noted), rollback ladder, trail entry
- AGENTS.md: topology row, ACTIVE section, contextWindow 114688, runbook
  index; mmproj name corrected to the publisher's mmproj-BF16.gguf
- LLM/index.md, index.md, mimo runbook link, log.md entry + path fix
2026-10-06 18:13:05 -07:00
Frank f2575a1018 frank-secondary: 2×114688 ctx (112k per-slot), ubatch 1024
Lower per-slot context from 131,072 (128k) to 114,688 (112k) to free
~1 GiB of KV, which funds the ubatch 512 -> 1024 increase (prefill
throughput). The per-slot window is the product contract, so models.yml
contextWindow follows to 114,688.

- CTX_SIZE: 262144 -> 229376 (2 × 114,688)
- UBATCH_SIZE: 512 -> 1024
- Test 194: per-slot 131072 -> 114688
- KV budget test: 131072 -> 114688, ubatch 512 -> 1024
2026-10-06 16:38:58 -07:00
Frank 87721ab0fe rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:

- Service trio: scripts/{frank-small.env,frank-small.service,
  serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
  frank-secondary-*.md (9 files); the FreeToken runbook stays
  freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
  LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
  3090, and the Dankserver service that became frank-medium) are
  preserved verbatim — they describe what the service was called
  at the time.
2026-10-06 15:37:13 -07:00
Frank c71a3066c0 frank-large: host-RAM pin-gate preflight in serve-frank-large.sh
The 2026-10-06 cutover crash-looped frank-large 8 boots (03:09:39-03:11:42):
the engine's pin gate refused to start while host MemAvailable was ~74.7 GiB,
3.34 GiB short of the 78.11 GiB it needs (62.11 GiB checkpoint + 16 GiB floor).
It self-resolved in ~2 min when the dying 27B engine freed RAM, but left the
host with only ~1.8 GiB MemAvailable - a restart would crash-loop again.

serve-frank-large.sh now checks MemAvailable against the pin gate BEFORE
podman run and WAITS up to FRANK_LARGE_RAM_WAIT (default 600 s) for the RAM
to clear, so a transient dip boots once instead of crash-looping; if it never
clears it refuses loudly. The 16 GiB floor is the engine's (not
HALOGEN_HOST_RESERVE_GIB=20, a separate overall-budget knob).

frank_large_unit.bats: FRANK_LARGE_MEMINFO seam + a refuse test. Runbook +
log: the pin-gate law, the incident, and the fix.
2026-10-06 12:58:34 -07:00
Frank df6b5bb6e8 MAX_SEQS 2->3: decode 33.06 t/s (+2.0% over depth-2 32.40; +4.2% over baseline). Bigger batch headroom co-schedules concurrent decode streams (decode is weight-streaming bound, so a fuller batch amortizes the weight read). KV pool 600k tokens (3x200k), above the 400k floor. Draft acceptance unchanged (0.755).
Result: {"status":"keep","decode_tps":33.0578,"draft_accept":0.7549,"prefill_tps":452.9596,"ttft_ms":96315.4258}
2026-10-06 04:18:53 -07:00
Frank 95923d3a98 SPEC_TOKENS 3->2: decode 32.40 t/s (+2.1% over 31.72). Acceptance 0.755 (between depth-3 0.831 and depth-4 0.497) — shallower drafts verify cheaper, net positive. Depth 2 is the decode optimum. Prefill/TTFT shifts are cold-run noise (prefill is orthogonal to spec depth).
Result: {"status":"keep","decode_tps":32.4031,"draft_accept":0.7549,"prefill_tps":372.9299,"ttft_ms":116984.442}
2026-10-06 04:09:39 -07:00
Frank 81e6122891 frank-primary: autoresearch bench harness (decode t/s @32k)
autoresearch.sh (dev box) scp's the candidate scripts/frank-primary.env
+ the bench script to dankserver and runs it over ssh.
scripts/autoresearch-frank-primary.sh (dankserver) follows the bench
policy: stop frank-primary (relink unit: stop, not mask), boot a
scratch radiance container (arbench) on the same two cards with the
candidate argv, poll /health, run ONE self-contained greedy completion
(fixed ~140k-char real-text prompt -> ~32k tokens, 256-token decode),
parse the engine timings into METRIC lines, tear the scratch down, and
restore production via a trap. Fresh prefix-cache tier per run for cold
prefill. Baseline validated: decode_tps=31.70, prefill_tps=465.4,
ttft_ms=93745, draft_accept=0.831 at prompt_n=43627.
2026-10-06 03:53:55 -07:00
Frank 2d89bf97cc docs: frank-large flash-server v2 cutover — runbook, index, AGENTS.md
- New runbook: frank-large-flash-server-v2-swift15.md (engine, shape,
  memory model, engine laws, boot ledger, probes, rollback)
- LLM/index.md: active entry + retired halogen 27B entry
- Vault index: link the new runbook from the LLM Fleet section
- AGENTS.md: fleet topology table, Model Config section,
  Runbook Index, retired table (halogen 27B → 10-06)
2026-10-06 03:35:27 -07:00
Frank 20d7182f4b docs index: link frank-large flash-server v2 page
The writer's 31b1ec3 added LLM/frank-large-flash-server-v2-swift15.md
but did not wiki-link it from index.md, which failed the docs-index
seam (test 135). Add the link as the new ACTIVE frank-large runbook
(supersedes halogen 0.1.4).
2026-10-06 03:26:43 -07:00
Frank 0f9faf05ea radiance runbook: MNBT A/B verdict + rad-tune tune-dir bind-mount
Snapshot of a parallel session's completed docs edit (mtime 03:07,
was blocking the deploy tree-clean gate): --max-num-batched-tokens
2048 A/B-verified (231/460/448 t/s cold prefill ladder), rad-tune
cache survives via FRANK_PRIMARY_STATE_DIR/tune bind-mount
(36 GEMM rows, generated 2026-10-06).
2026-10-06 03:08:30 -07:00
Frank 31b1ec397b frank-large: cutover to flash-server v2 (abliterated Swift1.5)
Engine: halogen-flash-server 0.16.2, digest-pinned (pulled 0c61bf84).
Payload: Quat3rnion qwen38-flash-next-v2-swift15-abliterated.hgn
(66,687,678,464 B, sha256-verified) + REQUIRED n-gram table
(51,200,246,144 B @peonist 5cc17cea) + vision tower (897,916,416 B
@peonist e053f488) + shared tokenizer (6 files). Shape: 2 slots x
200192 (KV pool 400384 positions, ~10.7 GiB), MTP always-on
(HALOGEN_SPEC_ADAPT=0, card-mandated), server-side sampler at the
card's 1.0/0.95/20, MAX_TOK 8192 with the 65536/7200 coupled pair,
vision ON. Rollback = revert this commit (27B payload stays on disk).

- serve script: byte-exact gates on all three .hgn files + flat
  tokenizer, foreground podman run (ledger to the log file)
- deploy-remote: twin gates for n-gram/vision, flashnext fetch pointer
- seam: 19 tests rewritten (spec-adapt law, pool product, compaction
  math, render shape); full suite 362/362 green
2026-10-06 03:07:40 -07:00
Frank 7fef3cb907 frank-primary: persist rad-tune kernel cache on a host mount
The engine's kernel selector reads $RADIANCE_HOME/tune/<machine>.tune at
boot; $RADIANCE_HOME defaults to /opt/radiance, which lives in the image,
so a cache written by a --rm scratch container dies with it. Pre-create
$FRANK_PRIMARY_STATE_DIR/tune and bind-mount it at
/opt/radiance/share/radiance/tune so the rad-tune cache (generated in a
stopped-service window) survives and is read at every production boot.
2026-10-06 03:02:51 -07:00
Frank bb3532dfec log: 2026-10-06 frank-small wave (two reshapes; RCO-IQ3_XXS + MTP-on, 489 MiB free, draft ctx ~2.4 GiB measured) 2026-10-06 01:41:23 -07:00
Frank 0f7274e55a frank-small docs: runbook renamed + rewritten for the RCO-IQ3_XXS MTP-on shape (489 MiB free, 63.3 t/s recipe, draft ctx ~2.4 GiB measured); AGENTS, vault indexes, models.yml comment, MiMo cross-link retargeted 2026-10-06 01:33:57 -07:00
Frank da36fa32cf frank-small: Swift-1.5-27B Uncensored RCO-IQ3_XXS with MTP back on (two 131072 slots, ubatch 512, q8_0 KV, publisher's BF16 mmproj) 2026-10-06 01:10:43 -07:00
Frank 713b126639 frank-primary: cutover measured — docs reflect the live service
AGENTS.md: row + section carry the measured 177,109 MiB container, floor
177000, live cutover 2026-10-06. Runbook: the three convert laws the live
runs proved (scale-name, file-shadow, symlink-through), the measured boot
(125.26 GiB page-in 38.1 s, PLE pin 47.68 GiB/43.9 s, ~2 min to health),
the MTP/vision/tool/reasoning contract results, and the SSD tier fix.
2026-10-06 01:06:11 -07:00
Frank 2d9cecae6d frank-primary: pre-create the SSD prefix tier path
The engine's create is not recursive: booting with only the state dir
mounted left the SSD prefix tier off ('cannot create
/data/kvcache/frank-primary: No such file or directory'). The serve
script now mkdir -p's the full path before podman run.
2026-10-06 01:00:54 -07:00
Frank fadad18da1 docs: frank-large FA_TQ A/B rejected at 80k; deep roles rerouted to frank-primary
Stop-window A/B, same 80k filler prompt, 128-token essays, runtime
timings: HALOGEN_FA_TQ=256 (upstream issue #12's carve workaround) moves
decode ±0.3% (noise, draft acceptance identical) and costs 9.5% prefill
on this topology. The depth collapse sits deeper than the FA tile knob;
the default stands. The harness's slow/advisor roles moved to
frank-primary (fp8 KV 7.32 KiB/token = 8.7x less KV read at depth).
2026-10-06 00:59:14 -07:00
Frank 306ff32427 frank-primary: rad floor under the measured payload (177109 MiB)
The first deploy refused: floor 178000 > actual 177109 MiB - the plan-only
estimate (172.76 GiB) prices the data planes but not the container's
metadata tail. Floor 177000 sits just under the real payload.
2026-10-06 00:55:30 -07:00
Frank d39424ffba docs: frank-large cache pin verified warm (16x prefill) + depth law
Probe on the live endpoint (2026-10-06): 17.3k cold prefill 27.0 s ->
warm turn resumes at cache_n 16,384, prefill 1.7 s. The boot WARNING
'smaller than ONE full-context entry' sizes against the native 262,144
window, not our 200,700 slot — documented as expected in the checklist.
Tuning notes record the depth law (decode 5.5 t/s @ 46k -> 3.3-3.8 @ 80k
vs 30-50 shallow; KV reads ~8 GB/s effective; no supported flag touches
decode) and the AGENTS.md unit paragraph regains the unit-facts
parenthetical the reflow dropped.
2026-10-06 00:34:59 -07:00
Frank 3eb4fa2b62 frank-primary: rad floor gate to the measured 172.76 GiB container
Plan-only measured the .rad at 172.76 GiB (176,906 MiB) - the checkpoint's
own fp8 mass, byte-stacked experts plus the PLE re-encoded at the
checkpoint's scale - not the ~161 GiB first estimate. Floor 158000 -> 178000.
2026-10-06 00:29:23 -07:00
Frank ba67ca5c9e frank-large: pin the prompt cache (16 GB) and capture the engine ledger on disk
Auto cache-sizing is OFF on this box: the boot ledger refused it at
36.9 GB (2 x full-entry) floor vs 35.7 GB free, so every deep agent
request re-prefills its whole 79-81k context (84.9-179.8 s measured on
serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. Pin
HALOGEN_CACHE_MB=16384 (>= one full-slot entry at ~70 KB/token) — the
documented override; the per-store reserve re-check keeps the host safe.

Also: the serve script's 'podman run -d > LOG' captured only the
container ID — the runbook's boot-ledger grep read a one-line hex file
and the ledger existed nowhere on disk. Foreground 'podman run' makes
the ledger land in the log file as documented; exit-code and ExecStop
semantics are unchanged. Test seam updated for both.
2026-10-06 00:26:28 -07:00
Frank 8343397584 frank-primary: merged PLE file must sort before the raw shards
The reader opens every index-named file in sorted order and emplaces
EVERY tensor each file physically holds, first-wins (checkpoint.cpp
add_shard + shard_list). The raw model-000NN shards carry tensors named
...ple.ple_embedding.ngram_embedding.shard_0.weight ([2500012,160] fp8)
- the identical name the plugin asks the merged table under - so
ple.merged.safetensors lost the emplace race and the plan read one raw
shard as the codes (declared [320001536,160] vs codes [2500012,160]).
Rename to 00-ple.merged.safetensors: '00-' wins the sort, the merged
bf16 tensor wins the name, table_bf16 declares and the rtn rule makes
the fp8 container form.
2026-10-06 00:21:40 -07:00
Frank 5e166543fa docs: record the two-slot no-spec frank-small shape (measured 1,397 MiB free, 48.0 t/s)
Runbook reshaped: 2 x 131072 no-spec identity, slot-2 arithmetic (drafter ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot), ngram-mod skipped on the recurrent-rollback law, MTP references kept (44.8-52.1% acceptance, 58.9-66.9 t/s), rollback = one revert of 812d9eb. AGENTS fleet row + section, LLM/index.md intro + entry, root index entry all flipped to the 2026-10-06 shape. 360/360 suite green; e2e gate 33/33 green at deploy.
2026-10-06 00:15:55 -07:00
Frank 812d9ebff5 frank-small: two 128k slots, no spec, ubatch 512 (Frank's tight-fit order, 2026-10-06)
CTX 262144 / PARALLEL 2 = two 131,072 conversations; SPEC_TYPE=none strips both speculative flags in the serve script (ngram-mod skipped on purpose — on a qwen35 hybrid it disables recurrent-state rollback); ubatch 1024->512 claws back ~0.3 GiB. Budget: non-KV ~15.5 GiB + two q8_0 slots 8,704 MiB lands AT the card — the 2-3 GiB floor is knowingly violated by explicit order; nvidia-smi free at boot is the measured truth. Seam retargeted: shape (2 slots, 131072/slot), budget (two fit, three don't), spec-off (absence of both flags), unit Description.
2026-10-06 00:07:03 -07:00
Frank fb1a4fd150 frank-primary: never write the stage index through the raw symlink
Run 3 rewrote the staged index THROUGH the relative symlink into the raw
checkpoint: the raw index.json was restamped with the stacked layout
(605 KB smaller), the raw verify then failed 143/144 and the overnight
run died before stacking. Two fixes:

- stack-qwen38-official-fp8.py: index/config writes are now temp+os.replace
  (an atomic swap replaces the symlink itself; the raw tree stays pristine).
- build-frank-primary-rad.sh: the stage's index.json is COPIED like
  config.json; only shards stay relative symlinks.
2026-10-06 00:03:58 -07:00
Frank 90a08ab4b5 comfyui: two proven Qwen-Image 2.1 workflows (art + legible text) + payload docs and next-model recommendations 2026-10-05 23:59:29 -07:00
Frank 272acb4c7f frank-primary: two stacking fixes from the first live run
- release the numpy frombuffer view before closing the source mmap
  (Python 3.14 BufferError: cannot close exported pointers exist).
- stacked scale planes rename: checkpoint.cpp scale_name() appends
  _scale_inv to the STACKED tensor name verbatim (no .weight segment) -
  ...experts.gate_up_proj_scale_inv, not ...gate_up_proj.weight_scale_inv.
2026-10-05 23:56:36 -07:00
Frank 13112a93f5 frank-small: record measured cutover values; unit Description follows the Swift swap
Boot ledger lines, 19,404 used / 4,721 MiB free (the q8_0 proof), decode arms (66.1 t/s @0.8, 66.9 @0, 58.9 @recipe 1.0 with acceptance 52.1%/52.2%/44.8%), vision Red probe, reasoning_effort xhigh/high contract, e2e 12-crawl transient + full-gate green rerun. Unit Description updated in-repo; the relinked unit picks it up at the next deploy/restart (no bounce for cosmetics). 360/360 suite green.
2026-10-05 23:41:05 -07:00
Frank baf33d016f frank-small: Swift-1.5 IQ3_S-mtp swap (parallel session) + fleet docs index
The parallel session's uncommitted swap, committed at Frank's direction
(commit whatever's in the working tree):
- llama.cpp master on the RTX 3090 now serves ukisai's
  Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE
  131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through
  the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to
  rollback target.
- env/serve/download + seam twins updated together (deployer payload gates
  rode the previous commit's deploy-remote.sh).
- docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology
  and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135
  seam green).
- comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/
  workflows/ (12 KiB).
2026-10-05 23:26:59 -07:00
Frank 1919ad7f31 frank-primary: radiance native-FP8 TP=2 cutover (engine, payload build, rollback)
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the
pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via
libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8
payload at native block-FP8 precision:

- scripts/build-frank-primary-rad.sh: staging (quantization_config drop =
  plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE
  --quant rule (PLE rtn fixed-scale); plan-only first.
- scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes
  (experts byte-stacked into the plugin's stacked layout; PLE's 128
  scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own
  scale; the rtn rule re-encodes at that same scale - codes round-trip
  exactly and land the one-element scale plane the plugin declares).
- scripts/hfget.sh + verified download (131/131 shards sha256-match).
- froggeric v22.5 chat template vendored (--override-chat-template).
- serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host),
  TTM preflight fixed for kernels with pages_limit only (no pages_max -
  an absent knob is skipped, not read as zero); pages_limit raised to
  52,428,800 pages = 200 GiB on the host pre-cutover.
- root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB
  + health; prometheus re-adds the /metrics job.
- rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio +
  Dockerfile); payload + image stay on disk until the window closes.

deploy-remote.sh also carries the parallel session's frank-small payload
gates (swift15) - committed here per Frank's direction to commit the tree.
2026-10-05 23:26:51 -07:00
Frank 8617b79a18 docs: index frank-primary-freetoken-pr587; mark the r9v q8 runbook retired 2026-10-05 21:20:05 -07:00
Frank 33eae71d33 docs: distill the managed skill library into the vault (waves 1-7)
All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
2026-10-05 21:19:15 -07:00
Frank 8618ec6dbe prometheus: drop the frank-primary scrape job (FreeToken exports no /metrics)
FreeToken has no /metrics endpoint — the job scraped 404s forever, a
structural never-green, not an outage (the Lucebox DS4.1 precedent). The
LlmEndpointDown alert regex self-adjusts (frank-.* matches existing series
only). Comments updated; seam now asserts frank-primary's absence. If
FreeToken grows /metrics, re-add the job.
2026-10-05 19:38:14 -07:00
Frank e2308dff71 docs: frank-primary runs FreeToken (PR587 ROCm) — runbook, index, AGENTS, log
New runbook frank-primary-freetoken-pr587 (build laws, shape, measured
cutover numbers, TP=2 non-boot signature, rollback); LLM/index.md intro +
mermaid + active/retired entries; AGENTS.md fleet row + section + bench
and model-config mentions + runbook index; log.md entry. Seam pins the
measured KV pool 299968 with the multiple-of-64 guard.

NOTE: intended PGP signature skipped — key 0x76EA601A11B178D7 has no
secret key in this machine's keyring (fresh ~/.gnupg); the PGP identity
lives elsewhere.
2026-10-05 19:35:31 -07:00
Frank 6ba19f6f48 frank-primary: deploy health gate reads the body (FreeToken 200-loading)
FreeToken's /health returns 200 with {"status":"loading"} while the
engine loads — the FastAPI front end binds before the scheduler is up.
Measured 2026-10-06: the deploy gate went green after 6s while 131 shards
were still paging in (vLLM 503'd during load, which made the bare curl
honest there). The frank-primary gate now loops until status:ok; the
MainPID stray-port guard is unchanged.
2026-10-05 19:27:21 -07:00
Frank 0ff016a6b2 frank-primary: KV pool snapped to the qsa_sparse page-64 multiple (299968)
Measured first boot: the auto attention backend is qsa_sparse, which
overrides page size to 64 and refuses --num-tokens 300000 ('nearest valid
values: 299968 or 300032'). 299,968 x 24 KiB = 7.0 GiB still clears the
177,356-token harness worst case and one native 262,144 conversation.
2026-10-05 19:19:24 -07:00
Frank bd5c28c4e9 frank-primary: FreeToken (PR587 ROCm) cutover — trio, seam, deployer
Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201
built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201
(scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated
--no-deps install per docs/install_amd.md).

Shape (first-boot conservative): ONE R9700 (second card idle;
--tensor-parallel-size exists for future 2-card tuning), KV pool
--num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token
harness worst case and one native 262,144 conversation), MoE LRU
--moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens
36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml
flips reasoningContentField on the harness side). Experts + PLE pin
~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity.

Container identity renamed to the unit name (Frank's call): frank-primary;
vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image
gate becomes podman image exists (a local build carries no RepoDigests;
the old digest grep refused exactly this shape). Rollback tag:
frank-primary-vllm-shapeG-20261003.
2026-10-05 19:17:13 -07:00
Frank 4a02dc0980 frank-primary: digest-pin FreeToken base image, symlink libhiprtc dev name
- FROM pinned to sha256:a223aee1... (the pulled rocm/pytorch rocm7.14
  pytorch 2.11.0 manifest); a re-pushed tag cannot change the stack.
- The pip-ROCm torch ships libhiprtc.so.7 only; the PR's _cpu_moe links
  -lhiprtc and ld needs the unversioned symlink in the already-linked
  _rocm_sdk_core/lib (first build died: cannot find -lhiprtc).
2026-10-05 19:11:15 -07:00
Frank 1f6bdac759 frank-primary: add FreeToken PR587 ROCm serving image recipe
Dockerfile builds the FreeToken engine (FlashML-org/FreeToken PR #587,
pinned head 77a7201) inside the PR's own known-good ROCm 7.14 base image
(torch 2.11/triton 3.7, gfx1201) on dankserver, rootless podman. Follows
docs/install_amd.md exactly: --no-build-isolation --no-deps engine
install, curated portable runtime deps, flashlib --no-deps, ninja for the
QSA JIT; no flashinfer/sgl_kernel on ROCm (Triton fallback).
2026-10-05 19:00:30 -07:00
Frank a6ad9af708 docs: vault link integrity restored after rogue-session doc WIP landed
The parallel session's stash pop left GGUF Compatibility, Multi-Device
Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving,
vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9
unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets
that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link,
and two multi-line-wrapped link tokens). All repointed to real pages;
six rows added to the vault index; seam green.
2026-10-05 18:40:31 -07:00
Frank 380aafa1cc frank-small: Frank overrules q8_0 V — f16 K+V, ubatch 1024 first, ONE native 262144 slot (the only f16/f16 shape that fits Q8 weights on 24 GiB) 2026-10-05 18:31:50 -07:00
Frank 5573ca171f frank-small: seam follows the f16 K + q8_0 V contract from 1862dc3
1862dc3 repriced the Ornith rung (nine full-attn layers: f16/f16 = dead
boot) and flipped CACHE_TYPE_V to q8_0, but its note that the seam was
'retargeted in 8f53df7' covered only the payload paths — the KV test
still asserted the 2026-10-03 unquantized-both-halves contract and held
the suite red. Test now pins the shipped contract: K=f16, V=q8_0,
flash-attn on.
2026-10-05 18:23:44 -07:00
Frank 1c725ea024 e2e: run.sh keeps playwright arg passthrough with the default project set
The default-project-set rewrite dropped non---project args (e.g. --list),
so 'run.sh --list' silently RAN the whole suite instead of listing.
Args now ride alongside the default set; an explicit --project still
takes over. Verified: default --list = 33 tests listed; opt-in --list =
34 with the frankbook walk.
2026-10-05 18:18:21 -07:00
Frank 1862dc3c0f frank-small: env arithmetic for the Ornith Q8_0 rung (f16 K + q8_0 V)
The rogue session's swap left this env reprice uncommitted in the tree
(2026-10-05): Ornith is a 33-block qwen35 hybrid with NINE full-attn
layers (18 KiB/token/half), +1,900 MiB of weights vs MiMo Q6_K — f16/f16
lands ~379 MiB free, a dead boot; pool-stepping cannot rescue it because
the compaction invariant needs slot >= 191,148. The only lever that
keeps 2 x 200,704 AND the 2-3 GiB floor is q8_0 V (~2.87 GiB free). K
stays f16. Deployer gate + seam already retargeted in 8f53df7; fetch
script manifest already flipped in 3ee88d7. MEASURE after boot; under
1.5 GiB free, UBATCH_SIZE down to 1024, never the window.
2026-10-05 18:18:06 -07:00
Frank 2930431052 frank-small: repoint the deployer payload gate and unit seam at the Ornith-1.5-9B payload
The 17:47 Ornith-1.5-9B swap (77decb5) updated frank-small.env and
serve-frank-small.sh but left two consumers on the MiMo-Q6K paths:
deploy-remote.sh's LLM_SMALL_MODEL_FILE/LLM_SMALL_MMPROJ defaults (the
payload-existence gate checked files the unit no longer loads) and
frank_small_unit.bats (stubs + env assertion), which held the whole
bats suite red. Both now point at the Ornith-Q8 payload; docs updated
with the conditional frankbook e2e project.
2026-10-05 18:13:01 -07:00
Frank 8f53df7b4f frank-small: complete the Ornith swap in seam + deployer twins
The rogue session's Ornith commits (3ee88d7/77decb5) swapped the env and
fetch script but left the deploy handler gates and the seam asserting
the MiMo-Q6K paths — the seam failures 195-203 and a deploy that gates
a file the unit no longer reads. Flip both to Ornith-Q8 (Q8_0 GGUF +
BF16 mmproj), keeping the anti-drift twin structure intact.
2026-10-05 18:10:59 -07:00
Frank 9c508218e9 e2e: scope the FrankBook-Pro leg to its own trigger-gated Playwright project
The 11-admin spec walked links that terminate on FrankBook-Pro
(192.168.0.112), which sleeps when idle — every deploy gate timed out
there (measured 2026-10-05, two deploy windows). Split: the frankbook
links move to tests/e2e/specs/19-frankbook.spec.ts under a frankbook
project that run.sh excludes by default and deploy.sh triggers ONLY
when a frankbook blast-radius path changed (the spec, frankbook-pro/,
the homepage registry, the Caddyfile arr vhosts). deploy.sh computes
the trigger from the same BASE..HEAD revision range the plan uses.
2026-10-05 18:04:02 -07:00
Frank 77decb545e frank-small: swap the payload to Ornith-1.5-9B Q8_0 + BF16 mmproj 2026-10-05 17:55:36 -07:00
Frank 69f10a17bb frank-small: swap the payload to Ornith-1.5-9B Q8_0 2026-10-05 17:55:36 -07:00
Frank be840b4881 frank-large: halogen container cutover, rootless rlimit law, and the DFlash2 drafter shape 2026-10-05 17:35:05 -07:00
Frank b0b840b149 frank-primary: document FRANK_PRIMARY_MNBT=2048 and the chunked-prefill flag surface 2026-10-05 17:32:22 -07:00
Frank bdbe814fac frank-large: correct the halogen payload contract to the Hub's truth
The first constants were wrong in three ways, found while resuming the
checkpoint (the file was COMPLETE; the manifest was not):
- checkpoint is 35,865,565,184 B / LFS sha256 274c3fc7... (not
  35,880,329,102 / 40874e7a...), 1352 tensors per the card;
- the tokenizer is a FLAT tokenizer/ DIRECTORY of 5 files mounted whole
  at /tokenizer (model-card docker-run parity; the engine's
  HALOGEN_TOKENIZER default is the directory) — there is no
  frank-large.tokenizer.gguf;
- plain git blobs (vocab/merges/jinja/config) carry no LFS oid, so the
  fetch script size-checks them and sha-verifies only the two LFS
  objects (checkpoint, tokenizer.json).

Gates flipped together (serve + deployer twins): tokenizer dir must be a
real directory and every named file a regular file, no symlinks (HF
snapshot links dangle in the mount). Fetch made idempotent: a file at
the published size skips the transfer (resuming it is a CDN 416 under
set -e). Seam rewritten to the same contract, 17/17.
2026-10-05 16:45:12 -07:00
Frank 87695c5126 download-frank-large-halogen: resume against the resolved CDN URL
curl drops the Range header across the HF /resolve/ 302, so '-C -'
against the resolve URL pulls the 1 KB redirect body and exits 0 with
the file still 14 MiB short (measured 2026-10-05). Resolve the
effective URL first; the Xet CDN honors Range.
2026-10-05 16:31:35 -07:00
Frank 0900551317 docs: frank-large halogen runbook, AGENTS.md + indices for the swap
New runbook frank-large-halogen27b-dflash2.md; LLM + vault indices
re-pointed, ROCmFPX q6agent runbook marked RETIRED (rollback target);
AGENTS.md topology row, detail section, image-capability paragraph
(two endpoints accept images now), bench-law prose and runbook index
updated.
2026-10-05 16:29:11 -07:00
Frank fece3ffdd0 frank-large: cut over to halogen container (qwen3.8-27b .hgn + DFlash2)
Runtime swap ROCmFPX llama.cpp -> peonist-ai/halogen (image digest
dc0a39a0, tag 0.1.4), serving qwen3.8-27b-p1w4d-d2.hgn (33.4 GiB, the
.hgn format loads in no other engine) with HALOGEN_DRAFTER=2 (DFlash2)
on ONE slot of 200700 ctx: slots>1 disables the drafter (engine law).
Accepted: TEXT-ONLY (vision encoder ignored; frank-small owns images)
and the censored base (rollback = revert this commit; payload, fork
build and build-rocmfpx.sh pin stay on disk).

- serve script: podman run under the root unit, exec podman wait so
  Restart=always is real; gates image digest, checkpoint byte-count,
  flat regular-file tokenizer, GPU nodes; engine stdout to log.
- deploy handler: gates the artifacts, sweeps the legacy bare
  llama-server + old container name by NAME (never pkill llama-server:
  frank-small), retires the 2026-09 quadlet, port cutover 7770.
- fetch script: -C - resume, -f, byte count, sha256, --verify-only.
- seam rewritten (17 tests): halogen is now the deployed runtime.
2026-10-05 16:29:11 -07:00
Frank aa9908257d docs(index): frank-small summaries carried the superseded 2,849 MiB free figure
Both indexes stated the pre-doubling VRAM as current. Now 2,279 MiB free with
the 4096/2048 pair named, so no index contradicts the runbook it links to.
2026-10-05 16:03:18 -07:00
Frank 67141c8c02 docs(frank-small): record the batch doubling and scope the ubatch law to its backend
The AGENTS.md lesson "`--ubatch-size` has an OPTIMUM on this stack" was load-
bearing and over-general: it was measured on the 2x R9700 layer split, and the
same knob on the CUDA 3090 behaves the opposite way (curve climbs monotonically
to 2048). That difference is now the reason frank-small ships ubatch 2048, so
the lesson names the backend it was measured on and points at the counterpoint.

Adds the 2026-10-05 boot-pair A/B to the runbook (+4.6 % prefill at 8,928
tokens for 616 MiB, decode flat, the sub-ubatch prompt as the control), marks
the superseded 21,276/2,849 VRAM row, and logs the day. The old "ubatch 2048
failed the scratch boot" line stays visible as what it was: a predecessor-shape
observation, not a law.
2026-10-05 15:53:06 -07:00
Frank e5893dc30d frank-small: double the compute-buffer pair to 4096/2048
BATCH_SIZE 2048 -> 4096, UBATCH_SIZE 1024 -> 2048.

Why the ubatch move is right on THIS card: the prefill curve is
card-specific. On the 2x R9700 layer-split sibling it peaks at ubatch 512
and every step up costs GiB for no speed; on this CUDA 3090 it climbs
monotonically to 2048 (710 -> 1188 t/s on the 35B MoE sweep, 1944 -> 2423
t/s on the Ornith 8k sweep) with decode flat to under 1 t/s on every rung.
The logical batch is free either way: two sweeps agree that at a fixed
ubatch, 2048/4096/8192 give identical prefill speed and identical peak
VRAM, because the compute buffers are sized by the ubatch alone.

The risk is all in the ubatch, and it is documented rather than assumed:
the step cost ~1,244 MiB of peak graph VRAM on the 35B MoE, the shape this
replaces is 2,815 MiB free, and the Q5_K_M predecessor recorded "ubatch
2048 failed the scratch boot" with MORE free than that. Compute buffers
are allocated at graph capture during boot, so a failure is loud and fast;
the step-down rungs (4096/1536, then 4096/1024) are named in the env.

The seam stops hard-coding a rung: serve_argv now carries BATCH_SIZE and
UBATCH_SIZE the way it already carried CTX_SIZE et al., and the pair test
asserts the argv follows the shipped env plus the llama.cpp invariant
ubatch <= batch. Verified both ways: retuning the env to 8192/4096 keeps
the seam green, inverting the pair to 2048/4096 turns it red.
2026-10-05 15:10:15 -07:00
Frank b06b47c7b4 docs(fleet): radiance standalone engine probe - qwen4exp refuses fp8 input, quant happens at convert
rad-convert --plan-only with the deployed orcarouter FP8 payload:
'no architecture plugin claims qwen4exp at quantisation fp8; served at
(unquantised)'. Our quantization is reproducible from a bf16/abliterated
source by recipe (libr4d reads fp8 experts), not from our file. Image
pulled on dankserver, no GPU touched.
2026-10-05 14:11:47 -07:00
Frank 2e64f66b82 docs(frank-primary): RESOLVED - the ~40k corruption was marginal DDR5 at 4800, fixed at 4600
The prefix-caching OFF/ON experiment was confounded (both arms ran on
the marginal speed); the align-mode re-seed theory is unproven, not
disproven. Caching ships ON with zero gdn_hip faults under heavy
re-send traffic on stable RAM; knob stays commented. Corrects the
env's 'RAM RULED OUT' claim and adds the AGENTS.md lesson.
2026-10-05 13:54:06 -07:00
Frank 35a45b8a66 frank-primary: prefix caching back ON — the OFF experiment did not fix the corruption
Frank: still corrupted at ~40k with --no-enable-prefix-caching deployed
and mechanically verified (zero gdn_hip faults in the container serving
that window, coherent 40,771-token boundary probe). The recurrence is
FAULT-FREE, so the align-mode page re-seed is not the (only) mechanism
and the mitigation bought whole-history re-prefill per turn for nothing.

Revert: knob FRANK_PRIMARY_NO_PREFIX_CACHING commented out -> the serve
branch renders --enable-prefix-caching (the shipped contract). The knob,
the branch and the seam stay documented for a future fixed fork build.

Open hypotheses recorded in the runbook: re-fed poisoned transcript (the
same conversation cannot distinguish fixed from unfixed), or a second
fault-free long-context path — fp8 KV at depth is the next suspect. Next
diagnostic: fresh-conversation reproduction, then a KV-dtype A/B.
2026-10-05 13:13:26 -07:00
Frank 1fee99c8ce docs(frank-primary): record the verified 2026-10-05 prefix-cache-off deploy
Deploy b820d34, measured: --no-enable-prefix-caching in the engine argv,
health 200, warm boot 76.57 s (compile cache intact), expert plan
unchanged (17.77 GiB / 7,762 slots), GPU KV cache 424,678 tokens (up
from 401,408 at the same byte ask - pool now prices ~7,456 B/token with
the per-block mamba checkpoint pages gone; still clears the 400,700
floor), zero gdn_hip lines, and the 40,771-token boundary probe
(13 x 3136 = 40,768 crossed) decoded the 1-400 count coherently
(0.997 monotone, tail ...398 399 400). e2e 32/33 - the failure is the
admin spec's FrankBook-Pro leg (mini asleep), environmental.
2026-10-05 05:55:39 -07:00
Frank b820d341f5 frank-primary: fix the prefix-cache flag render (127 crash loop)
Two bash defects in the 9d8b185 prefix-cache wiring, both measured live
on the 2026-10-05 deploy attempt (unit crash-looped 190x, exit 127,
endpoint down for the whole 3600 s health window):

1. A # comment block inside the backslash-continued podman run
   terminates the continuation: the comment swallows the joined line
   and the flag line then runs as its own command
   ('--no-enable-prefix-caching: command not found'). The comment moves
   above the run block, with a no-comments-in-chain note.

2. The ${VAR:+A}${VAR:--B} double-expansion is ambiguous: when the
   variable is set, ${VAR:--B} parses as a SUBSTRING OFFSET, not the
   :- default — measured render was '--no-enable-prefix-caching1'.
   The flag is now branched explicitly into
   FRANK_PRIMARY_PREFIX_CACHE_FLAG (exactly one of the two flags
   reaches the engine; unset keeps the historical flag).

Seam: the render test sources the shipped env through the branch and
asserts the exact flag word the engine receives (render-assert, the
same pattern the spec knobs use). bash -n clean; all 22 frank-primary
seams pass.
2026-10-05 05:18:53 -07:00
Frank 9d8b18553d frank-primary: prefix caching OFF — the fork's QFN align-mode GDN state fault
Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state
faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with
non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers
each — then the process-lifetime report cap. The guard's own text in
this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix
caching … Mitigation: --no-enable-prefix-caching'.

Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which
checkpoints each sequence's GDN recurrent state into a mamba page per
3136-token block and re-seeds it from that page when a request resumes
from a cached prefix. The NaN state landed at the 13th boundary
(13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page:
every later turn of that conversation re-seeded the same non-finite
state for two heads, silently (fault-report cap already spent).

RAM ruled out the same day: zero MCE/EDAC records since the boot under
test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the
JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN
signature is a state-path fault, not scattered bit corruption.

Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching
(flag renders in both directions from the knob). Cost: every turn
re-prefills the whole history instead of at most 3135 tail tokens.

Seam updated: the prefix-cache pin now asserts the conditional render
and the shipped OFF value. Runbook carries the evidence + acceptance
criteria; all 368 tests pass.
2026-10-05 03:45:26 -07:00
Frank a1c4701d7e docs(frank-small): Q8_0 rung evaluated and closed — floor break plus -24% decode 2026-10-04 18:17:43 -07:00
Frank c4603b130c frank-primary: pin the expert host region to 120 GiB (--expert-offload-mem)
The host region was the ONE arena-plan term still autofitting per boot: with
no cap the planner binds it to MemAvailable AT BOOT minus PLE/process/api/
margin terms (boot 2026-10-05: 237.97 GiB -> 159.39 GiB usable; the same host
reads 75.4 GiB available while running). Demand is 105.48 GiB
(52.74/rank x 2 ranks, 23,040 swap-target experts); a boot where co-tenant
RAM squeezed the bound near that silently pins next-hottest experts VRAM-only
(pinned_extra warning, smaller swap region, no flag change). 120 GiB =
measured demand + 14% headroom on 246.1 GiB total. Semantics verified in the
shipped image: cap binds as the bound, exceedance only warns.

The GPU-side arena is unchanged (deterministic arithmetic over the same
constants): verify after boot that host reads '--expert-offload-mem' and
slots stay 7,762. Seam: red first (wiring + value), then green; full suite
368+1 green.
2026-10-04 18:16:38 -07:00
Frank 9a144f779e docs(frank-primary): record VLLM_CUDAGRAPH_MEM_ESTIMATE_MB, the graph-term outright pin
Verified in the shipped image (v1/worker/gpu/cudagraph_mem.py):
estimate_cudagraph_memory() checks VLLM_CUDAGRAPH_MEM_ESTIMATE_MB FIRST and
returns that MiB total outright, ignoring the BASE/PER_GRAPH pair — the only
way to state the CUDA-graph plan term directly instead of deriving it. We
keep shipping the pair (0/12) so a future spec-decode shape re-prices
automatically; documented the switch condition.

Also corrected two stale shape-G claims in the runbook that predated the
1.00 revert: the deployed shape block and the lever table both still said
util 1.0 / 'le=1 ceiling', while shape G ships 0.982.
2026-10-04 17:51:27 -07:00
Frank 50459e4e31 docs: frank-small Q6_K cutover runbook + measured post-boot numbers
- new active runbook frank-small-llama-mimo-q6k-2x200704.md: 21,276 used /
  2,849 true free (inside the 2-3 GiB floor), decode 94.44 t/s vs 104.84 on
  Q5_K_M - the rung costs a measured -10 % (weight-bandwidth-bound card),
  512px red-PNG vision verified, and the new probe-size law (>=256 px: a
  10x10 image undersupplies image tokens and the model answers 'video'),
  nvidia-smi free-field law (driver holds ~450 MiB the subtraction hides)
- frank-small-llama-mimo-q5km.md retired; it keeps the boot-pair table and
  becomes the Q5 rollback reference (3,571 free, 104.84 t/s measured)
- AGENTS.md fleet row, endpoint section, --parallel lesson, runbook index
  (active row + two retired rows); LLM + root index entries
- Q6_K GGUF-vs-GGUF diff proven: template and all 46 KV keys byte-identical
  to Q5_K_M - the vision probe scare was probe size, not payload
2026-10-03 11:20:33 -07:00
Frank ff8c0ee264 frank-small: unit Description names the Q6_K rung
The journal still advertised Q5_K_M after the payload swap; the unit file
is the repo-side truth, fix it and let the next deploy daemon-reload it.
2026-10-03 11:20:10 -07:00
Frank 6ec0b1a85d frank-small: Q5_K_M -> Q6_K rung at the held 2x200704 f16-KV shape
The freed third slot funded exact V; the remaining ~3.5 GiB of honest
nvidia-smi free (3,571, not the 4,046 total-minus-used I quoted - the
driver holds ~450 MiB of reserve) funds the +875 MiB of Q6_K weights.
Projected landing ~2.7 GiB free, inside the 2-3 GiB floor; measure on
boot, step the pool one 12,288-token increment if it lands under 2 GiB.

- payload dir MiMo-Q6K/ made self-contained (mmproj hardlinked from
  MiMo-Q5KM/, link-count-2 verified)
- env/serve/deployer/bats repointed; download script now fetches and
  verifies Q6_K (7,793,710,624 B, ef96d05a...) with Q5_K_M documented as
  the rollback; payload sha verified live on the host before deploy
- seam green 14/14 against the stub engine
2026-10-03 11:00:19 -07:00
Frank b80886e318 docs: frank-large Q6 uncensored cutover + frank-small 2x200704 f16-KV reslice
- new active runbook frank-large-rocmfpx-q6agent-uncensored-mtp.md: the
  Q6_0_ROCMFPX_AGENT file (ftype 114, 339 type-102 + 159 type-103 tensors),
  27.6-28.1 t/s at 86.8% draft acceptance, the measured ngram-mod chain
  rejection (-15% prose: recurrent-state rollback disabled), the
  bench-window hardening laws
- frank-large-rocmfpx-q8agent-mtp.md retired; rollback section corrected
  (gufo payloads removed from disk 2026-10-03 - that rollback is now a
  re-download; the aligned Q8 file is the direct one, still on disk)
- frank-small-llama-mimo-q5km.md reshaped to 2 x 200704 with fully
  unquantized f16 KV: boot-pair table gains the measured row (20,530 used /
  4,046 free), 104.84 tok/s, slot-window contract at 200704, rollback to
  the 3x180224 + q8_0-V shape documented
- AGENTS.md: fleet table rows, both endpoint sections, the chain law
  extended with the recurrent-model measurement, the --parallel law with
  the reslice, runbook index (active + retired)
- LLM/index.md + root index entries updated
2026-10-03 08:01:44 -07:00
Frank 47a2c1287b frank-primary: shape G measured — arena 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x
Record the live numbers and the UTIL 1.0 verdict (unbootable: worker init
demands free > budget; ValueError on both ranks before a single weight loads).
Driver used 30.97 of 31.86 GiB; the 0.89 GiB/card margin held through a live
4,096-token generation at ~38-39 tok/s.
2026-10-03 03:21:55 -07:00
Frank c67a2aeb80 frank-primary: shape G v2 — UTIL back to the 0.982 law; the 1.00 force was unbootable
Worker init refuses budget >= device total on BOTH ranks:
  ValueError: Free memory on device cuda:0 (31.29/31.86 GiB ...)
NRestarts=2 crash loop, zero weights loaded. Experts stay at the v1 shape
(graph term 0.07 GiB, KV 3.1665e9 = the 400,700-token ask); only the budget
returns under the law. Arena 17.78 GiB / ~7,912 slots predicted.
2026-10-03 03:01:55 -07:00
Frank 0a4ff30db2 frank-primary: shape G — force the arena to the flag ceiling, KV to 400,700 tokens
Two plan terms were still unspent: the CUDA-graph charge (shape F's own boot
reported a NEGATIVE capture delta, -0.06 GiB, so the 256 MiB base pool was
fiction -> charge 0 + 12 MiB/graph = 0.07 GiB) and the budget (util 0.982 -> 1.0,
the field's le=1 maximum). Together +0.82 GiB of expert cache: plan target
18.35 GiB / ~8,016 slots. KV_BYTES 3.15e9 -> 3.1665e9 = the 400,700-token ask at
the engine's measured 7,891.8 B/token. Cost of the force, recorded: driver
high-water ~30.4 GiB of 31.86, so ~1.5 GiB of real margin per card; rollback is
UTIL=0.982.
2026-10-03 02:49:38 -07:00
Frank c682c6329c frank-primary: shape F — MNBT 2048, util 0.982, CUDA-graph plan term pinned to the measured capture
The arena is budget - non-expert - KV reserve - graphs - overhead. With util
capped at 0.982 (never 1.00) the graph estimate is the only plan term this repo
can shrink: the fork charges an analytic 256 MiB + 52 MiB/graph = 0.55 GiB where
the engine's own post-capture line reports 0.31 GiB, so pricing it at
256 + 12/graph returns ~0.24 GiB to the expert arena. MNBT 1024 -> 2048 is
arena-free: the registry overhead is base + 0.22 GiB per 1024 ABOVE its nbt 4096
anchor, and the fitted 5.25 GiB was fitted AT 4096.
2026-10-03 02:22:49 -07:00
Frank 6584c059d6 frank-small: two wide unquantized contexts (2 x 200,704) on the 3090
Frank's order, two messages: "extend the max context of frank-small to
200700 per individual stream without further quantization", then
"reduce it to 2 streams for a total of 401400-ish tokens, set ubatch to
1k", then "unquantize the KV if we have extra space".

CTX_SIZE 540672 -> 401408 with PARALLEL 3 -> 2 gives n_ctx_slot 200,704:
above the 200,700 the harness compacts at, so a full-length request
lands in one stream instead of a 400 exceed_context_size_error. The pool
must divide evenly across slots and stay 256-aligned for the FA path,
which is why it is 401,408 and not 401,400.

CACHE_TYPE_V q8_0 -> f16: the freed third slot is exactly what pays for
unquantized values (24.5 -> 32 KiB/token), and the budget lands back in
the 2-3 GiB band Frank set on 2026-10-02 instead of leaving ~6 GiB of
the card idle. The asymmetric-KV contract that created this endpoint is
spent with it; both --cache-type flags stay explicit.

--batch-size/--ubatch-size are now passed (2048/1024). The endpoint had
never passed either, so it ran on engine defaults nobody wrote down.

Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md
2026-10-03 02:04:54 -07:00
Frank 7d3dd33de0 frank-large: swap to the uncensored Q6_0_ROCMFPX_AGENT payload
Frank's order: replace frank-large with
kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF's
Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf, following that card.

The file is ftype 114 — 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159
GGML_TYPE_Q8_0_ROCMFPX (103) + 353 F32 over 851 tensors,
qwen35/64 layers/full_attention_interval 4/4 KV heads/262,144 window —
read out of its own header. The pinned ROCmFPX build already knows both
types (ggml.h 102/103 plus convert/cpy/getrows/mmq/fattn device paths
for 102), so no rebuild: the fork stays at 36e9acd and the payload
directory is the only new host state.

Card compliance, all already our flags: Q4_0 draft head (not Q8_0),
--spec-draft-n-max 4 (the knee on both cards), --spec-draft-ngl 99,
--jinja, --fit off, -fa on, -dio, one slot at full window, thinking with
the card's own sampling triple (which the file also embeds as its
general.sampling defaults). Kept from the deployed shape: the BF16
projector (this repo ships only a Q8_0 one; BF16 is the file verified
4/4 on spatial ground truth here) and the byte-identical MTP head, which
the new script hardlinks instead of re-fetching 1.6 GiB.

Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md
2026-10-03 02:04:22 -07:00
Frank 0ae923b879 autoresearch arms: add D/E context arms (KV 4.05e9 = 2.00x under MTP3; E adds MNBT 4096 + util 0.99) 2026-10-02 23:52:04 -07:00
Frank b7c3bdb2be autoresearch: temp sweep, engine evidence, env overlays, arms driver
Four harness gaps that block an honest MTP re-measurement:

- measure: AR_TEMPS sweeps sampling temperatures (default "0,0.7"). Draft
  ACCEPTANCE is sampling-dependent — a greedy draft matches a greedy target
  exactly and is rejected more under a non-greedy target — so judging a spec
  arm only at temperature 0 judges half the contract. The unsuffixed keys stay
  temperature 0, so every historical number remains comparable.
- window: capture an evidence.log sidecar (expert-plan lines, compile path,
  graph sizes, /metrics spec-decode and cache counters, OOM signatures). The
  arena/overhead/reserve numbers exist ONLY in the boot log; an arm that loses
  decode speed is otherwise unexplainable afterwards.
- orchestrator: FRANK_PRIMARY_AR_ENV_FILE stages an env other than the repo's,
  which is how arms A/B without mutating the checkout (the fingerprint covers
  whatever is staged, so the memo can never answer one arm with another's
  numbers), plus METRIC lines for the swept temperatures and the evidence dump.
- scripts/autoresearch/frank-primary-arms.sh: runs a sequence of full windows
  (baseline, MTP3+probabilistic with a step-4 capture ladder at precise memory
  ON, the same at the default overhead, and the vendor SEQS-16 shape), gating
  each staged env on its JSON knobs still parsing AFTER sourcing — an unquoted
  {"temperature":0.7} loses its quotes at source time and reaches the engine as
  {temperature:0.7}, which is a 20-minute boot wasted.
2026-10-02 16:38:17 -07:00
Frank d57174c8d3 frank-primary: env-driven spec/compilation/sampling knobs
Three engine knobs that were impossible to set without editing the run line:
FRANK_PRIMARY_SPEC (--speculative-config), FRANK_PRIMARY_COMPCONFIG
(--compilation-config) and FRANK_PRIMARY_GEN_CONFIG
(--override-generation-config). Each holds the whole JSON value; empty or
unset means the flag is absent, so the shipped shape E argv is unchanged.

The seam re-pins the law that justified the omission: "speculative decode is
ABSENT BY MEASUREMENT" compared the wrong pair — c1248d4 retired MTP by
weighing fp8+MTP4 (37.4-40.6 t/s, 54% acceptance) against the bf16 no-spec
boot (36.4-39.4), bf16 then proved unbootable, and fp8 came back WITHOUT MTP.
fp8+spec was never measured against fp8+no-spec on one shape, the no-spec
incumbent runs ~30 t/s under the fleet's 35 t/s target, and every vendor
flash-next recipe ships MTP3.

Also in the seam: a dead assertion is deleted (a failing `! grep` mid-test is
exempt from bats' errexit, so the pre-precise-memory `! grep
--expert-precise-memory` never ran), and the knob test asserts by RENDERING the
flag from the sourced env instead of grepping for an assignment.
2026-10-02 16:37:52 -07:00
Frank ea1132aa0a docs: record the unmeasured CPU-side recipe-diff candidates (HSA knobs, OMP_NUM_THREADS)
All six HSA names verified real by string presence in libhsa-runtime64.so, so
they are knobs and not no-ops — but none is shipped without its own A/B.
2026-10-02 15:15:13 -07:00
Frank 57f33e5289 frank-primary: record the measured shape-E and compile-cache numbers
Live after the 1ac4d48 deploy: 'expert_precise_memory': True, runtime overhead
5.25 GiB, expert cache space 17.22 GiB / 7522 slots, swap region 13.7 GiB /
5986 slots, GPU KV cache 398,658 tokens / 1.99x, pool 3,150,000,000, MNBT 1024
-- the probe's projection is now production.

Compile cache proven on consecutive boots of that shape: cold 'Compiling a
graph for compile range (1, 1024) takes 179.24 s' / 'torch.compile took
188.26 s'; warm 'Directly load the compiled graph(s) ... from the cache, took
2.143 s' / 'torch.compile took 3.97 s in total' / 'init engine ... took 41.13 s
(compilation: 3.98 s)'. Host dir 1.4 GB / 4,707 files. Time to /health stays
payload-bound (490 s warm), so the claim is ~184 s of inductor/AOT build per
restart, not a proportional boot cut.

Tests: 367/367.
2026-10-02 15:05:34 -07:00
Frank 1ac4d48371 frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache
Measured by boot probe on the shape-D base (production stopped, candidate
booted AS the service), plan numbers read off the engine's own log:

  runtime overhead   7.60 -> 5.25 GiB (precise gfx1201 TP2 registry)
  expert cache      14.87 -> 17.22 GiB
  slots              6,496 -> 7,522 (+1,026, +16%)
  swap region                13.7 GiB / 5,986 slots
  boot                       healthy in 420 s, no OOM

'--expert-precise-memory': True also appears in non-default args, so a silent
argparse drop is visible. The pool is NOT cut (option A) because the pool is
not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency,
and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded
as the conservative fallback.

Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no
/root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile
artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral
layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays
'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting
FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing.

GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B
measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes
needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB.

Tests: 367/367.
2026-10-02 14:37:57 -07:00
Frank 812193aa63 docs: the bare-deploy drain law + the plex gate fix in the work log
`deploy.sh <target>` with no `--service` also walks the per-service revision
markers and restarts everything behind HEAD whose paths changed — a drain, not
a sync. That is how the latent plex gate bug surfaced: the backlog fired all
at once and plex's http:// readiness probe, which can never succeed against a
PMS that drops plain HTTP off-loopback, burned its full 600 s and aborted the
run with every later service left pending.

Both facts recorded: AGENTS.md gains the drain law under Deployment, log.md
gains the plex entry with the evidence chain.
2026-10-02 12:26:55 -07:00
Frank 39f0e0d8d4 plex: probe the readiness gate over HTTPS — PMS drops plain HTTP off-loopback
deploy_plex gated on `curl http://127.0.0.1:32400/identity`, which can never
succeed: Plex accepts the TCP connection (it arrives on the container's eth0)
and then answers with nothing, so curl reports "Empty reply from server" (52).
Measured 2026-10-02 on a plex that was healthy the whole time —
`https://127.0.0.1:32400/identity` returned 200 in 44 ms while the gate burned
its full 600 s and failed the deploy.

The repo already knew this law in another place: tests/caddyfile.bats pins the
plex vhosts to a TLS upstream with the comment "PMS drops plain HTTP
off-loopback". The deployer's gate never got the same treatment, and plex's
per-service revision had been behind long enough that the gate had not run
against this posture until now.

-k because PMS serves its own self-signed cert on 32400.
2026-10-02 12:23:56 -07:00
Frank 656012a0fa docs: LLM index + work log for frank-primary shape D, harness window
index.md intro and mermaid nodes still described 4 x 262144 fp8/bf16 and
frank-small at 2 x 262144; the active-deployment list had no frank-primary
entry at all. Fixed both, plus a malformed GLM list item.

log.md gains the shape-D entry (the SEQS=2 constraint, the arena-lever
measurement, the 2x2 probe table, the measured 1.99x), and the frank-small
entry's stale consequence bullet now points at the resolution.

frank-small runbook: the compaction-fit note is corrected - the 200704 window
moved frank-primary's trigger to 140,492, which fits a 180,224 slot again.
2026-10-02 12:05:39 -07:00
Frank 295d5784d3 frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots)
Comments/docs only - no functional change. The live boot of c769ff2 reported:
  GPU KV cache size: 398,658 tokens,
  Maximum concurrency for 200,704 tokens per request: 1.99x
  expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB

The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x)
did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at
MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 ->
398,658). Never extrapolate a concurrency figure across geometries.

Also smoke-verified: three concurrent 67-token requests answered in parallel,
~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
2026-10-02 11:48:09 -07:00
Frank c769ff2f63 frank-primary: 3 slots x 200704, fp8, pool sized for 2 contexts
Re-applies the desired shape incrementally on the proven-good fp8 3x262144
base, after bisecting why the first attempt (30904ba) crash-looped.

Root cause: `--max-num-seqs 2` cannot compile on this model. AOT
build_guards(strict_error=True) inside profile_run -> _dummy_run specializes
`query_start_loc`'s width to SEQS + 1 (3 at SEQS 2, 4 at SEQS 3) and the
ConstraintViolationError kills engine init ~7 min in. MNBT is orthogonal:
SEQS 2 failed at MNBT 1024 AND 2048; SEQS 3 passed at both. KV dtype was a
red herring. Measured by four single-variable boot probes.

Arena lever measured: the expert arena is sized from `max_model_len` via the
registry-priced KV reserve (9190 B/token x max_model_len), NOT from
--kv-cache-memory-bytes. 262144 -> 14.30 GiB / 6,248 slots; 200704 ->
14.87 GiB / 6,496 slots. The 6.1e9 -> 3.15e9 pool cut bought zero arena and
is a runtime-headroom lever.

Shape: MAXLEN 200704, SEQS 3 (unchanged - the only slot count that compiles),
KV_BYTES 3.15e9/rank = 2.04x full-length concurrency at 7,701 B/token fp8,
MNBT 1024, gather 32x64, no spec decode, no prefetch, no precise-memory.
2026-10-02 11:33:27 -07:00
Frank e460bbbab5 frank-primary: revert to known-good fp8 3x262144
The 2 x 200704 fp8 / MNBT 1024 shape (30904ba) crash-loops at boot: the
engine dies in determine_available_memory -> profile_run -> _dummy_run with
the SAME torch.compile dynamo constraint that killed the bf16 2 x 262144
shape on 2026-10-01:

  RuntimeError: Worker failed with error 'Constraints violated
  (L['query_start_loc'].size()[0])! You marked it as dynamic but your code
  specialized it to be a constant (3)'

Not an OOM. Restoring the last shape measured serving (fp8 3 x 262144 /
KV 6.1e9 / MNBT 2048 / arena 6,248) so the endpoint serves again; the
desired edits are re-applied one at a time from this base.
2026-10-02 10:46:01 -07:00
Comrade Chief Designer 30904bac1b frank-primary: 2 x 200704 fp8, pool cut for the expert arena, MNBT 1024
The 2026-10-01 bf16 shape (2 x 262144, --expert-precise-memory)
crash-looped at graph capture — the arena is DEVICE-resident and that
shape had 0.13 GiB to spare. The plain bf16 2 x 262144 also did not stay
up: 0 bytes free at runtime, and a real client request (995-token prompt,
max_tokens=261149) killed the worker with .

The correction is a pool cut, not a downgrade. FP8 KV (7,701 B/token),
two 200704 windows (401,408 tokens -> 3.091e9 B floor, pinned 3.15e9),
MNBT 1024. The fork's expert planner sizes the arena AFTER the KV
override, so the ~2.95 GiB/rank reclaimed becomes arena: ~7,500 slots
vs 6,248. Two contexts at 200704 is a capacity contract, not a
throughput one (2-way concurrency summed BELOW parity at 4064 ctx), so
the slot buys the VRAM headroom long generations need.

Refs [[dankserver-qwen38-fp8-expert-offload-tuning]].

- scripts/frank-primary.env: MAXLEN 200704, SEQS 2, KV_BYTES 3.15e9,
  MNBT 1024, rewritten shape comment
- scripts/serve-frank-primary.sh: header comment updated
- tests/frank_primary_unit.bats: assertions updated, new seam pins the
  arena-floor pool law (the pool must move with a context change or the
  reduction buys nothing)
- docs + AGENTS.md: fleet topology and deployed shape corrected
2026-10-02 09:54:08 -07:00
Frank f8f845d86b autoresearch: harness setup
Benchmark entrypoint: bash autoresearch.sh
Goal: Make frank-primary faster: raise decode tok/s on the tcclaviger vLLM endpoint (dankserver :7770, Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700, automatic tiered-expert offload). Context size may be altered (MAXLEN/SEQS/KV_BYTES and friends); quantization may NOT be altered (the 131-shard FP8 payload and the digest-pinned image are frozen).
2026-10-02 07:50:50 -07:00
Frank 934285c2b2 docs(frank-small): record the 3 x 180,224 + vision shape and what it cost
Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new
Key Lessons, one stale SGLang label in the bench list) and the bundle log now
describe the deployed shape: pool 540,672, 180,224 tokens per conversation,
f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8
tok/s decode, 3,814 t/s prefill.

The claims that mattered:
- three contexts buy 180,224 tokens per conversation, not the native 262,144,
  and a longer prompt is a hard 400 - so models.yml contextWindow is 180224
  and frank-primary's 183,501-token compaction request no longer fits here
  (forward pointer added to the retired SGLang runbook, which asserted it)
- the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting
  two measured boots, not by a formula
- /v1/chat/completions ignores the legacy top-level 'images' field: the model
  invents a color; only the image_url content part reaches it
2026-10-02 03:46:50 -07:00
Frank 7fe881de01 frank-small: pool 540672 (3 x 180224) so the card keeps ~2.7 GiB free
Measured after the first 3-slot deploy: 3 x 196,608 booted (n_slots=3,
n_ctx_slot=196608, multimodal loaded) but used 22,972/24,576 MiB - 1,153 MiB
free, outside the 2-3 GiB band. The two measurements now recorded in
frank-small.env price the vision tower at ~1.4 GiB (non-KV 7,390 -> 8,859
MiB), so the pool drops 49,152 tokens (16,384/slot, 1,176 MiB) to land ~2.7
GiB free. The bats budget gate is recalibrated to the measured 9 GiB non-KV
cost, which now rejects the 3 x 196,608 shape instead of blessing it.
2026-10-02 03:17:35 -07:00
Frank a02850203d frank-small: three contexts, vision on, 2-3 GiB of the 3090 left free
CTX_SIZE 589824 / PARALLEL 3 -> three independent 196,608-token conversations
(a mamba hybrid cannot share one KV cache, so per-slot window = pool/slots;
3 x 262,144 would need 18.4 GiB of KV alone and busts the card).

Vision comes back via bartowski's mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
(918,166,048 B, sha256 ff348f31..., fetched + verified by
scripts/download-frank-small.sh): the Q5_K_M GGUF ships no v.* tensors, so
--mmproj IS the image path. MMPROJ_FILE without a colon default so an explicit
empty value is a real text-only opt-out, and the deployer now gates on the
projector exactly as the serve script does.

Budget is measured, not guessed: the 2 x 262,144 shape used 20,190/24,576 MiB,
which prices non-KV cost at 7,390 MiB. Three slots + projector land near
22.5 GiB, i.e. ~2 GiB spare - the bottom of the requested band, to be
confirmed by measurement and laddered if it lands outside 2-3 GiB.
2026-10-02 03:08:00 -07:00
Frank 8f129180d8 docs+scripts: reconcile payload inventory after both swaps
Staged 59 GiB of retired LLM payloads on Dankclaw to
~dank/.AGENTTRASH/20261002-retired-llm-payloads/ (vLLM MiMo BF16, Qwen3.5-9B
MTP source, both DFlash dirs, Gemma4-12B-QAT, SGLang MiMo INT8) and corrected
every doc claim about what is on disk: gufo's rollback pair (Q4_K_XL weights +
the Q8_K_XL DFlash2 drafter) and frank-small's Q6_K stay in place, so the
rollback paths in the tags' env files still resolve. The vLLM-era frank-large
fetch trio moves to old/ with its runtime; runbook links updated.
2026-10-02 02:31:39 -07:00
Frank 72bd4c967e docs: record the 2026-10-02 swaps — frank-large on ROCmFPX, frank-small on Q5_K_M
frank-large: new runbook frank-large-rocmfpx-q8agent-mtp.md — the ftype-115
tensor-type lock-in that forced the fork, the pinned build (set_rows history),
the measured MTP A/B (17.0 vs 7.6 t/s decode, acceptance 0.462), the live
client contract (reasoning_content, effort fold, vision restored) and rollback.
frank-small: new runbook frank-small-llama-mimo-q5km.md; q6k marked retired
(rollback target). gufo runbook marked retired (rollback target). Both indexes,
the fleet topology table, the vision/models.yml notes, the benchmarking law and
three new Key Lessons updated.
2026-10-02 01:51:11 -07:00
Frank 125d04967f frank-large: ROCmFPX llama.cpp on the Q8_0_ROCMFPX_AGENT file, MTP drafting
The repo's highest-quality rung is ftype 115: 311 tensors of
GGML_TYPE_Q8_0_ROCMFPX, a type only the ROCmFPX fork reads, so the runtime
moves from gufo to scripts/build-rocmfpx.sh's pinned fork build (36e9acd) —
the first build on this host measured to load this file without the
2026-08-25 set_rows segfault. One session at the native 262,144 window, f16
KV (64 KiB/token, 16 GiB), the repo's Q4_0 MTP head drafting 4 tokens, plus
the bundled BF16 projector so the endpoint has vision again.

The serve script now honors caller environment overrides over the env file
(a plain source let the file win, which made single-run overrides no-ops) and
defaults apply only when a value is unset, so an explicit LLAMA_BIN='' is the
error it reads as.
2026-10-02 00:47:16 -07:00
Frank d82c5c03b7 frank-small: Q5_K_M GGUF on llama.cpp, two full native windows
One quant step down from Q6_K on the same CUDA build and the same shape:
2 x 262,144 with K f16 + V q8_0. The 0.86 GiB freed from the weights is
headroom, not a new window — the native 262,144 cap is already reached,
and f16 V would still cost 3.96 GiB more than the pool can carry.
2026-10-02 00:00:17 -07:00
Frank 4bb10cd20e fetch: kingjones Q8_0_ROCMFPX_AGENT payload, frank-small Q5_K_M GGUF
The SGLang INT8 fetch recipe moves to old/ with its route; scripts/
download-frank-small.sh now carries the GGUF quant the endpoint serves.
2026-10-01 23:50:52 -07:00
Frank a7c68db9d0 docs: record the frank-small llama.cpp Q6_K cutover and the frank-large gufo rollback 2026-10-01 23:50:52 -07:00
Frank 8551bb006b frank-large: roll back to gufo Qwen3.8-27B Q4_K_XL, one session x 262144 f16 KV
frank-small: llama.cpp CUDA (master, sm_86) on MiMo Q6_K GGUF, 2 x 262144,
asymmetric KV (f16 K + q8_0 V, flash-attn pinned for the V quant)
2026-10-01 22:40:59 -07:00
Frank e452a1afbf harness: tolerate missing container in ASI capture; llama cmdline fallback 2026-10-01 22:11:34 -07:00
Frank 01ba41cab4 harness: llama-server lifecycle in pre-clean and health guard 2026-10-01 21:58:51 -07:00
Frank cde91c1a00 llama.cpp runtime arm: serve-script runtime switch, harness containerless support, master build 2026-10-01 21:56:30 -07:00
Frank 6513672f31 SPEC= (no draft): decode +46%, prefill +52%, ttft -35% vs MTP-2 baseline. Spec decode is NET-NEGATIVE on this endpoint for real chat+reasoning traffic at both depths; the bench-window win did not transfer.
Result: {"status":"keep","decode_tokens_per_sec":9.01,"prefill_tokens_per_sec":21088.7,"ttft_ms":851.8}
2026-10-01 21:00:19 -07:00
Frank 5df6ef3166 autoresearch: harness setup
Benchmark entrypoint: bash autoresearch.sh
Goal: Maximize frank-large decode throughput on the Strix Halo iGPU (gfx1151) via quantization >=q6/fp6, engine flags, spec-decode shape, or runtime swap — context limits unchanged, KV stays bf16.
2026-10-01 20:41:51 -07:00
Frank 3c339eeba2 frank-large: record deployed boot facts (pool 819,332 tokens, 3.13x, 230 s)
Live battery green after deploy #3: alias, reasoning split into
`reasoning`, tool_calls, /metrics acceptance pos0 69% pos1 44% - the
cross-model MTP head performs as measured in the bench window.
2026-10-01 19:48:56 -07:00
Frank 780c307354 frank-large: hand vLLM the CONTAINER model path; link the new runbook
The first crash-looping boot (51 restarts): serve received the host path
/home/dank/models/... but the models tree is mounted at /models inside the
container, so transformers failed 'Repo id must be in the form
namespace/repo_name' before the engine even built. Same conversion the spec
draft already used. Pinned by bats. Root docs index links the new runbook
(seam required it) and the work log records the crash-loop plus the 52 G
payload reclaim.
2026-10-01 19:39:55 -07:00
Frank de383fe53e frank-large: deployer spec gate must accept the sharded MTP layout
The 2026-10-01 first deploy refused /home/dank/models/Qwen3.5-9B because
the handler checked for a single model.safetensors (the dflash layout);
the Qwen3.5-9B MTP head ships 4 shards behind model.safetensors.index.json.
Same dual-layout acceptance as the serve script, now mirrored and gated by
the seam. The gate fired before the stop, so the live gufo unit was never
touched (health 200 throughout).
2026-10-01 18:52:17 -07:00
Frank 1127a52078 docs: frank-large vLLM MiMo-BF16 runbook; retire the gufo runbook
New runbook carries the roof math, the measured three-arm spec table, the
greedy-text numerics trap, the reasoning-field contract change and the
rollback tag. LLM index and AGENTS.md fleet table, bench policy and runbook
index follow; the frank-small index entries catch up to the SGLang cutover
(entry was missed on 2026-10-01). Work log records the cutover.
2026-10-01 18:48:51 -07:00
Frank dab483c79f frank-large: cutover to vLLM TheRock gfx1151, MiMo-V2.6 BF16 + cross-model MTP
Engine: digest-pinned docker.io/kyuz0/vllm-therock-gfx1151 (vLLM 0.30,
TheRock ROCm 10, Triton GDN kernels); the serve script gates the image by
REPO DIGEST, the payload (config + index + exact shard count), the spec
draft (BOTH weight layouts: single model.safetensors for dflash, sharded
index for the Qwen3.5-9B MTP head), and whole /dev/dri, sweeps the bench
identity vllm-mimo, then exec podman wait so a container death trips
Restart=always. Shape: 4 x 262144 at util 0.45 (KV grows only on the 8
full-attention layers), MTP 2 draft tokens.

Measured 2026-10-01 window (frank-large stopped): base 13.1 t/s decode
(~94% of the LPDDR5x BF16 roof), DFlash-8 17.4, MTP-2 19.8 with 71%/44%
per-position acceptance - the cross-model head works because vLLM reads the
draft from a separate path while MiMo ships zero mtp.* tensors. Spec arms
do not preserve greedy bf16 text (batched-verify reduction order).

The seam was rewritten around the new laws (18 tests): unit self-healing,
digest gating, engine wiring, the measured spec table, the retired gufo
gates, the deployer MainPID check. frank_small port test follows the
PORT -> FRANK_LARGE_PUBLISH rename. Rollback: tag frank-large-gufo-q4kxl-20261001.
2026-10-01 18:48:32 -07:00
Frank b3313340fa frank-large: fetch scripts for MiMo BF16, Qwen3.5-9B MTP source, DFlash drafter
All three verify every file by byte count + Hub LFS sha256, use -f (a bare
curl -o reports success on a 403 error body), support --verify-only, and
never put a token on the command line. Payloads are on Dankclaw and verified
against these manifests.
2026-10-01 18:48:05 -07:00
Frank e703b0c15c frank-primary: restore fp8 3x262144 (bf16 boot fails on a dynamo constraint)
bf16 at 2 x 262144 fails with and without --expert-precise-memory. The real
error (captured from the journal this time, container fixed via journald
rather than podman logs):

  RuntimeError: Worker failed with error 'Constraints violated
  (L['query_start_loc'].size()[0])! You marked it as dynamic but your code
  specialized it to be a constant (3)'

- a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a
  stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each
  podman run gets a fresh container filesystem.

The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 /
792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS
(3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window,
with --enforce-eager as the bypass.
2026-10-01 14:40:17 -07:00
Frank d3d180fd3f frank-primary: drop --expert-precise-memory (boot crash loop with bf16)
NRestarts climbed to 3, each attempt ~6.7 min then 'Deactivated successfully'
(API server drains after an EngineCore fatal error; Restart=always turned the
silent death into a visible loop). The arena is device-resident, so the budget
decides it: weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena
16.65 (precise overhead 5.25) + non-torch 1.45 = 31.73 of 31.86 GiB -> ~0.13
GiB free at boot, and graph capture is what allocates transiently.

Dropping the flag restores the 7.60 GiB default overhead: arena 14.31 GiB,
total ~29.4 GiB, ~2.4 GiB free. bf16 2 x 262144 (Frank's KV change) stays.
The serve script keeps the opt-in flag; the seam now asserts it is ABSENT.
2026-10-01 14:16:47 -07:00
Frank db54fa3af1 frank-small: record 2x262144 verification numbers
Live-verified post-deploy: context_len=262144, max_running_requests=2,
max_total_num_tokens=606208 in the boot log; a 200,000-token prompt
(compaction height is 183,501) prefilled+completed in one request
(97.3 s, ~2,056 tok/s eager); two concurrent 50k requests answered.
Docs-only change.
2026-10-01 14:08:51 -07:00
Frank 8029119b19 frank-large: Q4_K_XL payload + 4 x 163840 sessions (f16 KV)
Frank's call. Payload: ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB,
SWIFT OPEN LICENSE) -> unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB,
sha256 3f227079...b01e, vanilla/Apache-2.0). NOTE: the endpoint is no longer
uncensored — that was one of the reasons for the Swift payload on 2026-09-28.

New fetcher scripts/download-frank-large-q4.sh (no auth header: repo is public
and ungated; size + sha256 verified after transfer). The speculative drafter
and its fetcher are unchanged.

Shape: --sessions 4 --context 163840 with unquantized KV. gufo exposes no KV
quantization lever (execution_policy.hpp pins kv_cache_storage to kFp16), so
f16 is inherent.

KV arithmetic: f16 KV = 64 KiB/token (16 full-attention layers x 4 KV heads x
256 head_dim x 2 x 2B; the other 48 layers are linear-attention with a
per-sequence fp32 state). 4 x 163840 = 655,360 tokens = 40.0 GiB KV, each
session exactly 10 GiB. Budget ~62 GiB of the 114 GiB GTT.

Seam retargeted (FETCH -> the q4 script, TARGET_STUB, context/sessions pins).
2026-10-01 14:08:50 -07:00
Frank 35e24db7e6 frank-small: reslice to 2 x 262,144 native slots (was 4 x 151,552)
The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at
the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first
long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack.

Motivation: harness compaction. frank-primary's contextWindow 262,144
compacts at 70% = 183,501 tokens, and the compaction request carries a
32,768-token summary budget -> 216,269 must fit ONE frank-small request.
The 151,552 slot window could not hold it; the native window does.

No spec-decode interaction: checkpoint ships zero mtp.* tensors (config
lies), seam keeps every --speculative-* flag away from the engine.
2026-10-01 14:03:25 -07:00
Frank 389b605baf frank-primary: 2 bf16 windows + --expert-precise-memory
Both of Frank's changes at once.

KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.

--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
 term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.

Seam + docs + AGENTS row + log updated.
2026-10-01 13:52:13 -07:00
Frank 3f1860aa5f docs(log): record e2e 12-crawl flakiness (Plex redirect + transient ERR_ABORTED)
The gate failed twice in the same spec for two different reasons and then passed
33/33 on a manual re-run. Future sessions should re-run before treating the
DEPLOY INCOMPLETE banner as a real fleet fault.
2026-10-01 13:40:24 -07:00
Frank e3c3fad0f1 frank-primary: TimeoutStopSec=180, drop duplicate StartLimitIntervalSec
ExecStop runs 'podman stop -t 120' but the unit never set TimeoutStopSec, so
systemd's 90 s default cut the graceful stop short: journal shows 'Stopping
timed out. Terminating.' then 'Failed with result timeout' on 2026-10-01
13:27:17 — every stop of this unit was logging a failed stop. 180 s covers the
engine's drain with margin, and the seam now guards the RELATIONSHIP
(TimeoutStopSec > the podman grace) rather than a literal.

StartLimitIntervalSec is a [Unit] directive; the copy in [Service] made systemd
log 'Unknown key StartLimitIntervalSec in section Service' on every load. Seam
asserts it appears exactly once and not in [Service].
2026-10-01 13:30:29 -07:00
Frank a1b650aabc frank-primary: 3 full windows, pool at the contract floor, MNBT 2048
Frank's call 2026-10-01: MAXLEN 262144 x SEQS 3 = 786,432 tokens of contract.
Measured fp8 KV costs 7,698 B/token (4-slot boot 8,311,013,376 B -> 1,079,582
tokens; 5-slot boot 10,388,766,720 B -> 1,350,182 tokens), so the pool floor
for the contract is 6.054e9 B. Pinned at 6,100,000,000 (~792k tokens, ~5.9k
margin) so the remaining ~2.2 GiB versus the 4-slot shape goes to the expert
ARENA - the fork sizes its decode-critical expert cache from the remainder
after weights, pool and workspace.

MNBT 2048 (Frank): the arena sweep measured 6,499 slots at every rung <= 4096,
so the larger batch costs the arena nothing.

Seam retargeted (3 seqs / 6.1e9 / MNBT 2048); docs + AGENTS row + log updated,
including the closed DCP investigation.
2026-10-01 13:25:27 -07:00
Frank 6bbcaf3999 docs(frank-primary): parallelism survey — DCP rejected on 2 GPUs (GQA constraint)
--decode-context-parallel-size 2 fails VllmConfig validation at boot:
'Decode context parallelism for GQA/MQA requires --tensor-parallel-size (2) to
be greater than the model's total number of KV heads (2)'. DCP shards KV along
the sequence dimension AFTER KV heads spread across TP ranks, so it needs
TP > num_kv_heads; this model has 2 KV heads, so TP >= 4 — four GPUs, not two.
Recorded with the rest of the survey (PP/DP/EP/multi-node verdicts for this
box), the container-injection requirement for /home/dank/mtp_measure.py
(it imports /bench_iter.py by absolute path and must be copied in), and the
re-measured deployed capacity (1,079,582 tokens / 4.12x).
2026-10-01 13:17:45 -07:00
Frank 148a5f2575 frank-primary: wire --decode-context-parallel-size (opt-in)
DCP is passed only when FRANK_PRIMARY_DCP is set, so the deployed shape is
unchanged. Verified in this fork: DCP reuses the TP ranks when PCP is
disabled (tp % dcp == 0), so TP=2 + DCP=2 fits the two R9700s with no extra
ranks; the flash_attn backend carries DCP support, the 36 linear-attention
layers of this hybrid do not — which is what the A/B window tests.
2026-10-01 13:05:16 -07:00
Frank 667ec2987c frank-small: correct the unit description (no MTP, 4x151k) 2026-10-01 07:06:15 -07:00
Frank 0c5e054fb8 frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback).
SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0
nvcc toolchain for the sgl-kernel/flashinfer JIT.

Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba
states pinned at 32 slots (the engine autofills the ssm pool to a fixed
~5.1 GB regardless of dtype), eager prefill (no headroom for graph
capture), no spec decode (the checkpoint ships no mtp.* tensors despite
mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB
ghost at 0% acceptance).

Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k
requests, vision (in-checkpoint tower), reasoning_content field.
2026-10-01 06:55:48 -07:00
Frank 7da504a651 frank-primary: four full windows + Restart=always (runtime survivability)
Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
2026-10-01 04:14:25 -07:00
Frank bd311bf591 frank-primary: 5 x full 262144 contexts at fp8 (final shape)
Frank's call after the night ladder: five contexts at the model's full native
262,144 window beat six shortened ones. Six full-length contexts do NOT fit at
any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the
6-seq graph/activation footprint leaves too little for the pool); 5 x 262144
at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured
'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens
per request: 5.15x' (39,462 tokens of margin).

Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured
earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling
law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144)
re-pinned.
2026-10-01 03:52:10 -07:00
Frank 7cab15ac39 frank-primary: max-possible window (241664) + reasoning/tool contract
Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.

Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.

Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
2026-10-01 03:08:52 -07:00
Frank e8524d4046 frank-primary: util 0.90 — production bf16 boot OOM fix
The 0.98 boot died at KV allocation: torch OOM tried 10.28 GiB (the exact
3x262144 bf16 pool = 11,047,010,304 bytes) with 9.46 GiB free per rank.
The rootless bench probe had booted the same numbers; the root unit's
pinned-expert path holds ~0.8 GiB/rank more. Util 0.90 returns ~2.5 GiB/rank
of workspace with the pool ask unchanged; measured KV capacity law
(798,280 tokens at 11.05e9) unchanged.
2026-10-01 02:12:54 -07:00
Frank c1248d4a5d frank-primary: bf16 no-spec cutover (262144x3, gather 32x64, MNBT 1024)
Overnight redo campaign verdict + Frank's call: the fp8+MTP4 shape is retired
to git tag frank-primary-fp8-mtp-20261001. The new incumbent: KV cache
unquantized (auto), no speculative decode, KV_BYTES 11,047,010,304/rank
(engine-measured 798,280 tokens, 3.05x concurrency at 262,144-token
requests), MNBT 1024, MoE gather 32x64 (+18.7% decode, grows with depth).
Measured at the deployed shape: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
Seam re-pinned: the 7.2 GiB/rank KV ceiling was the MTP4 workspace, not VRAM;
spec decode absent by measurement; bf16 trade documented (clean cache +
262k windows for ~14% decode at 4064 ctx).
2026-10-01 01:54:27 -07:00
Frank 26f6f82c98 frank-primary: qwen cutover docs + deployer identity + seam re-pin
- AGENTS.md: fleet table + frank-primary section back to qwen FP8 + MTP4
  (37.4-40.6 tok/s, 54% acceptance); GLM runbook moved to the retired index.
- runbook: RETIRED verdict — DFlash2 drafter served and drafted but
  acceptance pinned 1.00-1.38 on every text class including verbatim
  repetition (mis-wired aux capture, not a weak drafter); 5x slower than
  baseline at ~1.2 acceptance.
- log.md: 2026-10-01 rollback entry.
- deploy-remote.sh: LLM_PRIMARY_CONTAINER back to frank-primary-qwen;
  glm-probe joins the legacy sweep list.
- frank_primary_unit.bats: re-pinned to the qwen shape — KV qualification
  law (7.2 GiB/rank, 3.65x concurrency; 9/16 GiB OOM), fp8 q8/q8 KV
  PRESENT, MTP4 PRESENT (3.1-3.4x over the 12.11 no-spec baseline), no tool
  dialect, automatic offload. The old 'MTP buys zero' law carried the
  prefetch tax and is retired.
2026-09-30 20:24:49 -07:00
Frank d91379cff2 frank-primary: roll back to qwen FP8 + MTP4 (2339fae trio)
DFlash2 experiment declared failed 2026-10-01: the drafter serves and
drafts (12.8 tok/s proposed) but acceptance is pinned at 1.0-1.38 on ALL
text classes including verbatim repetition — the aux-hidden-state capture
semantics are wrong (single-stream entering-state vs post-layer/comb is
the untested alternative). At ~1.2 acceptance the 8-token verify tax makes
decode 5x SLOWER than the 9.9-10.0 baseline (1.6-1.8 tok/s measured).

Restores the verified 37-40 tok/s shape: qwen3.8-flash-next FP8, MTP4,
prefetch off, 200k x 3 seqs, fp8 KV.
2026-09-30 20:13:23 -07:00
Frank caee26e3c3 llm: patch 04b v2 - plan-time signal is the spec-config METHOD
use_aux_hidden_state_outputs lives on the gpu_model_runner instance
(gpu_model_runner.py:623/678), not on the spec config, so the first
registry check never fired and the reserve stayed at 13,600 B/token.
Switch to method in {dflash, dspark, extract_hidden_states}.
2026-09-30 19:29:18 -07:00
Frank 9198da375f docs: dflash KV journey — 03 verified (drafter graphs capture), 04 capacity/aliasing wall 2026-09-30 19:22:45 -07:00
Frank d56db43aa3 llm: patch 04 - glm5 KV pool carries draft + hidden-state groups
The dflash boot cleared the unifier and the kpool tail (02/03) and died on
KV capacity: on the generic path the mamba states are NOT aliased into the
MLA pages (the glm5 tensor layout's whole point), so the capacity estimator
demanded ~30.6 GiB fixed against 0.92 GiB available (the plan's KV reserve
registry only knows the target's 13,600 B/token).

Patch 04 keeps the aliased glm5 pool and appends the drafter's uniform
group and per-layer hidden-state groups beside it: grouping partitions
hidden + draft specs, the layout returns a 10-tuple, the builder sizes one
shared backing allocation (pool + extras), and the memory estimator adds
the extras. Patch 04b extends the GLM registry kv_bytes_per_token by the
drafter (4,096 B/token) and hidden-state (5 x hidden x 2 B/token) costs so
the expert plan reserves enough.

Local smoke tests (CPU torch + real spec classes) validate grouping shape,
layout recognition, builder block math and memory accounting against
synthetic GLM+draft+hidden spec sets. Apply-verified: reproduces shas
5521e50b... / 1a623d5c...
2026-09-30 19:19:59 -07:00
Frank 2bc2f8c833 llm: patch 03 - pin the kpool tail block through the unifier
Patch 02 (MLA-multiple max page) cleared the KV page-size unifier: the
dflash boot now reaches cudagraph capture. New blocker: the kpool decode
kernel asserts tail_kv_cache.shape[2] == pool_size; KpoolTailSpec pins
block_size == index_kpool, so the unifier's divisible branch must never
grow it. Patch 03 = 02 + a KpoolTailSpec branch that pads page_size_padded
instead. Apply-verified: reproduces patched sha 5093e4b7... Runbook
updated with the full KV grouping journey.
2026-09-30 18:38:53 -07:00
Frank a1f14a1c0f docs: dflash patch v5 clears the EAGLE3 gate — wrapper is the real target
Runbook: the five-version evolution recorded, each trap named (method
hand-write, Protocol-base MRO kill, wrong-wrapper attrs). The decisive
discovery: the runner checks the LOADED instance, which for GLM-5.3-Flash
is the multimodal Glm5NextForConditionalGeneration holding the text model
as self.language_model — v4's attrs on Glm5NextForCausalLM were invisible
to it. v5 (attrs + methods on both wrappers) passes: both TP workers log
'Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, ...)' and the
DFlash2 drafter resolves + loads. Log: the DFlash bullet updated from
'the trunk simply does not implement' to 'gap closed; serving end state
still unproven (window RAM contention interrupted the boot twice)'.
2026-09-30 17:59:35 -07:00
Frank b877d87588 patches: glm53-dflash-01 v5 -- patch the CONDITIONAL wrapper (the real eagle3 target)
The v4 boot named the actual miss: model_runner.py:425 calls
set_eagle3_aux_hidden_state_layers(self.model) on the instance model_runner
LOADED -- and for GLM-5.3-Flash that is Glm5NextForConditionalGeneration (the
multimodal wrapper; APIServer log: 'Resolved architecture:
Glm5NextForConditionalGeneration'), which wraps the text model as
self.language_model = Glm5NextForCausalLM (glm5next/common/model.py:1113).
v4 put the three presence-checked attributes only on Glm5NextForCausalLM,
so the isinstance check on the conditional still failed with the same
RuntimeError.

v5 adds to BOTH wrappers, MRO-safely (attribute route, no Protocol base):
  - Glm5NextForConditionalGeneration: supports_eagle3 / has_own_lm_head /
    has_own_embed_tokens + the two methods delegating to self.language_model
    (the pixtral.py:498 pattern);
  - Glm5NextForCausalLM: the same three attributes + the two methods that
    unwrap to self.model (the EagleModelMixin trunk -- v1's correct bodies).

Also proven by MRO simulation: the ForCausalLM's five-base ordering cannot
take SupportsEagle3 as a base in ANY position, so inheritance was never a
viable route for this model.

base 008eeaa2... -> patched bc9913a1..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted file).
2026-09-30 17:41:19 -07:00
Frank 7560d40543 patches: glm53-dflash-01 v4 -- all three SupportsEagle3 members on the wrapper
The v4 window boot named the real presence-check rule: isinstance against a
runtime_checkable Protocol walks __protocol_attrs__ (which includes members
inherited from BASE protocols) and getattr_static()s each one off the instance,
breaking on the first missing attribute. SupportsEagleBase declares
has_own_lm_head/has_own_embed_tokens as plain instance-var annotations, so
v3's bare supports_eagle3 flag still failed. Replicated in-image with stock
typing: flag-only class -> False; flag + the two base members -> True.

v4 declares all three on Glm5NextForCausalLM:
    supports_eagle3: ClassVar[Literal[True]] = True
    has_own_lm_head: bool = False
    has_own_embed_tokens: bool = False
(False is correct for the two: the wrapper delegates embed/lm_head to its
trunk. The MRO-safe attribute path is kept from v3.)

base 008eeaa2... -> patched 60f27aad..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted file).
2026-09-30 17:24:56 -07:00
Frank e044277d8b patches: glm53-dflash-01 v3 -- declare supports_eagle3 as a class attribute
v2 (inheriting SupportsEagle3 as a base) taught the real lesson: the isinstance
check PASSES and the class then dies at creation with
'Cannot create a consistent method resolution order (MRO) for bases ...'
-- GLM's outer wrapper already inherits vLLM Protocols (SupportsPP,
MixtureOfExperts, HasInnerState, IsHybrid) whose ordering is incompatible with
appending another Protocol base.

v3 avoids the MRO entirely: declare
    supports_eagle3: ClassVar[Literal[True]] = True
as a class attribute on Glm5NextForCausalLM.  Runtime-checkable Protocols do
presence-based isinstance, so the bare flag satisfies supports_eagle3(model)
without touching the MRO.  The aux-layer plumbing stays on the trunk via
EagleModelMixin.

base 008eeaa2... -> patched 4c9a632b..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted dflash-patch
file byte for byte).
2026-09-30 17:08:50 -07:00
Frank 9c5c2494da patches: glm53-dflash-01 v2 -- satisfy SupportsEagle3 by inheriting the Protocol
v1 hand-wrote set_aux_hidden_state_layers/get_eagle3_default_aux_hidden_state_layers
and the boot reproduced the identical 'Model does not support EAGLE3 interface'.
Reason: supports_eagle3() is an isinstance check against a runtime-checkable
Protocol whose members include a required ClassVar flag
(interfaces.py:1663 supports_eagle3: ClassVar[Literal[True]] = True), so methods
alone do not satisfy it. Inheriting SupportsEagle3 -- the idiom every model in
this image uses, including the fork's own DFlashLagunaForCausalLM -- supplies the
flag, SupportsEagleBase's has_own_lm_head/has_own_embed_tokens, and working
default implementations of both methods; the hand-written methods are deleted.

base 008eeaa2... -> patched 5eea89c5..., apply-verified against the pinned image.
Runbook: corrected method name, recorded the Protocol trap as the generalizable
lesson for future spec-decode ports.
2026-09-30 16:53:24 -07:00
Comrade Chief Designer ce280e7295 fleet docs: AGENTS.md fork-work paragraph corrected and the DFlash port recorded
The frank-primary section still carried the retracted framing ('a latent bug the compile
exposes' in the mHC/KDA path) and listed only three compile patches. It now records:
six apply-verified artifacts across two WIP changelists (compile 01-05 + the DFlash2
enabling patch), the measured facts (lru_cache does not protect a probe; suppress_errors
does not cover breaks; the trunk can never satisfy fullgraph because its indexer
metadata syncs to host), the single remaining blocker
(KeyError: 'in_proj_qkvbfg_a', raised in a callee dynamo inlines into
Glm5NextDecoderLayer.forward, not at any of the four nn_modules lookups and not a literal
string lookup -> currently a dynamo-side name-resolution failure, not a demonstrated fork
bug), and the two operational gotchas this session paid for (the launcher silently
dropping VLLM_COMPILE_FULLGRAPH=0, and the PYTHONPATH sitecustomize fork bomb). The
DFlash2 paragraph now says the aux-hidden-state contract is written, and that the
engine's refusal was literally 'Model does not support EAGLE3 interface'.

tests/run.sh: 362 seams pass.
2026-09-30 16:29:19 -07:00
Comrade Chief Designer d2e059262f frank-primary: patch 05 is not part of the confirmed blocker
Clarifies scope in the artifact header: torch.compiler.disable is itself a graph break,
so 05 helps only a hypothetical fullgraph=True route (where the workspace acquisition
would be better restructured out of the forward than made opaque). It is kept because
the discovery is real and the wrapper is a no-op when nothing is compiling, but it is
not required for the break-tolerant route whose only confirmed blocker is the dynamo
name-map KeyError.

Also verified in this pass: glm53-dflash-01-aux-hidden-states.patch applies cleanly to
the pinned image and reproduces its recorded patched sha256 (4ff6ff8d...), so all six
artifacts in scripts/patches are now apply-verified.
2026-09-30 16:27:24 -07:00
Comrade Chief Designer 3cf543ead6 frank-primary: fork-bomb hazard, patch 06 documented, retired probe
Recording the incident: a diagnostic sitecustomize.py mounted at /probe on PYTHONPATH
produced a self-replicating chain of python3 processes 19+ levels deep inside the probe
container (sitecustomize runs in EVERY interpreter; the hook imported torch/vLLM at
startup; something in that import chain spawned a helper python which inherited
PYTHONPATH and the hook, and spawned again). Found alive 18 minutes later, 33 probe-tagged
processes. Killed with 'podman rm -f glm-probe' (the container cgroup owns every
generation) and verified by counting /proc/<pid>/environ for the marker variable --
never by pattern-killing an interpreter name, since this host runs python in production
containers (romm, jellyfin, plex, glances). Fixed at the source with an env marker that
children inherit, verified with a two-generation test: gen 1 installs, gen 2 stays inert.

The runbook now carries the hazard, the guard, the kill recipe, the conjunction the
restore must verify, and the no-overlapping-jobs rule. Patch 06
(glm53-dflash-01-aux-hidden-states.patch) is documented in the DFlash2 section, and the
/probe diagnostic is marked retired -- TORCH_LOGS answers dynamo questions with no
startup hook.

tests/run.sh: 362 seams pass.
2026-09-30 16:24:16 -07:00
Comrade Chief Designer c7db6d94e9 frank-primary: DFlash2 enabling patch — EAGLE3 aux-hidden-state interface for the GLM trunk
glm53-dflash-01-aux-hidden-states.patch: implements the interface whose absence makes
the engine refuse the drafter with 'Model does not support EAGLE3 interface'. Adds
EagleModelMixin to Glm5NextModel, captures the single-stream hidden state entering each
requested layer in the loop, returns (hidden_states, aux_hidden_states) as a tuple when
the list is non-empty, and exposes set_aux_hidden_state_layers /
get_eagle3_default_aux_hidden_state_layers on Glm5NextForCausalLM -- the contract read
out of interfaces.py:1694/1712, eagle3_utils.py:26-31, deepseek_v2.py:1551-1585 and
cudagraph_utils.py:621, with kimi_k25/pixtral as the delegation reference.

Independent of the compile changelist (01-05); it patches the same file as 01, so 01
goes first if both are applied. Scope is honest in the header: this makes the interface
present (the prerequisite for driving DFlash2), it does not promise draft quality and is
not a decode-speed fix.
2026-09-30 15:57:27 -07:00
Comrade Chief Designer 0465b9b5a7 frank-primary: pin down the compile blocker — a name-resolution KeyError in an inlined callee
Five runs of the changelist now reproduce the same single failure with breaks genuinely
tolerated (patch 01+02+03+04, VLLM_COMPILE_FULLGRAPH=0, VLLM_USE_AOT_COMPILE=0,
R4D_GLM_FUSE_QKV_RMSNORM=0): the trunk TRACES ('Dynamo bytecode transform time: 1.08 s')
and the boot dies on KeyError: 'in_proj_qkvbfg_a'.

What it is NOT, each ruled out by measurement rather than argument:
 - not one of dynamo's four nn_modules lookups: /home/dank/dynamo-probe wraps three of
   them and stayed silent over two runs, with the probe's install line confirmed in both
   the API-server and worker processes;
 - not a dynamo-internal failure: TORCHDYNAMO_VERBOSE=1 adds no internal stack trace;
 - not a literal string lookup: the name appears tree-wide only as the KDA layer's
   attribute (kda.py:217, used at :407) and in the load_weights shard table
   (model.py:780-785); the four  sites in the
   quantization utils do compute exactly this bare key but every one is guarded
   (), so none can raise it;
 - not the pluggable-layer registry, which is keyed by class name.

What it IS: a plain Python KeyError raised inside the traced
Glm5NextDecoderLayer.forward, in a callee dynamo INLINED (frame chain
profile_run -> _dummy_run -> execute_model -> glm4_1v.py:2358 -> the compile wrapper ->
eval_frame.compile_wrapper -> Glm5NextModel.forward:748 -> module._call_impl ->
Glm5NextDecoderLayer.forward:456; the reported line is that method's def line, which is
how inlined frames surface).

Patch 03's header and the runbook now record the remaining instrument (TORCH_LOGS=+dynamo
and grep, or wrap the inlining candidates — the mHC CustomOps and the indexer op — with a
KeyError probe), and the two breaks that are real but non-fatal in this configuration
(patch 04's fused-norm launcher cell, patch 05's workspace acquisition).

tests/run.sh: 362 seams pass.
2026-09-30 15:39:10 -07:00
Comrade Chief Designer 62779aa237 frank-primary: document the experiment-window trap that faked two successful restores
This engine ignores SIGTERM: after systemctl stop its container sits in podman ps
state 'Stopping' for 20+ minutes WHILE STILL ANSWERING /health with 200. So a liveness
probe is a lying oracle for a window restore — twice a restore reported PROD HEALTHY
while systemctl is-active said failed and the runtime mask was still in place, because
the old container was still serving.

Records the conjunction the restore must verify (is-active == active AND is-enabled ==
enabled AND no /run/systemd/system/<unit> mask symlink AND the container is owned by
the unit and not Stopping) and the four-step repair recipe (mv the runtime mask away —
it survives stop and blocks start; daemon-reload; reset-failed — a failed unit refuses
to start; podman rm -f — podman stop hangs on the same SIGTERM).

Also records the overlap hazard: a later job's 'podman rm -f glm-probe' cleanup
deleted an in-flight job's container, so one compile round produced no evidence at all
('no container with name glm-probe found'). Check proc:// for running jobs before
opening a window.

tests/run.sh: 362 seams pass.
2026-09-30 14:54:07 -07:00
Comrade Chief Designer 71db0debcc frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory
Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.

New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).

All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).

tests/run.sh: 362 seams pass.
2026-09-30 14:19:25 -07:00
Comrade Chief Designer fd864b5a42 frank-primary: compile changelist — real diffs, break diagnostics, patch 04, corrected overclaim
Working the dyno name-map blocker one step at a time, with the tools to see it:

- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
  hunks before, useless to patch(1)). Verified against the pinned image: each applies
  clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
  / 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
  MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
  already a constructor arg glm5next sets at attention.py:583). Reason: in
  models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
  its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
  breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
  VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
  /home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
  to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
  executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
  dropped env vars and wasted a boot.

Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.

tests/run.sh: 362 seams pass.
2026-09-30 13:59:09 -07:00
Comrade Chief Designer 1c18b9c808 frank-primary: DFlash2 load attempt + fp8 verdict + compile blocker narrowed
DFlash2: fetched incoai/GLM-5.3-Flash-DFlash2 (2,342,169,800 B, sha256 matches the
Hub LFS oid) and tried --speculative-config {"method":"dflash","model":"/draft"}.
Config and drafter are accepted; the worker dies with 'Model does not support
EAGLE3 interface' - so the only gap is the target-side aux-hidden-state contract
(set_aux_hidden_state_layers + per-layer capture + the two-value forward return,
as deepseek_v2.py demonstrates twice in this image). Recorded the exact interface,
the GLM-specific unknown (what the single-stream state entering layer k means for
an mHC multi-stream trunk) and the honest expectation: no speedup on a
token-proportional pointwise bound.

fp8 KV: recorded the deciding measurement - the fork's KV reserve model is
dtype-blind (the fp8 boot printed the identical 1760 MiB / 13600 B-per-token plan),
so fp8 cannot buy capacity even if the gfx12 MLA path were rewritten. Verdict: do
not spend effort there.

Compile blocker narrowed from 'a latent GLM bug' to dynamo's name map: the
KeyError is thrown by torch/_dynamo/output_graph.py's nn_modules lookup (:3000,
registered :1259/:1384, _modules[..] prefixes stripped :1225), i.e. the compiler
cannot resolve a submodule by name while tracing the decoder layer. Next step is
TORCHDYNAMO_VERBOSE=1, with the note that the scratch launcher resets ENVARGS.

tests/run.sh: 362 seams pass.
2026-09-30 12:49:26 -07:00
Comrade Chief Designer bd8c72ff47 frank-primary: profile the GLM decode step and stage the compile changelist
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:

  46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
   34.8 %  vectorized_elementwise_kernel      708,500 launches (1,428/token)
   10.0 %  r4d_gemm_moe_w4a16_nt_b16_m64      (the MoE GEMM)
    6.7 %  elementwise_kernel_manual_unroll
    6.4 %  wvSplitK_hf_sml_                    (split-K GEMM)
    8.9 %  router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
    6.5 %  r4d_mhc_*                           (hyper-connections)
    1.05 % r4d_moe_lru_gather_kernel           <-- the retracted suspect

The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).

Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:

  -01 models/glm5next/common/model.py    decorate Glm5NextModel (R4D_GLM_COMPILE)
  -02 model_executor/kernels/r4d_lib.py  freeze 10 capability probes that do
      os.path.exists/.so lookups inside the traced forward. functools.lru_cache
      does NOT protect them: dynamo re-traces the wrapped body even when warm
      (verified in-image); a plain module-global read compiles.
  -03 compilation/wrapper.py             make fullgraph=True opt-out, because
      suppress_errors does NOT cover graph breaks and this trunk can never
      satisfy fullgraph (indexer metadata syncs to host — .item() at
      glm5next/amd/sparse_indexer.py:263,419).

With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.

Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).

Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
2026-09-30 09:36:48 -07:00
Frank 979a4308f4 frank-primary runbook: record the allocator env and the e2e/FrankBook-Pro coupling
Two operational facts the cutover surfaced: PYTORCH_CUDA_ALLOC_CONF=
expandable_segments:True is part of the qualified shape (the plan fills VRAM to
its modelled headroom, so the default allocator's fragmentation fails
profile/graph capture — measured on the deployed unit), and the deploy's e2e
gate drives the admin open-links spec, which walks every guarded vhost
including 192.168.0.112:8989 on FrankBook-Pro — a sleeping Mac mini fails that
spec and reports DEPLOY INCOMPLETE while the engine is healthy.
2026-09-30 05:40:57 -07:00
Frank a3e1c97408 frank-primary: pin PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
The deployed unit failed its first boot with the engine core OOMing during
profile/graph capture: "Tried to allocate 662.00 MiB. GPU 0 has a total
capacity of 31.86 GiB of which 240.00 MiB is free ... try setting
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True". The expert-offload plan
fills VRAM to its modelled headroom by design, so the default allocator's
fragmentation leaves the capture phase nothing — every successful bring-up
probe (both scratch-port and 7770) carried this env, and the repo recipe
dropped it.

Also pins the seam: the setting is part of the qualified shape, not an
optional tuning.
2026-09-30 05:13:50 -07:00
Frank f2bcf84e88 frank-primary cutover: GLM-5.3-Flash W4A16 int4 (Intel AutoRound)
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).

Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.

Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.

Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).

Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.

Rollback: 2339fae (qwen FP8 + MTP4).
2026-09-30 04:22:11 -07:00
Frank 2339faeaa4 frank-primary: fix the SIGPIPE race in both image-digest gates
The classic 'podman images | grep -q' pattern races under set -o pipefail:
grep exits on its first match, podman dies SIGPIPE (141) writing the
remaining RepoDigests lines, and pipefail promotes 141 to the pipeline
status — the gate reads 'no match' and refuses to deploy (2026-09-30,
dankserver). Same bug in the serve script (it would restart-loop the
same way). Fix: capture podman's output into a variable first, grep the
variable. Patched in the R9V September deploys by luck of timing.
2026-09-30 01:05:05 -07:00
Frank 01fedb0081 frank-primary.env: quote the space-separated LEGACY_CONTAINERS list
Unquoted value + set -a sourcing parses 'qwen-fp8' as a command ->
ExecStart exit 127 -> unit restart loop (caught on the first deploy).
The deployer's wait_for_health was killed and will re-run on the fresh
deploy.
2026-09-30 00:58:32 -07:00
Frank 3bb0e82ccf frank-primary cutover: tcclaviger RDNA4 stack, 600k ctx / 3 sessions / q8-q8 KV
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the
digest-pinned image the 2026-09-29 tuning campaign measured), serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB,
already on disk) with MTP4 spec decode.

Qualified by boot probe on 2026-09-29:
- MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV):
  'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000
  tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK.
- KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to
  allocate the full size; ~7.5 GiB/rank is the ceiling with the arena).
- prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic);
  TunableOp off (corrupts numerics).
- decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots.

Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh
(direct podman run; entrypoint execs vllm serve; waits on the container),
frank-primary.service (root unit, same ExecStop law, new container name
frank-primary-qwen). Deployer gates rewritten (image digest + shard count
+ weight-map index). download-frank-primary.sh is now inventory/verify
(repo is GATED — 401 anonymous, masked LFS hashes; payload already on
disk). Seam rewritten for the new contract. Pre-cutover tag:
frank-primary-r9v-q8-20260929.
2026-09-30 00:46:30 -07:00
Frank 9cf4b1de82 docs: qwen-fp8 endpoint published on 7773; fix log bullet placement
PORTMAP=7773:7770 hook; health 200 on 127.0.0.1 and 192.168.0.235,
decode through the published path in band. 7773 chosen over 7770
(frank-primary owns 7770 when it returns). Log bullet recovered from a
misplaced insert into the 2026-09-28 cleanup section.
2026-09-29 23:51:53 -07:00
Frank d28ab8ef8c docs: record the qwen-fp8-tuned-20260929 tag + digest-pinned image
Repo tag qwen-fp8-tuned-20260929 at f532e03; serve script launches the
tcclaviger image by digest sha256:da402966... (local alias
qwen-fp8-runtime:tuned-20260929) — the floating :latest is no longer
referenced.
2026-09-29 23:31:13 -07:00
Frank f532e03bf4 docs: GLM-5.3-Flash readiness revised — tcclaviger fork has native MXFP4/DSFP4
Source reading in the qwen-fp8 container: glm5next is registered with an
amd/ backend dir + DSA indexer; r4d.so has native moe_dsfp4/moe_dsfp4_mt1
entries (has_dsfp4()==True), so the old 'MXFP4 = emulation, missing
kernels' verdict is dead for this fork. DSFP4 = MXFP4 weights + DeepSeek
activation law. Offload recipes exist for mxfp4 and W4A16 (R4dW4a16MoEExperts
in offloader/expert.py:114). GLM gate is now the payload format
(compressed-tensors MXFP4/W4A16, none public) + a HIP boot probe.
2026-09-29 23:28:04 -07:00
Frank 57072a44df docs: final 256k verdict — boots at 4.03-4.5 GiB KV, band 34.1-36.5 straddles target
The KV trim is inert: arena identical (5,648 slots) at 4.5 and 4.03 GiB
(planner buckets; the trim went to margin). Final 256k config
MAXLEN=262144 KV_BYTES=4328521728 prefetch off; full-speed alternative
MAXLEN=184224 at 3 GiB.
2026-09-29 21:55:58 -07:00
Frank 0ccb01708b docs: qwen38-fp8 expert-offload tuning — prefetch off hits 37.4-40.6 tok/s (target met)
Measured campaign on the ad hoc qwen-fp8 container (tcclaviger RDNA4
fork, 2x R9700, TP=2): telemetry (with_stack=false) measured the
prefetcher at 36% precision — ~half of gather traffic wasted; deleting
--expert-prefetch: 26.8 -> 40.57/37.44/37.57 tok/s. Prefetch-on shape
survived every other challenger (gather geometry, spec depth, TunableOp
[corrupts numerics], MNBT, concurrency, ngram [PLE fork bug]). 256k ctx
boots at 4.5 GiB KV (36.65/34.21/34.77, arena -14%). New runbook +
index links + log entry.
2026-09-29 21:22:21 -07:00
Frank 67672dc794 docs(llm): record a measured NEGATIVE tuning result for frank-large
Asked for iterative decode/prefill improvement at a 32k working point
without touching the model or the quantization. There isn't one: gufo's
served settings are already at this engine's tuned point, so no config
change was made and none is proposed.

Measured on the repo's own prose and shell sources at 32,762 prompt
tokens, salted per run against a warm prefix cache, 3-5 repetitions:

* --prefill-chunk 512 -> 8192 moves prefill 410.7-412.4 t/s. A 16x change
  in chunk size, 0.4% out. The lever is dead at this depth.
* --draft-tokens 5 -> 10 gives 13.79-15.86 t/s decode, all inside the
  run-to-run noise. The 5-rep confirmation REVERSED the 3-rep leaders:
  dt9+chunk4096 looked best and came last (12.86), the deployed config
  came first (14.71).
* --draft-policy fixed vs adaptive: no effect. --min-draft-tokens:
  rejected by the engine ("DFlash2 requires ... 1"). MTP: unsupported for
  Q8 targets (token_embd.weight is Q8_0).
* --context 262144 -> 40960, tested as a diagnostic: no gain, and it
  would break models.yml's declared contextWindow. Not shipped.

What does move the number is speculation itself — 6.65 t/s off vs ~15 t/s
on, 5.3x — and it is already enabled.

The two traps that cost most of the wall time are now in AGENTS.md's
benchmarking policy, because both produced FALSE results:

1. gufo bench cannot rank speculative settings and inverted the ranking.
   Its corpus reports acceptance=0.933 +/- 0.000 with an identical trace
   sha256 across draft widths — the same highly-predictable text every
   run. It reported 29-37 t/s where real traffic gives ~15, and made
   dt16 look 6% faster than dt7 when real text made it 14% slower.
2. A live A/B needs >=128 generated tokens. At 8, inter_token is
   first-token noise; the first pass produced a nonsense ordering that
   the 128-token pass reversed.

Plus the Server-Timing header is `ttft;dur=402.485` — the unit rides in
the NAME half, so splitting on "=" keys the dict "ttft;dur" and every
lookup silently returns nothing. Four configs reported "no Server-Timing"
while the header was present on all of them.
2026-09-29 05:06:06 -07:00
Frank 3a931ce2c1 chore(dankclaw): reclaim 510G of retired payloads; fix a stale fallback
The disk was at 93% with 137G free. Nine retired payloads are now moved
to ~/.AGENTTRASH/20260928-retired-models/ with a MANIFEST.md giving a
restore command per item — moved, not deleted, and nothing is freed until
the purge is approved. Every one maps to a runtime already retired in the
runbook index.

Only the .gguf was taken from Qwen3.8-27B-UD-Q8_K_XL/: the live
frank-large DFlash2 drafter lives in that directory and stays put. The
manifest warns against restoring the directory wholesale.

Kept: checkpoints/ (23G) — unreferenced by the repo, but ComfyUI is live
and its workflows were not searched, so it is not confirmed dead. Also
logs/, because x4e-run.log is a SPARSE file (3.3G apparent, 4K real) and
frees nothing.

The reference check caught a live bug: serve-frank-small.sh's fallback
MODEL_DIR still pointed at the retired GSQ-RCO payload frank-small served
on 2026-09-16/17. frank-small.env has always overridden it, so the drift
was invisible — but a missing or unreadable env file would have turned a
config problem into a "missing asset" naming a model that is no longer
there. Repointed at Gemma4-12B-QAT/, which the env file already sets.

Not taken, by request: podman images (279G reclaimable, 95%) and the
uv+pip caches (73G). Together 352G of regenerable garbage, larger than
what was reclaimed here.

The 29.3G payload rollback target went with the rest, so the payload-only
rollback is now a re-fetch first; the runbook and AGENTS.md say so.
2026-09-29 00:00:49 -07:00
Frank 7f9424554c frank-large: serve the abliterated Swift-1.5 UD-Q8_K_XL
The target moves from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL to
ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's UD-Q8_K_XL —
an abliterated fine-tune of the same base, under the non-OSI SWIFT OPEN
LICENCE 1.0. Port, alias, context, drafter and client contract are
unchanged.

Compatibility was settled from the GGUF headers before fetching anything:
both files parse to the same architecture (qwen35), the same 866 tensors
with identical names in identical order, zero per-tensor ggml_type or dim
mismatches, and the same MTP head. Two metadata differences that look
alarming are no-ops, confirmed against gufo's source — general.file_type is
deliberately not consulted (the file gufo already served records Q4_K_M
while containing no Q4_K tensor), and Swift's extra
qwen35.attention.recurrent_layers key is never read; layer kinds come from
full_attention_interval, which both files carry as 4 and which agrees.

The load test agreed: 9.2 s to listening, 66.5 GiB of the 116.7 GiB iGPU,
no loader warnings, and a green contract battery including a two-turn tool
loop.

The DFlash2 drafter did not move, so the payload is now two files in two
directories: DRAFTER_DIR is its own tunable. That broke the deployer's
stray-run sweep, which keyed on the target filename and would have stopped
matching — a stray run would then have held 7770 past the port gate. It
now keys on the drafter, which every frank-large command line carries.

The fetch script sends no auth header: the repo is public, and a token on
the curl command line is world-readable via /proc/<pid>/cmdline. The
anonymous transfer ran at 1.8 MiB/s; the host token through a mode-600
curl -K config reached 7 MiB/s.

Test seam: the mutation pass found four real gaps. Two fetch assertions
were satisfied by a header comment rather than the URL assignment, so
repointing a repository left them green, and nothing bound the env's
MODEL_FILE/DRAFTER_FILE to the filenames the fetch scripts download. Both
are now explicit. A harness bug worth recording: a bats --filter that
matches no test reports zero "not ok", indistinguishable from a mutation
that survived — the harness now checks the 1..N count. 16/16 caught.
2026-09-28 23:29:05 -07:00
Frank 32abc8d543 chore(llm): add dankstatus.sh — one verified parse path for Dankserver state
The ad-hoc status commands kept producing wrong readings, most seriously
reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free.
Cause: /proc/meminfo separates key from value with a TAB, so

  grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2

returns an empty field and awk prints 0.0. A false "host is out of memory"
reading is worse than no reading, because it invites an unnecessary abort.

dankstatus.sh parses on a colon split (tab-agnostic), cross-checks
MemAvailable against MemTotal before printing either, and refuses to print
a zero value. It also pins the rocm-smi CSV column order (device, total,
used) which was previously read swapped, and takes only the first line of
systemctl is-active/is-enabled, which emit more than one line here.

Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU)
as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing
the GTT aperture. The device is smaller than documented, not larger.

Tests: 346/346 pass (no test surface touched).
2026-09-28 23:27:50 -07:00
Frank b4b6777e49 fetch: add the Swift-1.5 frank-large candidate download
Public, ungated repo so no auth header is sent — a token on the curl
command line is world-readable via /proc/<pid>/cmdline. Size and sha256
are the Hub LFS values (31457990272 / 9076f99a…), verified after fetch.

Not wired into frank-large.env: the gufo load test has to pass first.
2026-09-28 22:48:53 -07:00
Frank 9d9d28cda6 docs(llm): record the gufo cutover verification
Live numbers from the 2026-09-28 deploy: 13 s load to listening, RSS
30.6 GiB, ttft 549.8 ms / inter-token 24.2 ms on a short prompt, and a
green contract battery (health, ready, /v1/models at 262,144, content +
reasoning_content, tool_calls, SSE, and a two-turn tool loop).

Also recorded: gufo accepts every request field the harness sends in
extraBody and the full reasoning_effort set, both verified 200 — which
is why models.yml's reasoningEffortMap fold is now optional rather than
required, and why it is being kept anyway.
2026-09-28 22:28:44 -07:00
Frank a196b0fa45 test: scope the halogen-quadlet check to tracked files
The .AGENTTRASH archive is ignored and exists only in a primary checkout,
so a filesystem grep fails there while passing in a worktree. git grep
searches the tracked tree, which is what the assertion is about.
2026-09-28 22:12:16 -07:00
Frank aaef419a6d frank-large: replace stock llama.cpp with gufo (gufo-org/gufo) on Dankclaw
gufo is a vertical inference engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large
has swapped runtimes seven times in search of a better one. This one
ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on
the host and the payload, port, alias and client contract stay exactly
where they are.

Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1,
built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4
through its RPATH — the opposite of the unsloth prebuilt it replaces,
which carried its own — so two host workarounds are encoded in that
script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set
that shadows AMD's and cannot be overridden with -I in HIP mode, and the
ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose
libstdc++ is actually installed.

Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL
target and the same DFlash2 drafter, greedy, text-only. Speculation
moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy
capped at 7 proposals.

Client-contract deltas, all documented: the chat template is compiled
into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token
failure mode is gone), reasoning_effort now takes the full OpenAI set
(so models.yml's reasoningEffortMap fold is no longer required — kept
anyway, so this commit does not also move every high role from xhigh to
high effort), and there is no KV-quant or prompt-cache-RAM flag.

The test seam now runs the serve script against a stub engine and
asserts on the argv it actually execs, instead of grepping the script's
text. The deploy handler gates on the runtime binary and returns before
it stops the serving unit.

Measured trade, from gufo's own Q8_K_XL tables against llama.cpp
b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero
depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the
8-bit target the crossover is ~32k, and frank-large runs one 262,144
conversation. The Q4_K_XL target shows no such regression.
2026-09-28 22:08:29 -07:00
Frank d29d104f01 docs: frontier-MoE feasibility on Dankserver (V4.1 / MiMo / GLM-5.3)
Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can
Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast.

Sizing, measured from the HF API rather than the model cards:
- DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB
  shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so
  they do not evict the expert page cache.
- 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency
  the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on
  Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any
  attention or dense work.
- The backend refuses layer-major prefill for V4.1, so the only legal
  prefill is token-by-token; 1000 t/s is ~20x away.

Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero
errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB
container leaves the cold expert set in a pinned host pool in RAM. V4.1 is
2.5x too big to repeat that. Port scope, from both checkpoints' real
safetensors indices: expert machinery ports unchanged (identical tensor
naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config
keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio
vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the
only key V4.1 lacks, so model.h:36's default of 3 silently survives and
would route layers 0-2 through a table that does not exist.

Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board
swap) and storage; the top-level index credited V4.1 with V4's geometry
(43 layers / 256 experts vs the actual 40 / 384).
2026-09-28 14:58:59 -07:00
Frank 564a781539 log: frank-primary deploy-state reconciled to ba2100e (autoresearch already live, no restart); e2e 32/33 (Mac mini offline) 2026-09-28 13:22:14 -07:00
Frank ba2100e22b docs: prefix caching deployed on frank-primary (E8-qualified, 9x/63x reuse TTFT) 2026-09-28 12:33:28 -07:00
Frank 41e918929f frank-primary: enable prefix caching (E8 qualified: 9x-63x reuse TTFT, zero VRAM cost, cold path unchanged) 2026-09-28 12:03:56 -07:00
Frank 92579b0674 log: frank-primary restore verified (f187165 deploy, ~11:00); e2e 32/33, 11-admin blocked by asleep Mac mini 2026-09-28 11:21:43 -07:00
257 changed files with 25757 additions and 1825 deletions
+446 -183
View File
@@ -12,8 +12,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the
RTX 3090). Caddy on
Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +25,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
@@ -40,7 +41,11 @@ the [runbook index](#runbook-index). The full swap history lives in
- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
scripts, systemd units, env tunables, bench harnesses
- `old/` — retired LLM scripts (reference only)
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format)
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
The managed skill library is distilled there one page per skill-cluster:
`LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
`Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
- `tests/` — bats suite + Playwright e2e
## Git Workflow
@@ -68,12 +73,20 @@ All edits flow through git. Never edit files on a remote machine directly.
The planner maps changed paths to services; the remote runner restarts only
the selected service and records its revision outside the checkout.
Documentation-only changes are a no-op. LLM services are selected only by
their own files — every endpoint (`frank-large`/`frank-small`/`frank-primary`)
their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`)
by its unit/env/serve trio in `scripts/` — so
unrelated
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
runs on a host without passwordless sudo; the value never enters the repo.
**`deploy.sh <target>` with no `--service` is a fleet-wide DRAIN, not a sync.**
It also walks every per-service revision marker under the deploy-state dir and
restarts each service whose marker is behind HEAD *and* whose paths changed
since that marker — so a backlog accumulated by service-scoped deploys fires
all at once, and one failing health gate aborts the run with every later
service left pending. Name the service unless a drain is what you want
(`./scripts/deploy.sh dankserver --service frank-primary`).
### Deployment completeness (MANDATORY)
A deploy is **not complete** when `deploy.sh` exits green. It requires:
@@ -82,7 +95,14 @@ A deploy is **not complete** when `deploy.sh` exits green. It requires:
2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as
the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a
behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass
only with `--skip-e2e` in an emergency.
only with `--skip-e2e` in an emergency. The gate runs the conditional
`frankbook` Playwright project (the `*arr`/qBittorrent walk,
`tests/e2e/specs/19-frankbook.spec.ts`) ONLY when a frankbook blast-radius
path changed since origin/main — that spec, `frankbook-pro/**`,
`shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the
arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy.
The default project set excludes it everywhere else (`E2E_INCLUDE_FRANKBOOK=1`
opts in; see `serverAppConfigs-docs/Architecture/E2E Suite.md`).
```mermaid
graph LR
@@ -208,13 +228,30 @@ archive.
**STOP the production service BEFORE benchmarking** — these servers serve
real users, and a bench beside a live service starves the GPU and corrupts
results. Use the runtime's own harness: `llama-bench` for every llama.cpp
endpoint (all three are llama.cpp now). Rolling your own bench
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench
requires asking first.
Scratch ports are not used on LLM hosts — a
test process cannot share the single GPU anyway; validate against the service
itself.
(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
settings, and will invert the ranking.**
Measured 2026-09-28 on frank-large at depth 32768: its corpus
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
draft widths, i.e. the same highly-predictable text every run, and 93 % of
every proposal accepted. It reported 29–37 t/s decode where real traffic
gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when
real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B,
replay **real** text through the live server and read the `Server-Timing`
header — but note the unit rides in the **name** half (`ttft;dur=402.485`),
so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently
returns nothing. Generate **at least 128 tokens** for the measurement: at 8
tokens, `inter_token` is first-token noise and the run-to-run spread swamps
every real difference.
This covers **any GPU experiment, not just benchmarks** — including a bare
2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run
against the live service "because it only needs 200 MB of VRAM"; the host went
@@ -227,11 +264,11 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
merely wedged kernel usually still answers ARP.
1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU):
- Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
- Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700):
- Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary`
- Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
2. Bench.
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -249,9 +286,17 @@ deletes the unit file from `/etc/systemd/system/`, not just the symlink.
The harness reaches the fleet through `.omp/agent/models.yml` in the
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
where the harness runs; there is no copy on the servers. Each provider is one
HOST:PORT; each model `id` is the served alias: `frank-primary` →
`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` →
`modelRoles`.
HOST:PORT; each model `id` is the served alias. **Official model ids are
the StarCraft unit names (2026-10-08) and since 2026-10-09 they are also the
ids on the wire:** `frank-archon` → `192.168.0.235:7770`
(`FRANK_PRIMARY_SERVED_NAME`), `frank-dragoon` → `192.168.0.143:7770`
(`HALOGEN_MODEL_ID`), `frank-zealot` → `192.168.0.143:7771` (`ALIAS`). The
SERVICES, units, scripts, env files and runbooks keep the old names
(`frank-primary`, `frank-large`, `frank-secondary`); only the served alias,
the `models.yml`/`config.yml` ids, the homepage cards, the OpenWebUI grants
and the Hermes routing were renamed. Verify a rename on `/v1/models`, never
on a chat response: radiance echoes the **requested** model name, so a 200
proves nothing about the served id (llama.cpp and halogen ignore it).
Resolution rules: the default model resolves as a `defaultProvider` +
`defaultModel` pair (the provider must serve the model); a subagent override
@@ -263,168 +308,273 @@ When renaming a service, retiring an endpoint, or moving a model: update the
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a provider the file does not define fails at call time. As
of 2026-09-26 both `frank-small` and `frank-primary` accept images
(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
(this build never autoloads projectors); `frank-large` remains text-only
because its stock llama.cpp deployment loads no
vision projector (the mmproj is not loaded). Note also that
the reasoning channel differs by runtime: llama.cpp endpoints emit
`reasoning_content`, vLLM emits `reasoning` — academic since the
2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
of 2026-10-06 **all three endpoints accept images** (`input: [text,
image]`) — `frank-large` returned to vision with the flash-server v2 swap
(the v2 vision tower loads beside the checkpoint; the text-only halogen
27B window closed 2026-10-06) — while
`frank-primary` takes them natively (radiance runs the checkpoint's
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
are the engine core; `--mm-max-patches` available),
and `frank-secondary` through
the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
Gemma4 era used). `frank-secondary`'s `contextWindow` is
**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
the harness must compact before llama.cpp answers a longer prompt with a 400.
Note also
that
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
models.yml sets `reasoningContentField` per endpoint to match.
There is **no audio**
input anywhere in the fleet except frank-small: that capability left with the
FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
projector's audio encoder (frank-small transcribes WAV verbatim even though
`audio` is omitted from its `input` list — OMP's schema rejects it).
input anywhere in the fleet: that capability left with the FLM NPU
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
(neither the MiMo nor the Qwen3.5 tower carries audio).
### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm
runtime — no host SDK gate) as a **root systemd unit**
(`scripts/frank-large.service`, `User=dank`), serving
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB,
sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full
262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec
decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4,
`SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and
**no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc
Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth
Qwen3.8-Flash-Next UD-IQ4_XS.
The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
run` FOREGROUND — the container's exit code is the service's exit so
`Restart=always` stays real, and the engine ledger lands on disk) on port
7770 from the Strix Halo iGPU. Engine stdout goes to
`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
Cutover 2026-10-06 from the halogen 27B-dense era
([[frank-large-halogen27b-dflash2]], retired the same window).
The spec method is DFlash2 (block-diffusion drafter) at depth 4. The
drafter is loaded via `--model-draft`; the chain is `--spec-type
draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the
unsloth b11030 build does not support fp8 KV; q8_0 is the
highest-precision 8-bit option, ~8 GiB at 262k). The serve script
(`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter
and exits with a pointer message if any is missing, then `exec`s
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
falls back to the built-in `peg-native` template, which produces a
~73,600-token prompt for a ~250-token request).
Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B
append-only original; the abliteration workflow appends and the repo's
tooling drops dead padding, live tensors byte-identical) + the **REQUIRED**
n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B,
peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn`
(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files,
fetched fresh from peonist). Cutover 2026-10-06 from the halogen
27B-dense shape, chosen by Frank for the ABLITERATED base.
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with
`--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers),
leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget
(`CACHE_RAM=8192`) is the qualified config.
**The format is the lock.** The v2 `.hgn` loads ONLY in the
flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the
engine moved with the model. The MTP head is INSIDE the checkpoint
(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn`
sidecar is deliberately not fetched).
Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
against its declared byte count + sha256; `--verify-only` re-checks).
**Product facts vs the 27B era, both regressions FIXED:** the base is
**abliterated** (the 27B was a censored-base finetune; refusals were an
accepted trade there) and **vision is ON** (the 27B engine silently
ignored its encoder — frank-secondary owned images for exactly one day at this
endpoint).
Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`.
Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime
intact on disk).
**Engine laws that shape the config** (all in `scripts/frank-large.env`
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
policy disables the MTP head for whole requests after low acceptance and
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
for it (accepted). `HALOGEN_CTX=262144` is the PER-SLOT hard ceiling (400
naming the limit, never truncation) — RAISED from 200192 on 2026-10-07
under the 400-zone law: the harness clamps max_tokens to
`min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts
the rendered prompt by ~3 % (measured: est 152,938 → server 157,388,
request 43,158 vs room 42,804), so a harness window == engine ceiling
made every request over ~146k est a guaranteed 400. `models.yml`
`contextWindow: 217088` leaves the 45k gap that absorbs the estimate
error (worst-case request ≈ 219k ≤ 262144); compaction fires at 85 % of
the harness window = 184,524 est ≈ 190k measured (Frank's 180k target).
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
law does NOT carry over; batching and speculation coexist (the card's
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
`HALOGEN_KV_POOL_POSITIONS=524288` (explicit 2 × ctx; ~28 KiB/position ⇒
14.0 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
longer greedy; `models.yml` extraBody carries the same values, consistent
not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm
100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual
`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported.
Thinking lands in **`reasoning_content`**; the v2 chat template rejects
`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold
stays mandatory. Thinking tokens COUNT against the budget (empty content +
`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI
`image_url` content parts only; ≥256 px probe images or the model answers
"video".
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
weights ~62 + KV 14.0 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
~2 ≈ 86 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~15 GiB
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
page cache beside the checkpoint — the card's quickstart arithmetic does
not bill it as resident, but READ the boot ledger and re-size if the
engine prints a clamp.
Mainline llama.cpp master `37b53fd` (CUDA build with
`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
(four FULL 262,144-token slots — the measured card ceiling), batch
**8192/1024**, greedy default. Deployed flags are in
`scripts/frank-small.env`; the serve script refuses to start if the
weights, the projector, the sidecar or the binary are missing.
Fetch: `scripts/download-frank-large-flashnext.sh` into
`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
sha256 + `--verify-only`, no token on the command line; resume goes
against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
The serve gates mirror the deployer EXACTLY (byte-exact checkpoint +
n-gram + vision tower, flat tokenizer with all six files as regular
files) — the 2026-10-01 crash-loop lesson.
Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
chat completion returns content plus `reasoning_content`; a tool call
returns `tool_calls`; an image request is answered from the projector;
an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
100 is WORSE at long ctx) is tabled in the runbook.
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]].
Runbook:
`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
Rollback: revert the swap commit and redeploy — the 27B payload
(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned
prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving
the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards)
with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI
PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at
**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE
n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0
template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork
`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from
llama.cpp b11030 the same day) at Frank's direction.
**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
(`scripts/frank-secondary.service`, `relink_unit`ed into
`/etc/systemd/system/`), serving
`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
LFS oid) **plus the publisher's own BF16 mmproj** (921,704,448 B), both
`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`, on port 7771. Cutover
2026-10-07 (late) from the Gemma4-26B-A4B Q4_K_M 1×200,704 shape (same day).
**The tuned shape lives on-host, not in the repo.** The q8 profile's setup
run qualified the `R9V_*` config into
`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env
file `scripts/frank-primary.env` carries ONLY production overrides
(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`,
`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`,
`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter
(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload
quantizes BOTH fused hyper-connection shards, so the shard-uniformity
check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the
overlay's env.always injects the uncensored 160-slot cache geometry, which
the q8 80-slot tiered cache rejects). Never copy tuned values from the
state into the repo — it would drift from the qualification.
**Architecture: qwen35 dense hybrid** — read from the GGUF's own header, not
the model card: 32 layers, `full_attention_interval = 4` ⇒ **8 full-attention
layers (3, 7, 11, …, 31) + 24 Gated DeltaNet linear layers**, 16 attention
heads / **4 KV heads** × 256 dim, `ssm` conv_kernel 4 / state_size 128 /
group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, vocab
248,320. Tensor types are Q6_K × 250 + F32 × 177 — standard quants only, so
the stock master build loads it (arch `qwen35` is registered in this build's
`src/llama-arch.cpp`). **KV accounting:** the pool scales with the 8 full
layers only — 8 × 4 × 256 = 8,192 elements/token per cache, so f16 K + f16 V
bills **32 KiB/token** = 12.25 GiB at the 401,408 pool; the 24 DeltaNet
layers bill a fixed recurrent state per SLOT, not per token, which is exactly
why two 200k slots fit where the 26B MoE fit one.
**Spec decoding is MTP4 only.** The fork's `speculative_config` is a
single dict — one method per engine, NO chaining — so the llama.cpp
`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired
(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along
architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative
single method, not a chainable one.
**The shape (2026-10-07):** `--ctx-size 401408 --parallel 2` = TWO
conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes
the per-slot number), `--flash-attn on`, **f16 K + f16 V** (exact KV),
batch 4096 / **ubatch 2048** (the measured CUDA optimum: +4.6 % prefill,
decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`,
`--mmproj` for vision, **speculative decoding OFF** (`SPEC_TYPE=none`).
**Serving = the fork's own launcher through a docker→podman shim.** The R9V
tooling is docker-pinned; the shim at
`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries
three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys
the image by its OCI manifest digest while podman keys by config digest but
preserves the manifest digest in `RepoDigests` — the shim answers
`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run
refs to `repo@sha256:X`; (2) podman inspect answers
`HostConfig.LogConfig.Config: null` where docker returns the log-opts map
(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs
and normalizes inspect output; (3) podman lacks docker's `SecurityOptions`
/`ServerVersion` info fields — the shim answers them.
**Why MTP is off**: it is not a choice — **this Q6_K file carries no
`nextn`/`mtp` tensors** (verified from the tensor-name list), and MTP is a
property of the file, not of the flags. The publisher card advertises "MTP
support" for the base architecture; the quantized export dropped the head.
**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is
54.4 GB, and a root unit grants the pin budget via PID1 (a rootless
quadlet would be capped by the user manager's limit).
The serve script (`scripts/serve-frank-primary.sh`) gates the setup state
(ready + paths), the pinned image (RepoDigests), the PLE byte count
(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes,
clears a stale container, then execs the fork's `scripts/launch.sh` through
the shim and **blocks on `docker wait`** — the launcher `--detach`es, so
without the wait a crashed engine is a green unit; `docker wait` cannot stop
the container child, hence `ExecStop=podman stop -t 120
frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/
PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/
restarts the unit and health-gates with the MainPID check (the false-green
stray law).
**Measured boots** (nvidia-smi, the referee): the deployed boot is
**21,436 used / 2,689 free** (ubatch 2048) — the pricing boots were the
MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry (**21,276 / 2,849** at
ubatch 1024, **21,846 / 2,279** at 2048), so the 415 MiB lighter weights
bought back what the projection predicted. The engine ledger confirms the KV
law to the MiB: `CUDA0 KV buffer size = 12544.00 MiB` (`K (f16): 6272.00`,
`V (f16): 6272.00`), compute buffers 1,136 MiB on CUDA0 + 848 MiB host, the
mmproj 879 MiB + 248 MiB. Decode is ubatch-invariant (Swift-era 3090 law). The
Q6_K rung costs ~10 % decode vs Q5_K_M (weight bandwidth; MiMo-measured 94.4
vs 104.8 t/s).
Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20
payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up
battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance
length 3.19–5.00, and the PLE pinned proof (`mapping_bytes =
mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool
round-trip + template probe re-verified on :7770 after the deploy. No
throughput bench was run (Frank declined); the probe-era 11 tok/s decode
was residency-bound (halved expert headroom at double the bytes/expert),
not the production number.
**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is
natively multimodal; the Q6_K file carries no vision tensors). Probe traps
carry over: ≥256 px probe images or the model answers "video", and only the
OpenAI `image_url` content part reaches the model.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`.
Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to
`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
and the template keys, redeploy (state + `restore-config-20260927.env` +
payload intact on disk) — see
[[frank-primary-r9v-radiance-qwen38-uncensored]].
**Compaction hub**: `models.yml` routes `compactionModel` for
`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here — 200,704 is the
only window in the fleet that fits a worst-case compaction input (~170k
tokens). With `--parallel 2` a compaction request no longer serializes behind
a live generation (the Gemma4 era's `--parallel 1` law).
Runbook:
`serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md`.
Rollback: revert the swap commit — the Gemma4 payload
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
(`/home/dank/models/MiMo-Q6K/`), the Swift payload
(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
### frank-primary — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
The **radiance engine** (StillDeadcode, Apache-2.0; image
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
for serve/convert/KLD; `libr4d` gfx1201 kernels, plain `podman run`, root
systemd unit, container name = unit name) serving
**ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB, 131 shards,
397/397 sha256-verified)** converted by `scripts/convert-swift15-w4dn8.sh`
with recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` to
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ
w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine
**1.2.0** via `RADIANCE_CALIB_DIR`/`RADIANCE_CALIB_TOKENS` — the only pin
with the token knob; the RADGRAM2 grams are format-identical across pins and
cross the boundary by design), **24 whole experts per layer cast bf16** (the
PROTECTED set, derived from the calibration grams' energy share),
**int8 trunk/lm_head**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE n-gram
table at fp8 fixed scale (47.7 GiB in the container, from 95.4 GiB bf16).
The plugin's **one-table-one-encoding law** (measured at rad-convert
declare): one expert table takes ONE encoding; only whole bf16 experts are
legal beside it — no w4/fp8 split of the down/gate_up pair (24-value
`rad_arch_quant_desc` array, one per layer). KLD gate vs the BF16
reference (recorded on the same 1.0.13 pin) is the precision referee;
the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
(122–173 t/s at 44.5k depth). Measured decode/prefill in the runbook.
Rollback: revert the swap commit — the fp8-native `.rad`
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
`--max-model-len 200000` (the harness contextWindow must quote this),
`--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM
partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (the vram-pair
replaced `--expert-vs-cache-ratio`; the pair sizes the fp8-KV shape at the
deployed boot), `--kv-cache-dtype fp8` (inherited from the fp8-native
env), `--num-speculative-tokens 2`,
`--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
`--override-chat-template`). Boot ledger: KV pool 3.46 GiB (~480k fp8
tokens), two expert pools 57.84 GiB, cards settled 1.41 GiB free.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
prefix cache, and the rule that a prompt over `--max-model-len` is
refused, never truncated. What changed: the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
40 GiB pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the deployed shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246); production shipped the fp8-KV + spec-2 pair at cutover —
the KLD gate scored at the fp8-KV shape, so the gate and the boot agree.
Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
buffers + arena onto the weights side, so `weights + kv` is capped near
29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
the card reads) + 4 GiB prefix host ≈ **132 GiB** charged against
`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
start until both knobs are raised in the stop window (sysfs write first,
kernel cmdline + reboot only if read-only).
**Contract facts** (source-verified): the HTTP listener binds AFTER the
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
200 serving / 503 stopping); sampling params temperature…preserve_thinking
accepted, **unknown params 400** (models.yml's extraBody set is fully
inside it); `response_format` supported; `logprobs` REFUSED; reasoning in
`reasoning_content` and tool calls in the qwen3.8 dialect via the auto
chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`.
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
@@ -590,15 +740,56 @@ linked runbooks.
the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check
`swapon --show` before reasoning about headroom.
- **Silent RAM corruption outranks every software theory.** frank-primary's
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
proof — combined, those two instruments said "ruled out" while the RAM
was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
not a margin guarantee. When the fault is data-shaped (NaN states,
garbage tokens) and a software revert doesn't move it, price in the RAM
before the runtime — and remember a revert tested under the same marginal
hardware is itself confounded. See the runbook's 2026-10-05 section.
**Runtime-specific guards**
- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
dies at container create, not at runtime.** runc reports
`error setting rlimit type 7: operation not permitted` (type 7 =
RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
are fine, so `memlock=-1:-1` works iff the unit grants
`LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
podman to runc without any raise). Upstream halogen's own run lines pass
NO `--ulimit` at all — copy vendor run lines before adding flags
(measured 2026-10-05 on the frank-large cutover).
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
`third_party/composable_kernel/include/ck/config.h` generated manually
(untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
non-greedy requests and reasoning budgets. Re-test chat before production.
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
stock binaries fail the draft load with a tensor-shape mismatch.
- **A GGUF's custom tensor type picks the runtime — read the histogram before
promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
the only door. Counting tensor types out of the GGUF KV section takes one
python read and settles a runtime argument in minutes. The build identity
matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
`ggml_compute_forward_set_rows` on this same file, so
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
`--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
`--spec-type`, or it measures a crash instead of the absence of drafting.
- **An orphaned scratch server taxes the whole host.** A bench child left
alive by a client-side exception kept ~26 GiB of weights resident on a
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
Record the child pid and kill it in a `finally`, then prove `free` came back
before believing a number. Kill by pid — never `pkill -f llama-server`,
which also hits frank-secondary.
**llama.cpp serving**
@@ -608,19 +799,26 @@ linked runbooks.
the MAIN model's weights (`shares_model`), so there is no second copy of a
12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
tensors at all, and pointing `--model-draft` at the file would load it twice.
- **What can be chained, and what cannot.** `--spec-type a,b` builds every
enabled implementation and chains drafting through them, in a fixed internal
order (ngram types first, then the drafters — CLI order is irrelevant). The
gate is `ctx_dft`: **draftless** types (the ngram family) are added
unconditionally, while `draft-mtp`, `draft-dflash`, `draft-dspark` and
`draft-eagle3` all require a draft CONTEXT. So a chain may hold one
context-backed drafter plus any number of ngram types — `draft-dflash,ngram-mod`
is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte-
identical to it on prose at the fleet's 48/64 window) — but
`draft-mtp,draft-dflash` dies at load: *"context type MTP requested but model
doesn't contain MTP layers"*, because MTP builds its context against whatever
model the draft context holds. Two context-backed types cannot share one
process, and MTP's head only loads when MTP **is** the draft model.
- **What can be chained, what cannot, and what a chain costs on a hybrid.**
`--spec-type a,b` builds every enabled implementation and chains drafting
through them, in a fixed internal order (ngram types first, then the
drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
types (the ngram family) are added unconditionally, while `draft-mtp`,
`draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
CONTEXT. So a chain may hold one context-backed drafter plus any number of
ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
drafter alone on prompt-echo traffic, byte-identical to it on prose at the
fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
type MTP requested but model doesn't contain MTP layers"*, because MTP
builds its context against whatever model the draft context holds. Two
context-backed types cannot share one process, and MTP's head only loads
when MTP **is** the draft model. **A LEGAL chain can still lose on a
recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
logs `recurrent state rollback is not compatible with 'ngram-mod' -
disabling rollback support`, and MTP verification depends on that rollback,
so acceptance drops on ALL traffic even where the matcher fires. Measure
chains per architecture, never by analogy.
- **A drafter's KV scales with the TARGET's context, and its window policy
decides how much that costs.** The draft context is built with
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
@@ -637,7 +835,8 @@ linked runbooks.
trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
36:29), which is the practical size of the effect. See
[[frank-medium-llamacpp-dflash2]].
- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
compute buffers land about TWICE as heavy on the card that holds the output
layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
shape (262144 over two slots, a 64k prefill measured at 64k depth,
@@ -655,6 +854,19 @@ linked runbooks.
its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
model — the sweep's or the seam's — can see it.)
- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
279-token prompt, which is below either ubatch, came back identical — that is
the mechanical proof the knob was the only variable, since a prompt shorter
than the ubatch cannot see it. The logical `--batch-size` was again free. And
the predecessor's "ubatch 2048 failed the scratch boot" was a shape
observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE`
drop-in, never a scratch port — one card holds one server. See
[[frank-secondary-llama-mimo-q6k-2x200704]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
@@ -671,6 +883,41 @@ linked runbooks.
host-RAM hash table) and its window decides which traffic it can see: too
short and it proposes on ordinary prose fragments, which costs verification
without acceptance.
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
(`400 exceed_context_size_error: request (184001 tokens) exceeds the
available context size (180224 tokens)`) even when the pool has room. So
frank-secondary's 2 → 3 reslice moved each conversation from the model's native
262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
the harness would discover the ceiling as a 400 on a long conversation. Side
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704,
pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again;
decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same
day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
itself is a further measured −10 %, weight-bandwidth, not slot geometry).
All moves measured; both slot directions cost or buy window, never pool.
See [[frank-secondary-llama-mimo-q6k-2x200704]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
**non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
that bills ~1.4 GiB, because the encoder's activation and compute buffers
ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
solve `KV ≤ card − non-KV − headroom` and round to a multiple of
`PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize
starts from measurement.
- **`images` and `image_url` are NOT interchangeable: only the content part
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
model still invents an answer if the client used the legacy top-level
`images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
failure is silent because a hallucinated color is a plausible answer. Probe
vision with a solid color and expect an exact word.
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
then served frank-large: every tool-using agent on that endpoint was broken
@@ -679,7 +926,7 @@ linked runbooks.
returns `tool_calls`. Check tool support before routing a role at an endpoint.
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
option and the same ~12x prefill collapse described in the ROCm note above.
f16 KV needs nothing; frank-small ran f16 for that reason until
f16 KV needs nothing; frank-secondary ran f16 for that reason until
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
endpoint could take q8_0 KV.
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
@@ -697,7 +944,7 @@ linked runbooks.
raises on anything else — HTTP 500, not a clamp — so the harness's levels
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
A role whose level is unmapped fails on its FIRST request, and the failure
names Jinja, not the model: every subagent on `frank-small` died with
names Jinja, not the model: every subagent on `frank-secondary` died with
`Unexpected reasoning effort high` until `high: xhigh` was added on
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
differ.
@@ -771,9 +1018,9 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -781,6 +1028,21 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-secondary`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-secondary-llama-gemma4-q4km-200704.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` |
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-secondary`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` |
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` |
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
@@ -791,15 +1053,16 @@ Retired (rollback targets and history):
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` |
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
| FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
| llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
+1 -1
View File
@@ -13,7 +13,7 @@ one deploy script.
| Host | User | IP | Podman stack | LLM service | LLM port |
|------|------|----|--------------|-------------|----------|
| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
## Active Services
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
# autoresearch.sh — canonical frank-primary benchmark entry point (dev box).
#
# The service runs on dankserver (192.168.0.235:7770), the two R9700s live
# there, and the benchmark must own those cards, so the whole stop -> boot
# scratch -> bench -> restore sequence runs ON dankserver (see
# scripts/autoresearch-frank-primary.sh). This wrapper:
# 1. copies the candidate config (the repo's scripts/frank-primary.env, the
# file the autoresearch loop edits) and the bench script to dankserver,
# 2. runs the bench over ssh, relaying its stdout (the METRIC lines) and
# exit code back here verbatim.
# It deliberately does NOT push to origin: the scp dirties dankserver's
# working tree, and the loop's uncommitted candidate config must not be
# pushed. Only this file and scripts/autoresearch-frank-primary.sh are
# committed; the candidate env rides over ssh each run.
set -uo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REMOTE=dankserver
REMOTE_REPO=/home/dank/serverAppConfigs
scp -q "$DIR/scripts/frank-primary.env" \
"$REMOTE:$REMOTE_REPO/scripts/frank-primary.env" || { echo "scp env failed" >&2; exit 1; }
scp -q "$DIR/scripts/autoresearch-frank-primary.sh" \
"$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-primary.sh" || { echo "scp script failed" >&2; exit 1; }
ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-primary.sh $REMOTE_REPO/scripts/frank-primary.env"
+10 -1
View File
@@ -337,7 +337,16 @@ http://:8017 {
try_files /docs.html
file_server
}
@data path /services.json /status.json /app.js /style.css /deployed.js
# /whoami echoes the outpost-injected identity headers so the page
# can gate the admin button client-side (the /admin* 403 above stays
# the real gate; this only decides what is shown). The outpost strips
# client-supplied X-Authentik-* before injecting, so this is spoof-proof.
handle /whoami {
templates
header Content-Type application/json
respond "{\"user\":\"{header.X-Authentik-Username}\",\"groups\":\"{header.X-Authentik-Groups}\"}"
}
@data path /services.json /status.json /app.js /style.css /deployed.js /img/* /sfx/*
handle @data {
file_server
}
@@ -0,0 +1,64 @@
{
"3": {
"class_type": "CLIPTextEncode",
"_meta": { "title": "Positive prompt" },
"inputs": {
"text": "A detailed illustration of a 1930s Soviet industrial design bureau at night: a well-measured engineer at a drafting table, a warm desk lamp, rolled blueprints, gears and calipers on the desk, deep red and cream palette, gouache painterly style, intricate mechanical detail, soft shadows",
"clip": [ "6", 0 ]
}
},
"4": {
"class_type": "CLIPTextEncode",
"_meta": { "title": "Negative prompt" },
"inputs": {
"text": "lowres, blurry, deformed, extra limbs, watermark, signature, jpeg artifacts",
"clip": [ "6", 0 ]
}
},
"5": {
"class_type": "UNETLoader",
"_meta": { "title": "Qwen Image 2.1 (int8)" },
"inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
},
"6": {
"class_type": "CLIPLoader",
"_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
"inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
},
"7": {
"class_type": "VAELoader",
"_meta": { "title": "VAE" },
"inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
},
"8": {
"class_type": "EmptySD3LatentImage",
"_meta": { "title": "Empty Latent 1024x1024" },
"inputs": { "width": 1024, "height": 1024, "batch_size": 1 }
},
"9": {
"class_type": "KSampler",
"_meta": { "title": "KSampler" },
"inputs": {
"model": [ "5", 0 ],
"seed": 1930,
"steps": 20,
"cfg": 1.0,
"sampler_name": "euler",
"scheduler": "simple",
"positive": [ "3", 0 ],
"negative": [ "4", 0 ],
"latent_image": [ "8", 0 ],
"denoise": 1.0
}
},
"10": {
"class_type": "VAEDecode",
"_meta": { "title": "VAE Decode" },
"inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
},
"11": {
"class_type": "SaveImage",
"_meta": { "title": "Save Image (art)" },
"inputs": { "filename_prefix": "qwen21-art", "images": [ "10", 0 ] }
}
}
@@ -0,0 +1,64 @@
{
"3": {
"class_type": "CLIPTextEncode",
"_meta": { "title": "Positive prompt (legible text)" },
"inputs": {
"text": "Clean minimalist movie-poster design, dark charcoal background with a subtle vintage grain. Centered large bold white sans-serif text reading \"SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS\", perfectly legible, crisp letterforms, even letter spacing. Below it, a smaller line of yellow text reading \"DANKCLAW DANKSERVER EST 2026\", equally legible. Flat graphic design, high contrast, generous margins, no other words, no watermark",
"clip": [ "6", 0 ]
}
},
"4": {
"class_type": "CLIPTextEncode",
"_meta": { "title": "Negative prompt" },
"inputs": {
"text": "blurry, illegible, misspelled, warped letters, extra text, watermark, signature, lowres, jpeg artifacts",
"clip": [ "6", 0 ]
}
},
"5": {
"class_type": "UNETLoader",
"_meta": { "title": "Qwen Image 2.1 (int8)" },
"inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
},
"6": {
"class_type": "CLIPLoader",
"_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
"inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
},
"7": {
"class_type": "VAELoader",
"_meta": { "title": "VAE" },
"inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
},
"8": {
"class_type": "EmptySD3LatentImage",
"_meta": { "title": "Empty Latent 832x1216 poster" },
"inputs": { "width": 832, "height": 1216, "batch_size": 1 }
},
"9": {
"class_type": "KSampler",
"_meta": { "title": "KSampler" },
"inputs": {
"model": [ "5", 0 ],
"seed": 2026,
"steps": 25,
"cfg": 1.0,
"sampler_name": "euler",
"scheduler": "simple",
"positive": [ "3", 0 ],
"negative": [ "4", 0 ],
"latent_image": [ "8", 0 ],
"denoise": 1.0
}
},
"10": {
"class_type": "VAEDecode",
"_meta": { "title": "VAE Decode" },
"inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
},
"11": {
"class_type": "SaveImage",
"_meta": { "title": "Save Image (text poster)" },
"inputs": { "filename_prefix": "qwen21-text", "images": [ "10", 0 ] }
}
}
+6 -6
View File
@@ -10,15 +10,15 @@ model:
provider: dankclaw-large
base_url: http://192.168.0.143:7770/v1
api_key: not-needed
default: frank-large
default: frank-dragoon
providers:
dankclaw-large:
api: http://192.168.0.143:7770/v1
discover_models: false
models:
frank-large:
context_length: 114688
frank-dragoon:
context_length: 262144
dankserver-small:
api: http://192.168.0.235:7770/v1
discover_models: false
@@ -29,7 +29,7 @@ providers:
# Auxiliary routing from the live configuration. Named providers route via
# their `api:` URL; `custom` entries route by their explicit `base_url`.
auxiliary:
vision: {provider: dankclaw-large, model: frank-large}
vision: {provider: dankclaw-large, model: frank-dragoon}
web_extract: {provider: dankserver-small, model: frank-medium}
compression: {provider: dankserver-small, model: frank-medium}
skills_hub: {provider: dankserver-small, model: frank-medium}
@@ -50,8 +50,8 @@ auxiliary:
effort: none
memory_query_rewrite: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
tts_audio_tags: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
triage_specifier: {provider: dankclaw-large, model: frank-large}
kanban_decomposer: {provider: dankclaw-large, model: frank-large}
triage_specifier: {provider: dankclaw-large, model: frank-dragoon}
kanban_decomposer: {provider: dankclaw-large, model: frank-dragoon}
profile_describer: {provider: dankserver-small, model: frank-medium}
curator: {provider: dankserver-small, model: frank-medium}
+10
View File
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env bash
# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
# speculative-decode arm (vLLM --speculative-config method=dflash). Public
# repo — no auth header on the command line.
#
# The drafter is a 6-layer qwen3 draft whose dflash_config pins
# target_layer_ids [1,5,9,13,17,21,25,29] and block_size 16; its hidden_size
# 4096 / vocab_size 248320 match the MiMo-V2.6-Distill-Qwen-9B target exactly
# (both preserve the Qwen3.5-9B geometry), which is what lets vLLM load it as
# the draft for that target.
#
# Usage:
# scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
set -euo pipefail
REPO="z-lab/Qwen3.5-9B-DFlash"
BASE="https://huggingface.co/${REPO}/resolve/main"
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/Qwen3.5-9B-DFlash}"
# name bytes sha256(- = non-LFS, size-only check)
FILES=(
"model.safetensors 2583816465 0a42274b32554f48de1faa0d42824e9c2ceda649c30ae0a731cddf410dd698c7"
"config.json 1222 -"
)
mkdir -p "$DEST"
status=0
for entry in "${FILES[@]}"; do
read -r name size sha <<<"$entry"
f="$DEST/$name"
have=0
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
have=1
fi
fi
if (( have )); then
echo "OK $name"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $name"
status=1
continue
fi
echo "FETCHING $name"
tmp="$f.part"
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $name"
status=1
continue
fi
echo "VERIFIED $name"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env bash
# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
# command line (a Bearer token in argv is world-readable via /proc).
#
# Per-file FRESH downloads, no resume: the house rule (see
# download-frank-secondary.sh); -f/--fail-with-body plus size+sha256 against the
# Hub LFS oid is what makes an access error fail instead of "succeeding".
#
# Usage:
# scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
#
# The verified set below is exactly what the 2026-10-01 bench window served:
# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
set -euo pipefail
REPO="XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B"
BASE="https://huggingface.co/${REPO}/resolve/main"
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16}"
# name bytes sha256(- = non-LFS, size-only check)
FILES=(
"model-00001-of-00004.safetensors 5276436216 aab052180118aee34abc3029b54eaa49096aac606b97d703866b420dceb703c3"
"model-00002-of-00004.safetensors 5033171280 7a0486565f06d25ac4628e9dba470dc3f604353471d240d5a0bf7128f64df396"
"model-00003-of-00004.safetensors 5234499504 6c73207563d1879bfd6c143a028cc70be68edff56280458c71a43df4240300f4"
"model-00004-of-00004.safetensors 3275613848 1379a7cf8c0b8555a39ab65a47e830e0eb45e776b46045734da3afd73c09eea2"
"model.safetensors.index.json 69217 -"
"config.json 2784 -"
"generation_config.json 126 -"
"chat_template.jinja 3916 -"
"tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
"tokenizer_config.json 1165 -"
"vocab.json 6722759 -"
"merges.txt 3353259 -"
"preprocessor_config.json 443 -"
"processor_config.json 1191 -"
"video_preprocessor_config.json 385 -"
)
mkdir -p "$DEST"
status=0
for entry in "${FILES[@]}"; do
read -r name size sha <<<"$entry"
f="$DEST/$name"
have=0
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
have=1
fi
fi
if (( have )); then
echo "OK $name"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $name"
status=1
continue
fi
echo "FETCHING $name"
tmp="$f.part"
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $name"
status=1
continue
fi
echo "VERIFIED $name"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+85
View File
@@ -0,0 +1,85 @@
#!/usr/bin/env bash
# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
# mtp tensors; that is why SGLang's NEXTN on frank-secondary draws 0.00
# acceptance). Public repo — no auth header on the command line.
#
# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
#
# Usage:
# scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
set -euo pipefail
REPO="Qwen/Qwen3.5-9B"
BASE="https://huggingface.co/${REPO}/resolve/main"
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/Qwen3.5-9B}"
# name bytes sha256(- = non-LFS, size-only check)
FILES=(
"model.safetensors-00001-of-00004.safetensors 5276436216 db6f444b43d318c92f360a13a25561a6a65b10c0631b8ed305a426dbaa6c380e"
"model.safetensors-00002-of-00004.safetensors 5335161512 31c7d7e2dd5d207840b31cc59083c8f4c4718959149e0358c0364052bb9a0330"
"model.safetensors-00003-of-00004.safetensors 5368717440 7ec36ba3a4176a44c3c0876ad80c56a2f70c84bf008d82e9501df642f17dadec"
"model.safetensors-00004-of-00004.safetensors 3325995712 b62b0c4cd7e44edee103ee8f4fe225f246d5e768e07bfd5f25b63a8aa1fdd0c6"
"model.safetensors.index.json 79657 -"
"config.json 3126 -"
"chat_template.jinja 7756 -"
"tokenizer.json 12807982 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42"
"tokenizer_config.json 16710 -"
"vocab.json 6722759 -"
"merges.txt 3353259 -"
"preprocessor_config.json 390 -"
"video_preprocessor_config.json 385 -"
)
mkdir -p "$DEST"
status=0
for entry in "${FILES[@]}"; do
read -r name size sha <<<"$entry"
f="$DEST/$name"
have=0
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
have=1
fi
fi
if (( have )); then
echo "OK $name"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $name"
status=1
continue
fi
echo "FETCHING $name"
tmp="$f.part"
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $name"
status=1
continue
fi
echo "VERIFIED $name"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-secondary
# (SGLang, RTX 3090). Public repo — no auth header on the command line.
#
# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
# the house rule.
#
# Usage:
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
#
# LFS shards are verified against the Hub LFS sha256; non-LFS text files
# against their declared byte size. Idempotent: complete files are skipped.
set -euo pipefail
REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
BASE="https://huggingface.co/${REPO}/resolve/main"
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
# name bytes sha256(- = non-LFS, size-only check)
FILES=(
"model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
"model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
"model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
"model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
"model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
"model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
"model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
"model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
"tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
"model.safetensors.index.json 106278 -"
"config.json 4608 -"
"generation_config.json 161 -"
"quantization_config.json 1654 -"
"chat_template.jinja 3916 -"
"tokenizer_config.json 1168 -"
"preprocessor_config.json 443 -"
"processor_config.json 1191 -"
"README.md 8448 -"
)
mkdir -p "$DEST"
status=0
for entry in "${FILES[@]}"; do
read -r name size sha <<<"$entry"
f="$DEST/$name"
have=0
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
have=1
fi
fi
if (( have )); then
echo "OK $name"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $name"
status=1
continue
fi
echo "FETCHING $name"
tmp="$f.part"
# no -C resume; -f + --fail-with-body so an HTTP error is not a success
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $name"
status=1
continue
fi
echo "VERIFIED $name"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+86
View File
@@ -0,0 +1,86 @@
# Subsurface Aquifer Monitoring Network - System Design Document (Revision 7)
This document describes the deployment of a subsurface aquifer monitoring network for the Cascadia sedimentary basin. It is written for the operations crew and the regional water authority. Every sensor node, relay, aggregation service, calibration procedure, and failure mode below carries a stable alphanumeric identifier that appears in every table and in the field notes.
## Sensor nodes
Node SN-000-08B9 sits in a discharge lens at 359.0 m below datum and reports a salinity of 2.38 ppt with the fractured interval reading 5.40 degC at dawn and the wellhead porous 1.72 m3/d while the electrode pair returns 3157 uS/cm.
Node SN-001-0A65 sits in a percolate lens at 25.9 m below datum and steps down to a salinity of 2.13 ppt with the diapir interval reading 8.55 degC at dawn and the wellhead silty 0.67 m3/d while the electrode pair returns 979 uS/cm.
Node SN-002-0571 sits in a diapir lens at 287.8 m below datum and peaks at a salinity of 7.63 ppt with the karst interval reading 4.26 degC at dawn and the wellhead spooned 11.50 m3/d while the electrode pair returns 2987 uS/cm.
Node SN-003-0CE9 sits in a confining lens at 169.5 m below datum and logs a salinity of 0.96 ppt with the alluvial interval reading 5.03 degC at dawn and the wellhead salinized 12.08 m3/d while the electrode pair returns 2607 uS/cm.
Node SN-004-0E2A sits in a freshened lens at 70.4 m below datum and settles near a salinity of 8.68 ppt with the spooned interval reading 4.88 degC at dawn and the wellhead siphon 4.38 m3/d while the electrode pair returns 2707 uS/cm.
Node SN-005-0DDD sits in a percolate lens at 44.6 m below datum and reports a salinity of 6.25 ppt with the alluvial interval reading 10.85 degC at dawn and the wellhead brackish 13.99 m3/d while the electrode pair returns 3618 uS/cm.
Node SN-006-0EFC sits in a till lens at 309.5 m below datum and reports a salinity of 3.49 ppt with the lacustrine interval reading 7.38 degC at dawn and the wellhead spooned 3.21 m3/d while the electrode pair returns 1253 uS/cm.
Node SN-007-0B02 sits in a recharge lens at 262.0 m below datum and steps down to a salinity of 2.38 ppt with the discharge interval reading 8.18 degC at dawn and the wellhead perched 4.05 m3/d while the electrode pair returns 3900 uS/cm.
Node SN-008-0448 sits in a karst lens at 406.5 m below datum and reports a salinity of 7.32 ppt with the upconing interval reading 6.17 degC at dawn and the wellhead fluvial 0.75 m3/d while the electrode pair returns 1448 uS/cm.
Node SN-009-043C sits in a alluvial lens at 159.3 m below datum and flags a salinity of 6.20 ppt with the percolate interval reading 7.60 degC at dawn and the wellhead brackish 13.01 m3/d while the electrode pair returns 2025 uS/cm.
Node SN-010-0FF5 sits in a karst lens at 264.2 m below datum and flags a salinity of 7.05 ppt with the freshened interval reading 7.88 degC at dawn and the wellhead siphon 8.41 m3/d while the electrode pair returns 1635 uS/cm.
Node SN-011-098A sits in a drain lens at 365.7 m below datum and reports a salinity of 8.11 ppt with the wedge interval reading 5.51 degC at dawn and the wellhead glacial 2.52 m3/d while the electrode pair returns 2916 uS/cm.
Node SN-012-0AE8 sits in a till lens at 259.6 m below datum and steps down to a salinity of 9.13 ppt with the recharge interval reading 11.60 degC at dawn and the wellhead spooned 0.46 m3/d while the electrode pair returns 3077 uS/cm.
Node SN-013-0B83 sits in a salinized lens at 171.2 m below datum and peaks at a salinity of 2.83 ppt with the lacustrine interval reading 5.56 degC at dawn and the wellhead discharge 0.35 m3/d while the electrode pair returns 3083 uS/cm.
Node SN-014-032E sits in a drain lens at 249.6 m below datum and flags a salinity of 1.09 ppt with the salinized interval reading 9.58 degC at dawn and the wellhead perched 12.00 m3/d while the electrode pair returns 2221 uS/cm.
Node SN-015-004F sits in a aeolian lens at 264.4 m below datum and reports a salinity of 7.34 ppt with the salinized interval reading 8.76 degC at dawn and the wellhead perched 0.31 m3/d while the electrode pair returns 1483 uS/cm.
Node SN-016-07C6 sits in a aeolian lens at 155.9 m below datum and reports a salinity of 0.64 ppt with the mineralized interval reading 11.75 degC at dawn and the wellhead diluted 13.46 m3/d while the electrode pair returns 524 uS/cm.
Node SN-017-06C5 sits in a salinized lens at 72.1 m below datum and averages a salinity of 4.52 ppt with the siphon interval reading 8.93 degC at dawn and the wellhead confining 3.98 m3/d while the electrode pair returns 3687 uS/cm.
Node SN-018-0E64 sits in a siphon lens at 334.9 m below datum and steps down to a salinity of 6.73 ppt with the salinized interval reading 7.63 degC at dawn and the wellhead freshened 9.64 m3/d while the electrode pair returns 1681 uS/cm.
Node SN-019-0122 sits in a spooned lens at 117.2 m below datum and logs a salinity of 3.24 ppt with the fractured interval reading 9.26 degC at dawn and the wellhead karst 3.50 m3/d while the electrode pair returns 1065 uS/cm.
Node SN-020-021D sits in a karst lens at 414.3 m below datum and recovers to a salinity of 0.76 ppt with the glacial interval reading 6.25 degC at dawn and the wellhead silty 9.60 m3/d while the electrode pair returns 1041 uS/cm.
Node SN-021-06C6 sits in a percolate lens at 379.2 m below datum and drifts to a salinity of 7.61 ppt with the wedge interval reading 5.84 degC at dawn and the wellhead karst 1.65 m3/d while the electrode pair returns 1913 uS/cm.
Node SN-022-0DC9 sits in a spooned lens at 327.1 m below datum and reports a salinity of 9.25 ppt with the freshened interval reading 5.05 degC at dawn and the wellhead silty 5.90 m3/d while the electrode pair returns 1544 uS/cm.
Node SN-023-0717 sits in a karst lens at 222.0 m below datum and steps down to a salinity of 4.02 ppt with the alluvial interval reading 6.60 degC at dawn and the wellhead siphon 3.77 m3/d while the electrode pair returns 3892 uS/cm.
Node SN-024-02A9 sits in a fractured lens at 479.7 m below datum and drifts to a salinity of 7.88 ppt with the silty interval reading 12.53 degC at dawn and the wellhead discharge 13.18 m3/d while the electrode pair returns 3592 uS/cm.
Node SN-025-0C8F sits in a wedge lens at 434.3 m below datum and holds a salinity of 1.63 ppt with the alluvial interval reading 4.22 degC at dawn and the wellhead till 5.73 m3/d while the electrode pair returns 3905 uS/cm.
Node SN-026-04BF sits in a colluvial lens at 460.0 m below datum and averages a salinity of 9.36 ppt with the diapir interval reading 8.98 degC at dawn and the wellhead brackish 10.29 m3/d while the electrode pair returns 802 uS/cm.
Node SN-027-0DA4 sits in a silty lens at 265.7 m below datum and steps down to a salinity of 7.06 ppt with the percolate interval reading 4.69 degC at dawn and the wellhead freshened 8.42 m3/d while the electrode pair returns 2201 uS/cm.
Node SN-028-01EB sits in a perched lens at 49.5 m below datum and recovers to a salinity of 1.83 ppt with the silty interval reading 9.32 degC at dawn and the wellhead drain 9.69 m3/d while the electrode pair returns 1126 uS/cm.
Node SN-029-042C sits in a karst lens at 30.6 m below datum and averages a salinity of 0.86 ppt with the percolate interval reading 9.85 degC at dawn and the wellhead recharge 8.16 m3/d while the electrode pair returns 1452 uS/cm.
## Relays
Relay RL-00-02181 reburses the upconing trunk from the brackish margin and peaks at a round-trip time of 131.8 ms with a frame loss of 1.04 percent across the night window.
Relay RL-01-00262 reburses the spooned trunk from the fluvial margin and holds a round-trip time of 372.2 ms with a frame loss of 3.65 percent across the night window.
Relay RL-02-03691 reburses the recharge trunk from the percolate margin and steps down to a round-trip time of 398.5 ms with a frame loss of 0.29 percent across the night window.
Relay RL-03-0E117 reburses the lacustrine trunk from the confining margin and averages a round-trip time of 373.9 ms with a frame loss of 3.44 percent across the night window.
Relay RL-04-0B41B reburses the aeolian trunk from the colluvial margin and peaks at a round-trip time of 70.7 ms with a frame loss of 3.25 percent across the night window.
Relay RL-05-0D12D reburses the recharge trunk from the diluted margin and steps down to a round-trip time of 264.7 ms with a frame loss of 0.03 percent across the night window.
Relay RL-06-043B4 reburses the colluvial trunk from the upconing margin and reports a round-trip time of 49.5 ms with a frame loss of 3.42 percent across the night window.
Relay RL-07-09AD6 reburses the fractured trunk from the freshened margin and averages a round-trip time of 225.3 ms with a frame loss of 1.06 percent across the night window.
Relay RL-08-04394 reburses the brackish trunk from the fluvial margin and steps down to a round-trip time of 88.6 ms with a frame loss of 2.47 percent across the night window.
Relay RL-09-017A0 reburses the wedge trunk from the lacustrine margin and drifts to a round-trip time of 363.0 ms with a frame loss of 3.30 percent across the night window.
Relay RL-10-0217D reburses the mineralized trunk from the lacustrine margin and peaks at a round-trip time of 26.2 ms with a frame loss of 1.30 percent across the night window.
Relay RL-11-08F97 reburses the perched trunk from the freshened margin and logs a round-trip time of 181.8 ms with a frame loss of 2.75 percent across the night window.
Relay RL-12-02628 reburses the fractured trunk from the glacial margin and steps down to a round-trip time of 378.5 ms with a frame loss of 2.69 percent across the night window.
Relay RL-13-06E06 reburses the silty trunk from the mineralized margin and flags a round-trip time of 153.4 ms with a frame loss of 2.15 percent across the night window.
## Aggregation services
Aggregator AG-00-3BB logs a queue depth of 5.2 messages and a histogram bucket of 6.82 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-01-0FF reports a queue depth of 86.4 messages and a histogram bucket of 4.10 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-02-37F drifts to a queue depth of 34.7 messages and a histogram bucket of 10.25 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-03-3B3 averages a queue depth of 93.5 messages and a histogram bucket of 13.59 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-04-33E flags a queue depth of 83.2 messages and a histogram bucket of 17.58 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-05-2F2 settles near a queue depth of 84.8 messages and a histogram bucket of 0.84 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-06-2AD averages a queue depth of 75.4 messages and a histogram bucket of 7.65 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-07-06E drifts to a queue depth of 26.5 messages and a histogram bucket of 14.26 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-08-0E3 averages a queue depth of 83.9 messages and a histogram bucket of 15.51 seconds before the dedupe pass settles the stream for the next rotation.
Aggregator AG-09-138 averages a queue depth of 78.6 messages and a histogram bucket of 8.50 seconds before the dedupe pass settles the stream for the next rotation.
## Calibration procedures
Procedure CP-00-047 steps the probe through a two-point rinse, then applies a trim of 0.2084 at the 66 milli-bar mark and records a residual of 0.02052 before the next field seal is crimped.
Procedure CP-01-0A4 steps the probe through a two-point rinse, then applies a trim of 0.7793 at the 96 milli-bar mark and records a residual of 0.01468 before the next field seal is crimped.
Procedure CP-02-054 steps the probe through a two-point rinse, then applies a trim of 0.4634 at the 68 milli-bar mark and records a residual of 0.04234 before the next field seal is crimped.
Procedure CP-03-02D steps the probe through a two-point rinse, then applies a trim of 0.8459 at the 12 milli-bar mark and records a residual of 0.04855 before the next field seal is crimped.
Procedure CP-04-098 steps the probe through a two-point rinse, then applies a trim of 0.5267 at the 97 milli-bar mark and records a residual of 0.00287 before the next field seal is crimped.
Procedure CP-05-09B steps the probe through a two-point rinse, then applies a trim of 0.3968 at the 72 milli-bar mark and records a residual of 0.01637 before the next field seal is crimped.
Procedure CP-06-030 steps the probe through a two-point rinse, then applies a trim of 0.8860 at the 12 milli-bar mark and records a residual of 0.04508 before the next field seal is crimped.
Procedure CP-07-0EC steps the probe through a two-point rinse, then applies a trim of 0.2367 at the 70 milli-bar mark and records a residual of 0.00108 before the next field seal is crimped.
Procedure CP-08-0BC steps the probe through a two-point rinse, then applies a trim of 0.7278 at the 68 milli-bar mark and records a residual of 0.04707 before the next field seal is crimped.
Procedure CP-09-0AA steps the probe through a two-point rinse, then applies a trim of 0.6069 at the 37 milli-bar mark and records a residual of 0.00443 before the next field seal is crimped.
## Failure modes
Mode FM-00-27 declares a fluvial excursion when two independent probes disagree beyond the band for three rotations; the fleet reports a mean time between failures of 36902 hours for this class of excursion.
Mode FM-01-13 declares a drain excursion when two independent probes disagree beyond the band for three rotations; the fleet peaks at a mean time between failures of 29042 hours for this class of excursion.
Mode FM-02-14 declares a till excursion when two independent probes disagree beyond the band for three rotations; the fleet peaks at a mean time between failures of 26976 hours for this class of excursion.
Mode FM-03-0C declares a diapir excursion when two independent probes disagree beyond the band for three rotations; the fleet drifts to a mean time between failures of 22578 hours for this class of excursion.
Mode FM-04-39 declares a perched excursion when two independent probes disagree beyond the band for three rotations; the fleet recovers to a mean time between failures of 27383 hours for this class of excursion.
Mode FM-05-19 declares a siphon excursion when two independent probes disagree beyond the band for three rotations; the fleet logs a mean time between failures of 23152 hours for this class of excursion.
## End of document
Summarize the network's dominant risk in one paragraph, then list the three node identifiers whose readings most limit the model this month.
File diff suppressed because one or more lines are too long
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env bash
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
# derive-protected-experts.py over the calib dir, splice its cast lines into
# the marked section of the recipe in place, keep the pre-edit recipe in the
# campaign trash. Refuses to run twice (marker text says derived).
set -euo pipefail
CALIB=${1:-/home/dank/corpora/calib}
RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
PY=/home/dank/venvs/radcorpus/bin/python
DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
TS=$(date +%Y%m%d-%H%M%S)
grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
import sys
recipe, castfile = sys.argv[1], sys.argv[2]
lines = open(recipe).read().splitlines()
cast = [l for l in open(castfile).read().splitlines() if 'cast dtype=bf16' in l]
out, insec = [], False
for l in lines:
if 'PROTECTED-BEGIN' in l:
insec = True
out.append(l)
out.extend(cast)
elif 'PROTECTED-END' in l:
insec = False
out.append(l)
out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
elif not insec:
out.append(l)
open(recipe, 'w').write('\n'.join(out) + '\n')
print(f'spliced {len(cast)} cast lines into {recipe}')
PYEOF
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env python3
"""autoresearch-bench.py — deterministic greedy battery for frank-large.
Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
Fixed embedded corpus, temperature 0, streaming so decode speed is measured
between real token arrivals (not request totals). Median of REPS samples per
metric is the reported value; medians are far more stable than means under
GPU thermal drift.
Metrics emitted (stdout, one per line):
METRIC decode_tokens_per_sec=<float> primary, higher better
METRIC prefill_tokens_per_sec=<float> secondary
METRIC ttft_ms=<float> secondary, lower better
Exit non-zero if the endpoint is unhealthy or the served alias is wrong.
"""
import argparse
import json
import statistics
import sys
import time
import urllib.request
# Fixed corpus: deterministic, ~28k tokens after tokenization (same scale as
# the 2026-10-01 bench window). Built by repeating a fixed paragraph block a
# fixed number of times - NO random, NO clock, NO network.
_BLOCK = (
"The rendering pipeline walks the scene graph once per frame, resolving "
"material bindings into descriptor sets before any draw call is recorded. "
"Because the allocator recycles command buffers only after their fence "
"signals, a submission that outlives its frame must pin the pool entry; "
"otherwise the next frame overwrites recorded commands in flight. The "
"validation layer flags exactly this class of lifetime bug, but only for "
"buffers it can see - GPU-side timestamps and queue-family ownership "
"transfers hide in the driver's own bookkeeping, which is why release "
"builds must lean on explicit barriers and why a frame-time regression "
"often shows up as corruption one frame later than its cause.\n"
)
LONG_CTX = _BLOCK * 220 # ~28k-token prefill, matching the deployed workload
_LONG = LONG_CTX + "\nNow, in one focused paragraph (80-150 words), explain " \
"which lifetime bugs this pipeline can still suffer and what explicit " \
"mitigations the text implies."
_SHORT = "Design a rate limiter for a homelab reverse proxy that must not " \
"lock out the LAN. Think it through, then give the algorithm."
REPS = 3
MAX_TOKENS = 256
REQ_TIMEOUT = 900
def stream_completion(url: str, model: str, prompt: str):
"""Greedy streaming completion.
Returns (ttft_s, decode_tps, prefill_tps, prompt_tokens, n_tokens).
ttft is RELATIVE to request send: time.perf_counter()'s epoch is host
boot, and returning the raw stamp read as an 11-day first token.
"""
body = json.dumps({
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": MAX_TOKENS,
"temperature": 0,
"stream": True,
"stream_options": {"include_usage": True},
}).encode()
req = urllib.request.Request(
url.rstrip("/") + "/v1/chat/completions", data=body,
headers={"Content-Type": "application/json"})
t0 = time.perf_counter()
first = None
stamps = []
usage = {}
with urllib.request.urlopen(req, timeout=REQ_TIMEOUT) as resp:
for raw in resp:
line = raw.decode(errors="replace").strip()
if not line.startswith("data:"):
continue
payload = line[5:].strip()
if payload == "[DONE]":
break
chunk = json.loads(payload)
if chunk.get("usage"):
usage = chunk["usage"]
for ch in chunk.get("choices", []):
d = ch.get("delta") or {}
# reasoning AND content deltas both arrive token by token;
# count either so greedy text length drives the rate.
# (llama.cpp names the thinking field reasoning_content.)
if d.get("content") or d.get("reasoning") or d.get("reasoning_content"):
now = time.perf_counter()
if first is None:
first = now
stamps.append(now)
if first is None or len(stamps) < 2:
# A zero-argmax degenerate answer would poison the median; raise.
raise RuntimeError(f"stream produced {len(stamps)} tokens")
ttft_s = first - t0
decode_s = stamps[-1] - stamps[0]
decode_tps = (len(stamps) - 1) / decode_s if decode_s > 0 else 0.0
prompt_tokens = usage.get("prompt_tokens") or 0
prefill_tps = prompt_tokens / ttft_s if prompt_tokens else 0.0
return ttft_s, decode_tps, prefill_tps, prompt_tokens, len(stamps)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--url", default="http://127.0.0.1:7770")
ap.add_argument("--model", default="frank-dragoon")
args = ap.parse_args()
with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",
timeout=10) as r:
ids = [m["id"] for m in json.load(r)["data"]]
if args.model not in ids:
print(f"FATAL: served ids {ids}, expected {args.model}",
file=sys.stderr)
sys.exit(2)
# Warmup (compile/graph warm + allocator settle), NOT measured.
stream_completion(args.url, args.model, _SHORT)
decodes, prefills, ttfts = [], [], []
for rep in range(REPS):
for name, prompt in (("long", _LONG), ("short", _SHORT)):
ttft, dt, pt, ptk, ntok = stream_completion(
args.url, args.model, prompt)
decodes.append(dt)
ttfts.append(ttft * 1000.0)
if name == "long":
prefills.append(pt)
print(f" rep{rep} {name:5s} ttft={ttft*1000:8.1f}ms "
f"prefill={pt:8.1f} t/s decode={dt:6.2f} t/s "
f"({ntok} tok, prompt {ptk})", file=sys.stderr)
med = lambda xs: statistics.median(xs)
print(f"METRIC decode_tokens_per_sec={med(decodes):.2f}")
print(f"METRIC prefill_tokens_per_sec={med(prefills):.1f}")
print(f"METRIC ttft_ms={med(ttfts):.1f}")
if __name__ == "__main__":
main()
+187
View File
@@ -0,0 +1,187 @@
#!/usr/bin/env bash
# autoresearch-frank-primary.sh — RUNS ON DANKSERVER, driven over ssh by the
# dev-box autoresearch.sh. It is the canonical frank-primary benchmark entry
# point for the autoresearch loop.
#
# Method (mandated by the Benchmarking Policy): the production service serves
# real users and shares the two R9700s, so a bench beside it starves the GPUs
# and corrupts results. This script therefore:
# 1. stops the production frank-primary unit (root, relink symlink: stop,
# NOT mask — mask refuses to replace a symlinked unit file),
# 2. boots a scratch radiance container named `arbench` on the SAME two
# cards with the candidate engine argv (the env file the loop edits),
# 3. polls /health until the listener is live (it binds AFTER the full
# load, so a bare curl is an honest gate),
# 4. runs ONE self-contained completion request locally (127.0.0.1, no LAN
# jitter): a fixed ~140k-char real-text prompt (~32k tokens) decoded for
# 256 tokens, greedy. The engine's `timings` object reports the honest
# prefill and decode rates for that single request.
# 5. prints METRIC lines and tears the scratch down,
# 6. restores production (systemctl start), via a trap so a failed bench
# never leaves frank-primary down (the stale-probe-tail lesson).
#
# Determinism: the prompt is built from a fixed, sorted corpus of repo files
# with a fixed random seed, so prompt_n is stable across runs. Sampling is
# greedy (temperature 0), so the decode length is fixed; the only residual
# nondeterminism is the wht6 lossy TP all-reduce, which is a constant across
# runs and cancels in A/B comparisons.
#
# The scratch mirrors the production container contract exactly (serve-frank-
# primary.sh): --network host, --init, seccomp=unconfined (io_uring for the
# n-gram table), memlock unbounded (the ~132 GiB pinned ledger), /dev/kfd +
# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
# differences: the container name (arbench, so prod's identity is untouched)
# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
# prefill is cold — the prod tier at kvcache/frank-primary is left intact.
set -uo pipefail
ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-primary.env}"
SCRATCH=arbench
PORT=7770
HEALTH="http://127.0.0.1:$PORT/health"
COMPLETE="http://127.0.0.1:$PORT/v1/completions"
BODY=/tmp/arbench-body.json
LOG=/home/dank/logs/arbench.log
restore() {
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
sudo systemctl start frank-primary >/dev/null 2>&1 || true
}
trap restore EXIT
# --- source the candidate config (same file the production unit reads) ------
if [ ! -f "$ENV_FILE" ]; then
echo "no env file at $ENV_FILE" >&2; exit 1
fi
# shellcheck disable=SC1090
set -a; . "$ENV_FILE"; set +a
# --- stop production; the scratch needs both cards to itself ----------------
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
FRESH_KV="$FRANK_PRIMARY_STATE_DIR/kvcache/$SCRATCH"
rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
# --- boot the scratch: the production run line, scratch name + fresh KV ------
podman run -d --name "$SCRATCH" \
--network host \
--init \
--security-opt seccomp=unconfined \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-v "$FRANK_PRIMARY_STATE_DIR":/data \
-v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
-v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
"$FRANK_PRIMARY_IMAGE" \
--model "/models/$FRANK_PRIMARY_MODEL_FILE" \
--host :: --port "$PORT" \
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
--override-chat-template /templates/template.jinja \
--tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
--max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
--max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
--placement "$FRANK_PRIMARY_PLACEMENT" \
--host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
--gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
--vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
--vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
--prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
--prefix-cache-dir "/data/kvcache/$SCRATCH" \
--prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
--ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
--num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
--max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
> "$LOG" 2>&1
if [ $? -ne 0 ]; then echo "scratch boot failed; see $LOG" >&2; exit 1; fi
# --- poll health (listener binds after the full load; cold boot ~110 s) ------
ok=0
for _ in $(seq 1 360); do
code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 3 "$HEALTH" 2>/dev/null || echo 000)
if [ "$code" = "200" ]; then ok=1; break; fi
sleep 2
done
if [ "$ok" -ne 1 ]; then
echo "scratch never reached health 200; tail of $LOG:" >&2
tail -30 "$LOG" >&2; exit 1
fi
# --- build the workload: a fixed ~140k-char real-text prompt (repo corpus) ---
python3 - "$BODY" <<'PY'
import json, sys, random, glob
random.seed(1234)
root = "/home/dank/serverAppConfigs"
files = sorted(
glob.glob(f"{root}/serverAppConfigs-docs/**/*.md", recursive=True)
+ glob.glob(f"{root}/scripts/*.sh")
+ glob.glob(f"{root}/scripts/*.py")
+ glob.glob(f"{root}/AGENTS.md")
)
corpus = []
total = 0
for f in files:
try:
t = open(f, encoding="utf-8", errors="ignore").read()
except Exception:
continue
if not t.strip():
continue
corpus.append(t)
total += len(t)
if total > 240000:
break
corpus = "\n".join(corpus)
TGT = 140000
parts, n = [], 0
while n < TGT:
start = random.randint(0, max(0, len(corpus) - 500))
seg = corpus[start:start + random.randint(200, 500)]
parts.append(seg)
n += len(seg) + 1
prompt = " ".join(parts)
json.dump(
{"model": "frank-archon", "prompt": prompt,
"max_tokens": 256, "temperature": 0.0, "stream": False,
"stop": ["<|im_end|>"]},
open(sys.argv[1], "w"),
)
PY
if [ ! -f "$BODY" ]; then echo "prompt build failed" >&2; exit 1; fi
# --- the bench request (prefill ~32k + decode 256, greedy) -------------------
RESP=$(curl -s --max-time 600 -X POST "$COMPLETE" \
-H 'Content-Type: application/json' -d "@$BODY")
echo "$RESP" > /tmp/arbench-resp.json
# --- parse the timings into METRIC lines -------------------------------------
python3 - <<'PY'
import json, sys
try:
r = json.load(open("/tmp/arbench-resp.json"))
except Exception as e:
sys.stderr.write(f"unparseable response: {e}\n"); sys.exit(1)
if "timings" not in r:
sys.stderr.write(f"no timings in response: {json.dumps(r)[:500]}\n"); sys.exit(1)
t = r["timings"]
dtps = t.get("predicted_per_second")
ptps = t.get("prompt_per_second")
pms = t.get("prompt_ms")
dn = t.get("draft_n", 0)
da = t.get("draft_n_accepted", 0)
if dtps is None:
sys.stderr.write("predicted_per_second null (no decode produced)\n"); sys.exit(1)
print(f"METRIC decode_tps={dtps:.4f}")
if ptps is not None:
print(f"METRIC prefill_tps={ptps:.4f}")
if pms is not None:
print(f"METRIC ttft_ms={pms:.4f}")
if dn:
print(f"METRIC draft_accept={da/dn:.4f}")
print(f"INFO prompt_n={t.get('prompt_n')} cache_n={t.get('cache_n')} predicted_n={t.get('predicted_n')}")
PY
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env bash
# frank-primary-arms.sh — run a SEQUENCE of frank-primary bench arms and keep
# every arm's evidence. Runs on the operator box; each arm is one full window
# of ./autoresearch.sh (stop+mask -> boot the arm as the service -> measure ->
# restore production and verify), so the fleet is only ever down for one arm at
# a time and the production unit is back up between arms.
#
# An arm is the repo's own frank-primary.env plus an OVERLAY of engine knobs
# appended after it (later assignment wins when the serve script sources it),
# staged through FRANK_PRIMARY_AR_ENV_FILE so the checkout is never mutated and
# each arm gets its own fingerprint (the memo can never answer one arm with
# another arm's numbers).
#
# Usage: scripts/autoresearch/frank-primary-arms.sh [arm ...]
# scripts/autoresearch/frank-primary-arms.sh --list
# Output, under /tmp/frank-primary-arms-<UTC stamp>/ (override with ARMS_DIR):
#
# <arm>.out full autoresearch output (metrics + engine evidence)
# <arm>.env the exact staged env
# <arm>.rc window exit code
# summary.tsv arm, rc, decode_tps, decode_tps_t0.7, kv_tokens, boot_s
#
# Exit 0 when every arm produced metrics; 1 if any arm failed (a failed boot is
# a RESULT — the window still restored production, and the rc says which arm).
set -uo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
REPO="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
BASE_ENV="$REPO/scripts/frank-primary.env"
ARMS_DIR="${ARMS_DIR:-/tmp/frank-primary-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
# --- arms ------------------------------------------------------------------
# baseline: the shipped shape E, measured with THIS harness so the comparison is
# matched (the ~29.6 t/s figure in the docs is an older shape, and no shape-E
# decode number exists yet).
OVER_baseline=()
# B: MTP3 with the vendor's probabilistic draft sampling, a capture ladder whose
# step is num_speculative_tokens + 1 = 4, and the sampling override. Precise
# memory stays ON — this arm is simultaneously the MTP test and the test of
# whether MTP's drafter workspace and the fitted 5.25 GiB overhead can coexist.
SPEC3='{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
LADDER3='{"cudagraph_capture_sizes":[4,8,12],"max_cudagraph_capture_size":12,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
# The value must arrive in the staged env file SINGLE-QUOTED: sourcing an env
# file performs quote removal, so an unquoted {"temperature":0.7} would reach
# the engine as {temperature:0.7}.
GEN07='{"temperature":0.7}'
OVER_B=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
)
# B2: identical to B with the default (7.60 GiB) overhead, i.e. precise memory
# OFF. It is the fallback if B dies at graph capture, and it prices the arena
# that precise memory buys (7,522 slots -> ~6,496).
OVER_B2=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
'FRANK_PRIMARY_PRECISE_MEMORY='
)
# C: the vendor's concurrency shape — SEQS 16 with the [4..32] ladder and util
# 0.95. This is also the direct test of the theory that pinning the capture
# ladder unlocks SEQS > 3 (the SEQS=2 failure was query_start_loc specialised to
# the constant 3, which is NOT in the default ladder; 4 is).
LADDER16='{"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
OVER_C=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER16'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
'FRANK_PRIMARY_SEQS=16'
'FRANK_PRIMARY_MNBT=4096'
'FRANK_PRIMARY_UTIL=0.95'
)
# D/E: the CONTEXT arms. Measured 2026-10-02 (arms run 20261002T233855Z): the
# KV pool is a hard byte request divided by a PER-TOKEN PRICE, and MTP raises
# that price because the drafter keeps its own KV per position:
# no spec: 3.15e9 / 398,658 tokens = 7,901 B/token (1.99x of 200,704)
# MTP3: 3.15e9 / 313,002 tokens = 10,064 B/token (1.56x)
# So arm C lost a fifth of the context to the drafter, not to util. Frank's
# contract is >=401,400 tokens over >=2 seqs = 2.00x of 200,704, which at
# 10,064 B/token costs 4.04e9 -> ask 4.05e9 (+0.83 GiB/rank).
# D is arm B with EXACTLY ONE variable changed (the KV bytes), so its result
# says whether the 4k-depth collapse (B 23.09 vs baseline 33.99) was KV or the
# batch shape.
KV401K='FRANK_PRIMARY_KV_BYTES=4050000000'
OVER_D=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
"$KV401K"
)
# E: D with C's max_num_batched_tokens (4096 — C is the only arm that did NOT
# collapse at 4k depth: 33.15 where B gave 23.09) and util 0.99, the value
# Frank asked about. 0.99 is affordable on paper — production at util 0.98
# measures 30.37/34.21 GiB per card, 3.58 GiB physically unused — but it leaves
# only 0.32 GiB above the allowance while the CUDA-graph cost is an ANALYTIC
# estimate (C's was 1.47 GiB over 24 graphs), so it is measured, not assumed.
OVER_E=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
"$KV401K"
'FRANK_PRIMARY_MNBT=4096'
'FRANK_PRIMARY_UTIL=0.99'
)
arm_overlay() {
case "$1" in
baseline) ;; # the shipped env, no overlay
B) printf '%s\n' "${OVER_B[@]}" ;;
B2) printf '%s\n' "${OVER_B2[@]}" ;;
C) printf '%s\n' "${OVER_C[@]}" ;;
D) printf '%s\n' "${OVER_D[@]}" ;;
E) printf '%s\n' "${OVER_E[@]}" ;;
*) return 1 ;;
esac
}
if [ "${1:-}" = "--list" ]; then
printf 'baseline B B2 C D E\n'
exit 0
fi
[ -f "$BASE_ENV" ] || { echo "[arms] missing $BASE_ENV" >&2; exit 1; }
if [ "$#" -gt 0 ]; then ARMS=("$@"); else ARMS=(baseline B B2 C); fi
mkdir -p "$ARMS_DIR"
echo "[arms] dir $ARMS_DIR arms: ${ARMS[*]}" >&2
: >"$ARMS_DIR/summary.tsv"
printf 'arm\trc\tdecode_tps\tdecode_tps_t0.7\tkv_tokens\tboot_s\n' >>"$ARMS_DIR/summary.tsv"
overall=0
for arm in "${ARMS[@]}"; do
if ! arm_overlay "$arm" >"$ARMS_DIR/$arm.overlay"; then
echo "[arms] unknown arm: $arm" >&2
overall=1
continue
fi
{ cat "$BASE_ENV"; echo; echo "# --- arm $arm overlay ($(date -u +%FT%TZ)) ---"
arm_overlay "$arm"; } >"$ARMS_DIR/$arm.env"
# Gate the staged env BEFORE burning a ~20-minute window: every knob that
# holds JSON must still be one valid JSON object after the file is SOURCED
# (that is the step that silently eats unquoted braces).
if ! bash -c 'set -a; . "$1" >/dev/null; set +a
for k in FRANK_PRIMARY_SPEC FRANK_PRIMARY_COMPCONFIG FRANK_PRIMARY_GEN_CONFIG; do
v="${!k:-}"
[ -n "$v" ] || continue
case "$v" in
"{"*) python3 -c "import json,sys; json.loads(sys.argv[1])" "$v" \
|| { echo "[arms] $k is not valid JSON: $v" >&2; exit 1; } ;;
*) echo "[arms] $k is not a JSON object: $v" >&2; exit 1 ;;
esac
done' _ "$ARMS_DIR/$arm.env"; then
echo "[arms] $arm: staged env rejected, window not opened" >&2
echo 99 >"$ARMS_DIR/$arm.rc"
overall=1
continue
fi
echo "[arms] === $arm: opening window $(date -u +%H:%M:%S)Z ===" >&2
FRANK_PRIMARY_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
"$REPO/autoresearch.sh" >"$ARMS_DIR/$arm.out" 2>&1
rc=$?
echo "$rc" >"$ARMS_DIR/$arm.rc"
echo "[arms] === $arm: rc=$rc $(date -u +%H:%M:%S)Z ===" >&2
grep -E '^METRIC ' "$ARMS_DIR/$arm.out" | sed 's/^METRIC //' >"$ARMS_DIR/$arm.metrics" || true
metric() { local v; v="$(sed -n "s/^$1=//p" "$ARMS_DIR/$arm.metrics" | tail -1)"; printf '%s' "${v:-$2}"; }
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$arm" "$rc" \
"$(metric decode_tps FAIL)" "$(metric decode_tps_t0.7 FAIL)" \
"$(metric kv_tokens ?)" "$(metric boot_seconds ?)" >>"$ARMS_DIR/summary.tsv"
[ "$rc" -ne 0 ] && overall=1
done
echo "[arms] summary:" >&2
cat "$ARMS_DIR/summary.tsv" >&2
exit "$overall"
+216
View File
@@ -0,0 +1,216 @@
#!/usr/bin/env python3
"""frank-primary decode measurement — runs INSIDE the engine container.
The method is the fleet's established one (the 2026-09-29 / 2026-10-01
tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
under speculative decoding because rejected drafts change the accounting, so
decode is measured from the server's own usage counters:
decode_tok_s = completion_tokens / (wall - ttft)
with ttft = wall time to the FIRST content-bearing chunk (thinking arrives in
`reasoning`/`delta.reasoning` on this stack and is excluded from the decode
window — and is deterministic per prompt at temperature 0, so the exclusion
cancels in any A/B on the same workload).
Workload (fixed, deterministic, offline):
* prompts built from a large fixed-seed shuffled word list so tokens rarely
repeat (no n-gram/prefix predictability of the text itself), sized with
the server's own /v1/messages/count_tokens counter
* depths 115 / 1056 / 4096 target prompt tokens
* one 64-token warm pass per depth (warms the expert arena), then
max_tokens=256, 3 runs, median — repeated for every temperature in AR_TEMPS
(default 0 and 0.7). The prompt workload is deterministic (fixed seed, no
network, no time-of-day dependence); at temperature > 0 the generated text
is not, but the measured window is bounded by max_tokens, so the rate stays
comparable and the non-determinism is exactly the sampling that a
speculative-decode arm must survive.
Env:
AR_BASE endpoint (default http://127.0.0.1:7770)
AR_MODEL served name (default frank-primary)
AR_PROMPT_CEILING hard prompt ceiling in tokens (default 100000)
AR_TEMPS comma-separated sampling temperatures (default "0,0.7")
Output: progress on stderr, ONE json object on stdout (the last line).
Exit codes: 0 ok, 2 endpoint unhealthy / no usable completion, 3 bad usage.
"""
import json
import os
import random
import statistics
import sys
import time
import urllib.error
import urllib.request
BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
MODEL = os.environ.get("AR_MODEL", "frank-archon")
PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
DEPTHS = (115, 1056, 4096)
WARM_TOKENS = 64
MAX_TOKENS = 256
RUNS = 3
# Temperature sweep. The unsuffixed keys are temperature 0 — byte-comparable to
# every historical measurement — and AR_TEMPS adds more. The second one matters
# for speculative decode: draft ACCEPTANCE is sampling-dependent (a greedy draft
# matches a greedy target exactly and is rejected more under a non-greedy
# target), so an MTP arm judged only at temp 0 would be judged on half the
# contract. Production traffic is non-zero temperature (the payload's
# generation_config says 1.0; an override to 0.7 is under test).
TEMPS = tuple(float(x) for x in os.environ.get("AR_TEMPS", "0,0.7").split(","))
# 43-word x 12 = 516-word vocabulary, seed 1234 — the campaign's exact list
# and seed, so prompts are byte-identical to the historical measurements.
_VOCAB = ("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima "
"mike november oscar papa quebec romeo sierra tango uniform victor whiskey "
"xray yankee zulu cardinal ordinal topology kernel scheduler resident "
"unified aperture expert arena slab offload prefetch gemm triton quant "
"tensor shard checkpoint manifest embed router cache page table window "
"gradient sparse dense moe attention head dim block stream decode prefill "
"bandwidth latency throughput residency eviction tier swap pin memlock").split()
_R = random.Random(1234)
_ORDER = list(range(len(_VOCAB)))
_R.shuffle(_ORDER)
def log(msg):
print("[measure] %s" % msg, file=sys.stderr, flush=True)
def post_json(path, body, timeout):
req = urllib.request.Request(
BASE + path, data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=timeout) as r:
return json.load(r)
def count_tokens(text):
"""Token count without a prefill, from the server's own counter."""
body = {"model": MODEL,
"messages": [{"role": "user", "content": text}]}
r = post_json("/v1/messages/count_tokens", body, timeout=300)
n = r.get("input_tokens") or r.get("count") or r.get("token_count")
if not n:
raise RuntimeError("count_tokens returned no count: %r" % (r,))
return int(n)
def words(n):
return " ".join(_VOCAB[_ORDER[i % len(_VOCAB)]] for i in range(n))
def calibrate():
"""Measure words->tokens from the server's own counter."""
return 2000.0 / count_tokens(words(2000))
def prompt_for(target_tokens, wpt):
n = max(1, int(target_tokens * wpt))
text = words(n)
pt = count_tokens(text)
while pt > PROMPT_CEILING:
n = max(1, int(n * (PROMPT_CEILING / pt) * 0.98))
text = words(n)
pt = count_tokens(text)
return text, pt
def measured(prompt, max_tokens, temperature=0):
"""Return (ttft_s, decode_tok_s, completion_tokens) via usage counters."""
body = {"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens, "temperature": temperature,
"stream": True, "stream_options": {"include_usage": True}}
req = urllib.request.Request(
BASE + "/v1/chat/completions", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
t0 = time.time()
ttft = None
usage = {}
with urllib.request.urlopen(req, timeout=1800) as r:
for raw in r:
line = raw.decode("utf-8", "replace").strip()
if not line.startswith("data:"):
continue
s = line[5:].strip()
if s == "[DONE]":
break
try:
ev = json.loads(s)
except json.JSONDecodeError:
continue
if ev.get("usage"):
usage = ev["usage"]
for ch in ev.get("choices", []):
if (ch.get("delta") or {}).get("content"):
if ttft is None:
ttft = time.time() - t0
wall = time.time() - t0
ct = int(usage.get("completion_tokens") or 0)
if ct <= 0:
# No usage on the stream: the endpoint is unhealthy or the model
# returned nothing (e.g. HTTP 200 with an error payload).
raise RuntimeError("no completion_tokens in usage: %r" % (usage,))
dec_window = max(wall - (ttft or 0.0), 1e-9)
return (ttft or wall), (ct / dec_window), ct
def fmt_temp(t):
return ("%.1f" % t).rstrip("0").rstrip(".") if t else "0"
def main():
log("base=%s model=%s ceiling=%d temps=%s"
% (BASE, MODEL, PROMPT_CEILING, ",".join(fmt_temp(t) for t in TEMPS)))
try:
wpt = calibrate()
except Exception as e:
log("endpoint unhealthy during calibration: %s" % e)
return 2
log("words/token: %.3f" % wpt)
result = {"words_per_token": round(wpt, 5)}
for tgt in DEPTHS:
try:
p, pt = prompt_for(tgt, wpt)
measured(p, WARM_TOKENS) # warm the arena and the prefix cache
except Exception as e:
log("warm pass failed at depth %d: %s" % (tgt, e))
return 2
for temp in TEMPS:
suffix = "" if temp == 0 else "_t%s" % fmt_temp(temp)
rates, tts, cts = [], [], []
for i in range(RUNS):
try:
tt, r, ct = measured(p, MAX_TOKENS, temp)
except Exception as e:
log("run %d/%d failed at depth %d temp %s: %s"
% (i + 1, RUNS, tgt, fmt_temp(temp), e))
continue
rates.append(r)
tts.append(tt)
cts.append(ct)
if len(rates) < 2:
log("only %d usable run(s) at depth %d temp %s — refusing to median"
% (len(rates), tgt, fmt_temp(temp)))
return 2
result["decode_tps_%d%s" % (tgt, suffix)] = round(statistics.median(rates), 2)
result["ttft_ms_%d%s" % (tgt, suffix)] = round(1000.0 * statistics.median(tts), 1)
result["completion_tokens_%d%s" % (tgt, suffix)] = int(statistics.median(cts))
result["prompt_tokens_%d%s" % (tgt, suffix)] = pt
log("depth %5d tok temp %-4s decode %7.2f t/s ttft %.3f s (runs=%d)"
% (pt, fmt_temp(temp), result["decode_tps_%d%s" % (tgt, suffix)],
result["ttft_ms_%d%s" % (tgt, suffix)] / 1000.0, len(rates)))
for temp in TEMPS:
suffix = "" if temp == 0 else "_t%s" % fmt_temp(temp)
meds = [result["decode_tps_%d%s" % (t, suffix)] for t in DEPTHS]
result["decode_tps%s" % suffix] = round(statistics.median(meds), 2)
print(json.dumps(result))
return 0
if __name__ == "__main__":
sys.exit(main())
+218
View File
@@ -0,0 +1,218 @@
#!/usr/bin/env bash
# frank-primary-window.sh — one full bench window for frank-primary, run ON
# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
#
# Discipline (AGENTS.md Benchmarking Policy): the production unit is
# STOPPED + MASKED first, the candidate config boots AS THE SERVICE itself
# (same serve script, same container name, same port 7770 — no scratch
# ports, no co-tenant GPU), the in-container measure runs, and the window
# ALWAYS ends by restoring the production unit and verifying its health.
# The EXIT trap makes the restore unconditional: a boot failure, a measure
# failure, or a dead ssh session all end with production back up.
#
# Staging layout (created by the orchestrator, ephemeral — never a config
# source; the repo checkout is):
# /home/dank/.autoresearch-frank-primary/
# serve.sh staged candidate serve script
# frank-primary.env staged candidate env (the candidate knobs)
# measure.py the in-container measure client
# hash candidate fingerprint
# run.json result (metrics + boot + kv + fingerprint)
# serve.out/measure.out/fail.log
# last/ memo: run.json + hash of the last FULLY
# successful window (bench AND verified restore)
#
# This script logs to stderr and exits 0 only when the measure produced a
# usable result (the restore happens in the trap regardless of the bench
# outcome; a failed restore forces exit 1).
set -uo pipefail
STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
HEALTH_URL="http://127.0.0.1:7770/health"
MODELS_URL="http://127.0.0.1:7770/v1/models"
BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
SERVED="${SERVED:-frank-primary}"
RESTORED=0
log() { printf '[window %s] %s\n' "$(date '+%H:%M:%S')" "$*" >&2; }
container_state() { podman inspect -f '{{.State.Status}}' "$CTR" 2>/dev/null || echo absent; }
wait_port_free() {
local i
for i in $(seq 1 90); do
[ "$(container_state)" != "running" ] && return 0
sleep 2
done
return 1
}
wait_health() { # $1 = timeout seconds
local t0 code
t0=$(date +%s)
while :; do
code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$HEALTH_URL" 2>/dev/null || echo 000)"
[ "$code" = "200" ] && return 0
[ "$(( $(date +%s) - t0 ))" -ge "$1" ] && return 1
sleep 2
done
}
capture_bench_failure() {
log "capturing bench-boot failure evidence"
{
echo "=== podman logs (tail 400) ==="
podman logs --tail 400 "$CTR" 2>&1 || true
echo "=== staged serve.out (tail 60) ==="
tail -60 "$STAGE/serve.out" 2>/dev/null || true
echo "=== engine log (tail 60) ==="
tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
} >"$STAGE/fail.log" 2>&1 || true
grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
"$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
}
restore() {
[ "$RESTORED" = 1 ] && return 0
RESTORED=1
log "restoring production unit $UNIT"
if [ -f "$STAGE/serve.pid" ]; then
kill "$(cat "$STAGE/serve.pid" 2>/dev/null)" 2>/dev/null || true
fi
podman stop -t 120 "$CTR" >/dev/null 2>&1 || true
wait_port_free || log "WARNING: bench container still draining after 180s"
sudo systemctl unmask "$UNIT" 2>/dev/null || true
if ! sudo systemctl start "$UNIT"; then
log "ERROR: systemctl start $UNIT failed — operator intervention required"
return 1
fi
if wait_health "$RESTORE_TIMEOUT"; then
if curl -s --max-time 10 "$MODELS_URL" 2>/dev/null | grep -q "$SERVED"; then
log "production restored and verified ($SERVED answering on :7770)"
# Memo promotion lives HERE: a cached result is only ever the
# product of a fully successful window (bench AND verified
# restore), never of a bare bench success.
cp "$STAGE/run.json" "$STAGE/last/" 2>/dev/null || true
cp "$STAGE/hash" "$STAGE/last/" 2>/dev/null || true
else
log "ERROR: health 200 but /v1/models does not list $SERVED"
return 1
fi
else
log "ERROR: production not healthy within ${RESTORE_TIMEOUT}s — unit left to systemd (Restart=always)"
return 1
fi
return 0
}
trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
main() {
local f
for f in serve.sh frank-primary.env measure.py hash; do
[ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
done
mkdir -p "$STAGE/last"
rm -f "$STAGE/run.json" "$STAGE/fail.log" "$STAGE/measure.out" "$STAGE/evidence.log"
local t_start t0 i kvline kv_tokens kv_conc measure_json boot_seconds
t_start="$(date +%s)"
log "stopping and masking production unit $UNIT"
sudo systemctl stop "$UNIT" 2>/dev/null || true
# Explicit stop is not auto-restarted even with Restart=always; the mask
# is belt-and-braces against a stray start during the window.
sudo systemctl mask "$UNIT" 2>/dev/null || \
log "WARNING: mask failed — relying on explicit-stop semantics"
wait_port_free || log "note: production container still draining"
t0="$(date +%s)"
log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
echo "$!" >"$STAGE/serve.pid"
for i in $(seq 1 45); do
[ "$(container_state)" = "running" ] && break
sleep 2
done
if [ "$(container_state)" != "running" ]; then
log "bench container never reached running state"
tail -40 "$STAGE/serve.out" >&2 || true
exit 1
fi
if wait_health "$BOOT_TIMEOUT"; then
boot_seconds=$(( $(date +%s) - t0 ))
log "bench healthy after ${boot_seconds}s"
else
if [ "$(container_state)" = "running" ]; then
log "bench boot timed out after ${BOOT_TIMEOUT}s (still loading)"
else
log "bench container died during boot"
capture_bench_failure
fi
exit 1
fi
log "running measure in container (timeout ${MEASURE_TIMEOUT}s)"
if ! podman cp "$STAGE/measure.py" "$CTR":/tmp/ar-measure.py; then
log "podman cp of measure client failed"
exit 1
fi
if ! timeout "$MEASURE_TIMEOUT" podman exec "$CTR" python3 /tmp/ar-measure.py \
>"$STAGE/measure.out" 2>&1; then
log "measure failed or timed out"
tail -40 "$STAGE/measure.out" >&2 || true
exit 1
fi
# Capacity facts for the run (context-size experiments change these).
kvline="$(podman logs "$CTR" 2>&1 | grep -E 'GPU KV cache size' | tail -1 || true)"
kv_tokens="$(printf '%s' "$kvline" | sed -E 's/.*GPU KV cache size: ([0-9,]+) tokens.*/\1/' | tr -d ',' || true)"
kv_conc="$(printf '%s' "$kvline" | sed -E 's/.*per request: ([0-9.]+x).*/\1/' || true)"
# Human-facing evidence, in a sidecar file because it is multi-line prose,
# not metrics: the expert-plan lines (the ONLY place the arena / runtime
# overhead / KV-reserve numbers exist — they are never in /metrics), the
# compile path (cold build vs cache hit), the graph-capture sizes, the
# speculative-decode counters, and any OOM signature. Without this an arm
# that loses decode speed is unexplainable afterwards.
{
echo "=== engine plan / capacity / spec evidence ==="
podman logs "$CTR" 2>&1 | grep -E \
'KV cache size|Maximum concurrency|layers split|expert cache|runtime overhead|non-default args|r4d all-reduce|Using \[|cudagraph|CUDA graph|Capturing|graph for compile range|torch.compile took|Directly load the compiled|speculative|Speculative|acceptance|Acceptance|offload|precise|OutOfMemory|Tried to allocate|Engine core initialization failed' \
| tail -150 || true
echo "=== /metrics counters (spec decode, caches, success) ==="
curl -s --max-time 15 http://127.0.0.1:7770/metrics 2>/dev/null \
| grep -E '^vllm:(spec_decode|gpu_cache|prefix_cache|request_success|generation_tokens|iteration)' \
| head -80 || true
} >"$STAGE/evidence.log" 2>&1 || true
log "engine evidence -> $STAGE/evidence.log ($(wc -l <"$STAGE/evidence.log") lines)"
measure_json="$(grep -E '^\{' "$STAGE/measure.out" | tail -1 || true)"
if [ -z "$measure_json" ]; then
log "measure produced no JSON result line"
tail -40 "$STAGE/measure.out" >&2 || true
exit 1
fi
# The measure line is already JSON, so the run object can be assembled
# without any host-side interpreter.
printf '{"hash":"%s","boot_seconds":%s,"kv_tokens":%s,"kv_concurrency":"%s","total_seconds":%s,"measure":%s}\n' \
"$(tr -d '[:space:]' <"$STAGE/hash")" \
"$boot_seconds" "${kv_tokens:-0}" "${kv_conc:-unknown}" \
"$(( $(date +%s) - t_start ))" "$measure_json" >"$STAGE/run.json"
log "bench window complete in $(( $(date +%s) - t_start ))s"
# Memo only after a FULL success; the restore below still has to verify.
exit 0
}
main "$@"
exit "$?"
+1 -1
View File
@@ -57,7 +57,7 @@ run_bench() {
# Warmup (small request)
curl -s -m 60 http://127.0.0.1:$PORT/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"frank-large","messages":[{"role":"user","content":"Hi"}],"max_tokens":16,"stream":false}' > /dev/null 2>&1 || true
-d '{"model":"frank-dragoon","messages":[{"role":"user","content":"Hi"}],"max_tokens":16,"stream":false}' > /dev/null 2>&1 || true
# Build 8K prompt
local prompt
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env python3
"""bench-probe.py — RUNS ON DANKSERVER. The campaign's decode probe: one
/completion request, a prompt assembled from calib.jsonl to ~TARGET tokens,
temperature 0, max_tokens 512. Reports the engine's own timings: decode rate,
prefill rate, draft accept. Run 3x; the median is the arm's number."""
import json, sys, urllib.request
TARGET = 43600 # tokens
BYTES_PER_TOK = 3.8
URL = "http://localhost:7770/v1/completions"
def build_prompt():
want = int(TARGET * BYTES_PER_TOK)
parts = []
n = 0
with open("/home/dank/corpora/out/calib.jsonl") as f:
for line in f:
d = json.loads(line)
p = d.get("prompt") or d.get("messages", [{}])[0].get("content", "")
parts.append(p)
n += len(p.encode())
if n >= want:
break
return "".join(parts)
def probe(salt):
prompt = f"[probe {salt}]\n" + build_prompt()
body = json.dumps({
"model": "frank-archon",
"prompt": prompt, "temperature": 0.0, "max_tokens": 512,
}).encode()
req = urllib.request.Request(URL, data=body,
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=1800) as r:
d = json.load(r)
t = d.get("timings", {})
return {
"prompt_n": t.get("prompt_n"),
"prefill_tps": round(t.get("prompt_per_second", 0), 1),
"pred_n": t.get("predicted_n"),
"decode_tps": round(t.get("predicted_per_second", 0), 2),
"draft_n": t.get("draft_n"),
"accept": (round(t["draft_n_accepted"] / t["draft_n"], 4)
if t.get("draft_n") else None),
}
if __name__ == "__main__":
print(json.dumps(probe(sys.argv[1] if len(sys.argv) > 1 else "a")))
+1 -1
View File
@@ -39,7 +39,7 @@ def send_prefill_request():
with open(PROMPT_FILE, "r") as f:
prompt = f.read()
body = json.dumps({
"model": "frank-large",
"model": "frank-dragoon",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 1,
"stream": False
+166
View File
@@ -0,0 +1,166 @@
#!/usr/bin/env bash
# build-frank-primary-rad.sh — build frank-primary's .rad container from the
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint, at NATIVE precision.
#
# bash scripts/build-frank-primary-rad.sh # convert
# bash scripts/build-frank-primary-rad.sh --plan-only # declare, no I/O
# bash scripts/build-frank-primary-rad.sh --restage # redo staging
# bash scripts/build-frank-primary-rad.sh --force # rebuild existing
#
# WHY A STAGING DIR (the one non-obvious step): rad-convert selects its
# architecture plugin by the checkpoint's quantisation DESCRIPTOR —
# derived from config.json's quantization_config (core/format/checkpoint.cpp:
# "fp8" here). No shipped plugin claims (qwen4exp, "fp8"): the shipped
# qwen4exp plugin claims (qwen4exp, ""), the descriptor recipe-built
# containers carry (they are built from the bf16 upstream, which has no
# quantization_config at all). The match is exact with no fallback, so an
# unmodified convert refuses before declare. The descriptor is only a
# selection key; per-weight encodings are data — the checkpoint reader
# derives them from the tensors themselves (X.weight_scale_inv is the
# scale plane; checkpoint.h). Staging drops the quantization_config key
# (everything else is a relative symlink into the pristine download) so
# the convert selects the shipped plugin. EVERY weight is then copied through
# in its own encoding — block-FP8 codes + bf16 scale planes stay block-FP8 —
# except TWO value-preserving reshapes that stack-qwen38-official-fp8.py
# performs on the staging dir first, and ONE recipe rule that finishes the
# PLE table:
# * experts: the plugin maps the STACKED layout (mlp.experts.gate_up_proj
# [E,2F,H] with the _scale_inv plane at the same index) while the official
# repo ships per-expert unfused tensors — staging byte-stacks them
# (gate rows first, no requantisation).
# * PLE: the plugin's fp8 branch wants the table codes beside a ONE-ELEMENT
# bf16 scale plane; the official repo ships 128 SCALELESS fp8 shards plus
# one bf16 scalar. Staging decodes to bf16 with the checkpoint's own
# scale, and the convert's rtn rule re-encodes at that same scale —
# every original code is restored exactly and the scale plane lands in
# the form the plugin declares.
#
# WHY A CONTAINER AT ALL: a checkpoint dir has no mapped/file tiers, and a
# weight only a host kernel reads (the n-gram / PLE table, 51B params)
# cannot be served from one (GUIDE §3.4). rad-convert also bakes the
# tokenizer and the checkpoint's chat template; the runtime replaces the
# template with froggeric's (--override-chat-template).
#
# The convert runs rad-convert inside the digest-pinned radiance image
# (the host has no ROCm and needs none): /home/dank/models is mounted at
# /models so the staging dir's RELATIVE symlinks resolve inside the
# container. Output lands beside the raw download, owned by dank
# (rootless podman maps container-root to the invoking user).
set -euo pipefail
SRC="/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8"
STAGE="/home/dank/models/stage-qwen4exp-fp8"
OUT_DIR="/home/dank/models/rad"
OUT="$OUT_DIR/qwen4exp-fp8-native.rad"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-primary.env"
IMAGE="$FRANK_PRIMARY_IMAGE"
plan_only=0 restage=0 force=0
for a in "$@"; do
case "$a" in
--plan-only) plan_only=1 ;;
--restage) restage=1 ;;
--force) force=1 ;;
*) echo "unknown flag: $a" >&2; exit 2 ;;
esac
done
if [ ! -f "$SRC/config.json" ]; then
echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-primary.sh --fetch)" >&2
exit 1
fi
echo "verifying raw checkpoint first..."
bash "$SCRIPT_DIR/download-frank-primary.sh"
if [ -f "$OUT" ] && [ "$plan_only" -eq 0 ] && [ "$force" -eq 0 ]; then
echo "container already exists: $OUT"
echo "rebuild it with --force (the old file moves to ~/.AGENTTRASH)"
exit 0
fi
# ---- staging ----------------------------------------------------------------
if [ "$restage" -eq 1 ] && [ -e "$STAGE" ]; then
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-stage"
mkdir -p "$TRASH"
mv "$STAGE" "$TRASH/stage-$(date +%H%M%S)"
echo "old staging dir moved to $TRASH"
fi
if [ ! -f "$STAGE/config.json" ]; then
mkdir -p "$STAGE"
python3 - "$SRC/config.json" "$STAGE/config.json" <<'PY'
import json, sys
src, dst = sys.argv[1], sys.argv[2]
cfg = json.load(open(src))
dropped = cfg.pop("quantization_config", None)
json.dump(cfg, open(dst, "w"), indent=2)
print(f"staging config.json written (quantization_config dropped: {dropped is not None})")
PY
# The index is COPIED like config.json, never symlinked: the reshape step
# rewrites the stage's index, and a symlink would write through to the
# raw checkpoint (measured 2026-10-05: the raw index was restamped with
# the stacked layout and the raw verify failed 143/144). Shards stay
# relative symlinks — relative so the targets resolve inside the convert
# container (the stage's parent is mounted at /models).
for f in "$SRC"/*; do
name="$(basename "$f")"
case "$name" in config.json|model.safetensors.index.json) continue ;; esac
ln -sfn "../Qwen/Qwen3.8-Flash-Next-FP8/$name" "$STAGE/$name"
done
cp "$SRC/model.safetensors.index.json" "$STAGE/"
echo "staging dir ready: $STAGE ($(ls "$STAGE" | wc -l) entries)"
fi
# ---- checkpoint reshape (stacked experts + bf16 PLE table) -------------------
# stack-qwen38-official-fp8.py reshapes the staged checkpoint in place
# (idempotent) and writes the PLE scale to $STAGE/ple_scale.txt — see the
# script's docstring for why each transform is value-preserving.
echo "stacking experts + merging the PLE table (value-preserving)..."
python3 "$SCRIPT_DIR/stack-qwen38-official-fp8.py" --stage "$STAGE"
PLE_SCALE="$(tr -d '\n' < "$STAGE/ple_scale.txt")"
echo "PLE fixed scale: $PLE_SCALE"
# ---- convert ----------------------------------------------------------------
# rad-convert needs no GPU: declare resolves kernels and asks each about the
# encoding it will read — CPU-side layout questions only. (If a plugin load
# ever demands a card, add --device /dev/kfd --device /dev/dri to this run.)
# --plan-only runs the same declare phase and exits before a byte is written.
echo "rad-convert: $SRC (staged) -> $OUT"
mkdir -p "$OUT_DIR"
CONVERT_ARGS=(/models/stage-qwen4exp-fp8 -o /models/rad/qwen4exp-fp8-native.rad
--home /opt/radiance/share/radiance)
[ "$plan_only" -eq 1 ] && CONVERT_ARGS+=(--plan-only -v)
# THE ONE RECIPE RULE (see header): the PLE table arrives from staging as a
# bf16 merge of the checkpoint's own fp8 codes decoded with the checkpoint's
# own scale; rtn re-encodes it at that same scale — the codes come back
# exactly and the plugin's one-element scale plane is produced.
CONVERT_ARGS+=(--quant "blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=$PLE_SCALE")
[ "$force" -eq 1 ] && [ -f "$OUT" ] && {
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-rad"
mkdir -p "$TRASH"
mv "$OUT" "$TRASH/qwen4exp-fp8-native.rad.$(date +%H%M%S)"
echo "previous container moved to $TRASH"
}
podman run --rm \
--entrypoint rad-convert \
-v /home/dank/models:/models \
"$IMAGE" "${CONVERT_ARGS[@]}"
if [ "$plan_only" -eq 1 ]; then
echo "plan-only run complete (no container written)"
exit 0
fi
# ---- record -----------------------------------------------------------------
echo "rad-info summary:"
podman run --rm --entrypoint rad-info -v /home/dank/models:/models:ro \
"$IMAGE" "/models/rad/qwen4exp-fp8-native.rad" | head -20
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
echo "container: $OUT (${MIB} MiB; env floor $FRANK_PRIMARY_RAD_MIN_MIB MiB)"
if [ "$MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
echo "ERROR: container below the env floor — the convert was truncated" >&2
exit 1
fi
echo "build complete"
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env bash
# Build the gufo runtime that serves frank-large on Dankclaw.
#
# bash scripts/build-gufo.sh # build + install the pinned tag
# bash scripts/build-gufo.sh --verify # only check the installed tree
#
# gufo (gufo-org/gufo) is the Strix Halo inference engine: it targets
# gfx1151 specifically and has no prebuilt Linux release binaries, so the
# runtime is built from the pinned git tag and installed to
# /home/dank/GitPackages/gufo-<tag>/. Everything below is what Dankclaw
# needs on top of AMD's Linux instructions.
#
# The host already carries the ROCm 7.2.4 stack (/opt/rocm -> /etc/
# alternatives/rocm) with the HIP dev packages, so this script installs
# only the ordinary system libraries gufo links against.
#
# Two host facts this build depends on, both discovered on 2026-09-28:
#
# 1. Ubuntu's librocwmma-dev ships a TRUNCATED rocWMMA header set (the
# public headers without rocwmma/internal/) into /usr/include, where
# it shadows AMD's complete copy under /opt/rocm/include. The first
# build died on "'internal/accessors.hpp' file not found" from
# /usr/include/rocwmma/rocwmma.hpp. A -I cannot fix it: in HIP mode
# the host compilation pass searches the system directories first.
# The package has no reverse dependencies, so it is removed.
# 2. The ROCm clang picks the NEWEST /usr/include/c++ tree, which on
# Ubuntu 26.04 is GCC 16's. Its <atomic_wait.h> emits a symbol
# reference the installed libstdc++ (GLIBCXX_3.4.35, GCC 15) does not
# export, and the link fails with "undefined symbol:
# std::__detail::__notify_impl". --gcc-install-dir pins the headers to
# the same GCC the runtime library came from.
set -euo pipefail
GUFO_TAG="${GUFO_TAG:-v0.1.1}"
GUFO_REPO="${GUFO_REPO:-https://github.com/gufo-org/gufo}"
# The commit the tag resolved to when this runtime was qualified. A tag that
# moves is an unreproducible runtime, so the pin is checked, not assumed.
GUFO_COMMIT="${GUFO_COMMIT:-b0f8673470a3dcc2b5e9c9546c48a40b82b5ff31}"
SRC_DIR="${GUFO_SRC_DIR:-/home/dank/GitPackages/gufo-src}"
PREFIX="${GUFO_PREFIX:-/home/dank/GitPackages/gufo-${GUFO_TAG}}"
JOBS="${JOBS:-$(nproc)}"
# The GCC toolchain that owns the installed libstdc++ on this host.
GCC_DIR="${GCC_DIR:-/usr/lib/gcc/x86_64-linux-gnu/15}"
# check_pin — refuse a source tree that is not the qualified commit.
check_pin() {
local commit
commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
if [[ -n "$commit" && "$commit" != "$GUFO_COMMIT" ]]; then
echo "GUFO_COMMIT mismatch: $SRC_DIR is at $commit, pinned $GUFO_COMMIT" >&2
exit 1
fi
}
if [[ "${1:-}" == "--verify" ]]; then
# The pin first: a moved tag is the more urgent thing to hear about, and
# it is the check that must not be masked by a missing install.
check_pin
[[ -x "$PREFIX/bin/gufo" ]] || { echo "no gufo at $PREFIX/bin/gufo" >&2; exit 1; }
"$PREFIX/bin/gufo" diagnose | tail -n 5
exit 0
fi
echo "== system libraries gufo links against =="
sudo -n apt-get install -y -qq --no-install-recommends \
build-essential cmake ninja-build pkg-config \
libicu-dev libcurl4-openssl-dev libssl-dev libpng-dev libjpeg-dev ffmpeg
echo "== dropping Ubuntu's truncated rocWMMA headers =="
if dpkg -s librocwmma-dev >/dev/null 2>&1; then
sudo -n apt-get remove -y -qq librocwmma-dev
fi
echo "== source: $GUFO_REPO @ $GUFO_TAG =="
if [[ ! -d "$SRC_DIR/.git" ]]; then
mkdir -p "$(dirname "$SRC_DIR")"
git clone --depth 1 --branch "$GUFO_TAG" "$GUFO_REPO" "$SRC_DIR"
fi
git -C "$SRC_DIR" fetch --depth 1 origin "refs/tags/$GUFO_TAG"
git -C "$SRC_DIR" checkout -q --detach "refs/tags/$GUFO_TAG"
echo " $(git -C "$SRC_DIR" rev-parse HEAD)"
check_pin
echo "== configure + build (release, HIP gfx1151) =="
cd "$SRC_DIR"
toolchain="--gcc-install-dir=$GCC_DIR"
cmake --preset release \
-DCMAKE_INSTALL_PREFIX="$PREFIX" \
-DCMAKE_HIP_FLAGS="$toolchain" \
-DCMAKE_EXE_LINKER_FLAGS="$toolchain"
cmake --build --preset release --parallel "$JOBS"
cmake --install build/release
echo "== verify =="
"$PREFIX/bin/gufo" diagnose
echo "frank-large runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
+89
View File
@@ -0,0 +1,89 @@
#!/usr/bin/env bash
# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
#
# bash scripts/build-rocmfpx.sh # clone/fetch + build the pin
# bash scripts/build-rocmfpx.sh --verify # only check the installed tree
#
# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
# stock llama.cpp, gufo and every other loader on the host reject the file.
# This is the same tree that served the Qwen3.8-Flash-Next ROCmFP4 STRIX
# endpoint 2026-08-30 → 09-11 (see the kingjones runbook), rebuilt here so a
# fresh host can be brought up from the repo alone.
#
# Why this commit and not a newer one: 36e9acd (2026-08-28) is what was
# qualified against this payload on 2026-10-02 — it loads ftype 115, survives
# warmup, and drafts with the MTP head at acceptance 0.65. The five legacy
# builds of 2026-08-25 crashed this same file in ggml_compute_forward_set_rows
# (see kingjones-q8agent-rocmfpx.md), so the pin is load-bearing: do not bump
# it without re-running that crash test.
#
# Host notes:
# * -DLLAMA_BUILD_WEBUI=OFF is REQUIRED on this box. The tree provisions its
# embedded UI assets via npm or the HF llama-ui bucket; npm is absent here
# and the bucket download half-failed, aborting the link at 98%. The API
# server is unaffected.
# * The build links against the host ROCm (/opt/rocm); no bundled toolchain
# like the gfx1201 R9700 build needed.
set -euo pipefail
ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
# The commit frank-large was qualified with (measured 2026-10-02).
ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
GPU_TARGET="${ROCMFPX_GPU_TARGET:-gfx1151}"
JOBS="${JOBS:-$(nproc)}"
# check_pin — refuse a source tree that is not the qualified commit. A moved
# fork means different kernels under the same path, and on this model line a
# wrong build is a segfault at first request, not a clean error.
check_pin() {
local commit
commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
if [[ -n "$commit" && "$commit" != "$ROCMFPX_COMMIT" ]]; then
echo "ROCMFPX_COMMIT mismatch: $SRC_DIR is at $commit, pinned $ROCMFPX_COMMIT" >&2
exit 1
fi
}
if [[ "${1:-}" == "--verify" ]]; then
check_pin
[[ -x "$BUILD_DIR/bin/llama-server" ]] || {
echo "no ROCmFPX llama-server at $BUILD_DIR/bin/llama-server" >&2
exit 1
}
# The format the endpoint exists for: the fork must carry the ROCmFPx
# tensor type, or the payload cannot load at all.
"$BUILD_DIR/bin/llama-server" --help 2>&1 | grep -q -- '--spec-type' || {
echo "$BUILD_DIR/bin/llama-server has no --spec-type: MTP drafting is impossible" >&2
exit 1
}
"$BUILD_DIR/bin/llama-server" --version | head -1
exit 0
fi
echo "== source: $ROCMFPX_REPO @ $ROCMFPX_COMMIT =="
if [[ ! -d "$SRC_DIR/.git" ]]; then
mkdir -p "$(dirname "$SRC_DIR")"
git clone "$ROCMFPX_REPO" "$SRC_DIR"
fi
git -C "$SRC_DIR" fetch --quiet origin "$ROCMFPX_COMMIT" || true
git -C "$SRC_DIR" checkout -q --detach "$ROCMFPX_COMMIT"
check_pin
echo " $(git -C "$SRC_DIR" rev-parse HEAD)"
echo "== configure + build (release, HIP $GPU_TARGET) =="
cmake -S "$SRC_DIR" -B "$BUILD_DIR" \
-DGGML_HIP=ON \
-DGPU_TARGETS="$GPU_TARGET" \
-DGGML_NATIVE=ON \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_BUILD_WEBUI=OFF
cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
echo "== verify =="
"$BUILD_DIR/bin/llama-server" --version | head -1
echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
+110
View File
@@ -0,0 +1,110 @@
#!/usr/bin/env python3
"""calib-push.py — drive a calibration corpus through a radiance scratch
server's prefill so its tap writes layer Hessians (docs/MOE-W4.md).
Runs ON DANKSERVER next to the scratch server (the one autoresearch- /
serve-style scratch container booted with RADIANCE_CALIB_DIR set).
Reads the corpus JSONL produced by make-calib-corpus.py
({"prompt": "..."} a line), POSTs each prompt to /v1/completions with
max_tokens=1 (prefill-only cost, one token out), at N-way concurrency
matching the scratch server's --max-num-seqs budget.
Corpus rounds repeat until the engine prints the tap's completion line
("calibration wrote layer Hessians") in `podman logs <container>` — that
is the ground truth that RADIANCE_CALIB_TOKENS was reached. After that
line, further prompts are pure waste, so the driver exits.
Mechanics notes:
- requests carry no sampling params that matter (max_tokens=1, greedy).
- a per-request timeout of 30 min: at ~1k tok/s prefill a 12k-token
prompt is a few seconds; the timeout only catches a wedged engine.
- failures print and continue: the tap pools over tokens, a few lost
prompts do not bias the Hessians (the corpus is drawn round-robin over
source classes, so loss is class-neutral as long as failures are rare;
a systematic 4xx is loud in the log and must be fixed, not ignored).
Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
[--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
"""
import argparse
import concurrent.futures
import json
import subprocess
import sys
import time
import urllib.request
DONE_MARK = "calibration wrote"
def tap_done(container):
"""True if the engine logged the tap's completion line."""
try:
out = subprocess.run(
["podman", "logs", "--tail", "50", container],
capture_output=True, text=True, timeout=30,
)
except Exception:
return False
return DONE_MARK in (out.stdout + out.stderr)
def push_one(url, prompt):
body = json.dumps({
"model": "frank-archon",
"prompt": prompt,
"max_tokens": 1,
"temperature": 0,
"stream": False,
}).encode()
req = urllib.request.Request(
url, data=body, headers={"Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=1800) as r:
r.read()
return True
except Exception as e:
print(f"FAIL {str(e)[:120]}", flush=True)
return False
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--port", type=int, default=7770)
ap.add_argument("--parallel", type=int, default=64)
ap.add_argument("--container", default="arbench")
ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
ap.add_argument("--max-rounds", type=int, default=4)
a = ap.parse_args()
url = f"http://127.0.0.1:{a.port}/v1/completions"
prompts = []
with open(a.corpus) as f:
for line in f:
line = line.strip()
if line:
prompts.append(json.loads(line)["prompt"])
print(f"corpus: {len(prompts)} prompts", flush=True)
for rnd in range(1, a.max_rounds + 1):
if tap_done(a.container):
break
t0 = time.time()
ok = 0
with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
for good in ex.map(lambda p: push_one(url, p), prompts):
ok += 1 if good else 0
dt = time.time() - t0
print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
flush=True)
if tap_done(a.container):
break
else:
print("max rounds exhausted, tap not done", flush=True)
sys.exit(1)
print("TAP_DONE: calibration wrote layer Hessians", flush=True)
if __name__ == "__main__":
main()
+73
View File
@@ -0,0 +1,73 @@
#!/usr/bin/env bash
# convert-swift15-w4dn8.sh — RUNS ON DANKSERVER. The final recipe convert of
# the Swift 1.5 checkpoint: gate_up experts through GPTQ against the 10M-token
# calibration Grams, down projections block-fp8, protected experts bf16,
# trunk int8, PLE fixed-scale fp8 (scripts/recipes/
# qwen4exp-swift15-w4dn8-i8-hc8m.recipe).
#
# The convert engine is 1.0.13 — the SAME pin that will serve the result, so
# the .rad crosses no version boundary. Only the Gram files cross versions
# (1.2.0 wrote them; the RADGRAM2 magic, gram.r%d.%s.bin naming and the
# calib_gram/calib_resolve/calib_stem symbols are byte-identical between the
# two libquant.so). rad-convert runs on the host backend (no GPU visible in
# the container) — production can stay up through this script.
#
# Prereqs (campaign order): calibration complete (calib-push printed TAP_DONE),
# the protected-expert cast lines rewritten into the recipe by
# derive-protected-experts.py, and the 335 GiB bf16 container deleted —
# the peak-disk arithmetic needs that space (checkpoint 335.3 + output ~132
# GiB on the volume; --reuse is NOT used, see the recipe header).
set -euo pipefail
PIN_1013=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
IMAGE="stilldeadcode/radiance@$PIN_1013"
CKPT=/home/dank/models/ukisai/Swift1.5-Qwen3.8-Flash-Next
RECIPE=/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe
CALIB=/home/dank/corpora/calib
OUT=/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad
LOG=/home/dank/logs/swift15_convert.log
# gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert table;
# the plugin's one-encoding law dropped the block-fp8 down tier -- see the recipe header).
MIN_MIB=${FRANK_SWIFT15_RAD_MIN_MIB:-108000}
# Gate 1: inputs. A missing Gram directory converts UNCALIBRATED with a
# per-tensor stderr note, which is silently worse than failing here.
[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT" >&2; exit 1; }
[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 1; }
GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
[ "$GRAMS" -ge 48 ] || { echo "calib dir $CALIB holds $GRAMS gram files (<48)" >&2; exit 1; }
if [ -e "$OUT" ]; then echo "$OUT already exists; move it to trash first" >&2; exit 1; fi
if [ -e /home/dank/models/rad/qwen4exp-swift15-bf16.rad ]; then
echo "the bf16 container is still on disk; delete it first (peak disk)" >&2; exit 1
fi
# Gate 2: disk. Output ~132 GiB plus convert staging, 20 GiB of slack.
FREE_MIB=$(df --output=avail -m /home/dank/models | tail -1 | tr -d ' ')
[ "$FREE_MIB" -ge $((MIN_MIB + 20480)) ] || {
echo "only ${FREE_MIB} MiB free on /home/dank/models" >&2; exit 1; }
# The recipe names $CALIB; rad-convert expands it from the environment
# (MOE-W4.md:275). The recipe's cast section must name THIS checkpoint's
# protected experts — derive-protected-experts.py rewrote it.
echo "convert: $CKPT -> $OUT (gptq calib=$CALIB, $(date -Is))" | tee "$LOG"
podman run --rm --name swift15-convert \
-v /home/dank/models:/models \
-v /home/dank/serverAppConfigs-swift15:/campaign:ro \
-v "$CALIB:/data/calib:ro" \
-e CALIB=/data/calib \
--entrypoint /opt/radiance/bin/rad-convert \
"$IMAGE" "/models/${CKPT#/home/dank/models/}" \
--recipe /campaign/"${RECIPE#/home/dank/serverAppConfigs-swift15/}" \
-o /models/rad/"$(basename "$OUT")" 2>&1 | tee -a "$LOG"
rc=${PIPESTATUS[0]}
[ "$rc" -eq 0 ] || { echo "rad-convert rc=$rc; log: $LOG" >&2; exit "$rc"; }
# Gate 3: the artifact. rad-info's recipe echo is the receipt that GPTQ ran.
SIZE_MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
[ "$SIZE_MIB" -ge "$MIN_MIB" ] || {
echo "$OUT is ${SIZE_MIB} MiB < floor ${MIN_MIB}" >&2; exit 1; }
podman run --rm --entrypoint /opt/radiance/bin/rad-info \
-v /home/dank/models:/models:ro "$IMAGE" /models/rad/"$(basename "$OUT")" \
2>&1 | tee -a "$LOG" | grep -c 'gptq' >/dev/null \
|| echo "WARNING: rad-info shows no gptq rule in the recorded recipe" >&2
echo "CONVERT_DONE ${SIZE_MIB} MiB -> $OUT"
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
# Single source of truth for Dankserver GLM-5.3 state.
#
# This exists because the ad-hoc status commands kept producing WRONG readings.
# The specific trap: /proc/meminfo separates key from value with a TAB, so the
# common idiom
# grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
# yields an empty field (tab is not a space) and awk then prints 0.0 -- which
# reads as "host out of memory" when the host actually has 240 GiB free.
# Everything below parses on a colon split, which is tab-agnostic, and the
# memory figure is cross-checked before it is shown.
set -uo pipefail
MB=1048576
GLM_CTR="${GLM_CTR:-glm-w4a16}"
mem_field() { awk -v k="$1:" '$1==k {printf "%.1f", $2/'"$MB"'; exit}' /proc/meminfo; }
section() { printf '\n== %s ==\n' "$1"; }
section "time / host"
printf ' %s %s\n' "$(date +%H:%M:%S)" \
"$(awk -F: '/model name/{print $2; exit}' /proc/cpuinfo | sed 's/^ *//')"
section "memory (GiB)"
TOT=$(mem_field MemTotal); AVAIL=$(mem_field MemAvailable)
printf ' MemTotal %8s\n' "$TOT"
printf ' MemAvailable %8s\n' "$AVAIL"
printf ' Cached %8s\n' "$(mem_field Cached)"
awk -v t="$TOT" -v a="$AVAIL" 'BEGIN{
if (t+0 <= 0 || a+0 <= 0) { print " *** PARSE BUG: zero value ***"; exit }
if (a+0 > t+0) { print " *** PARSE BUG: available > total ***"; exit }
printf " check ok (%.0f%% free)\n", 100*a/t
}'
section "swap"
swapon --show=NAME,SIZE,USED --noheadings 2>/dev/null \
| awk '{printf " %-12s %6s used of %s\n",$1,$3,$2}' || echo " none"
section "gpu"
# CSV layout: device,VRAM Total Memory (B),VRAM Total Used Memory (B)
rocm-smi --showmeminfo vram --csv 2>/dev/null | awk -F, '
NR>1 && $1 ~ /^card[0-9]+$/ {
printf " %-8s %7.2f GiB used of %.2f\n", $1, $3/1073741824, $2/1073741824
}'
printf ' compute util: '
rocm-smi --showuse --csv 2>/dev/null | awk -F, 'NR>1 && $1 ~ /^card[0-9]+$/ {printf "%s=%s%% ", $1, $2}'
echo
section "disk"
df -h /home/dank 2>/dev/null | awk 'NR==2{printf " %s %s used %s free (%s)\n",$1,$3,$4,$5}'
section "llm containers"
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qE 'glm|frank'; then
podman ps -a --format '{{.Names}}|{{.Status}}' 2>/dev/null \
| awk -F'|' '$1 ~ /glm|frank/ {printf " %-18s %s\n",$1,$2}'
else
echo " none"
fi
section "frank-primary unit"
printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
section "glm serve tail"
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
podman logs "$GLM_CTR" 2>&1 | grep -viE 'multiproc_executor\.py:943' | tail -5 | sed 's/^/ /'
else
echo " (container $GLM_CTR not present)"
fi
section "done"
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env python3
"""deep-probe.py — one chat completion at ~43.6k prompt tokens, temp 0, 512
generated, streamed; decode = completion_tokens / (total - ttft)."""
import json, time, urllib.request
TARGET_BYTES = int(43600 * 3.8)
def build():
parts, n = [], 0
with open("/home/dank/corpora/out/calib.jsonl") as f:
for line in f:
p = json.loads(line)["prompt"]
parts.append(p)
n += len(p.encode())
if n >= TARGET_BYTES:
break
return "".join(parts)
def main(salt):
body = json.dumps({
"model": "frank-archon",
"messages": [{"role": "user", "content": f"[probe {salt}] Continue and summarize this material in English, 400 words.\n\n" + build()}],
"temperature": 0.0, "max_tokens": 512,
"stream": True, "stream_options": {"include_usage": True},
}).encode()
req = urllib.request.Request("http://localhost:7770/v1/chat/completions",
data=body, headers={"Content-Type": "application/json"})
t0 = time.time(); ttft = None; ctok = 0; usage = {}
with urllib.request.urlopen(req, timeout=2400) as r:
for raw in r:
line = raw.decode().strip()
if not line.startswith("data:"):
continue
d = line[5:].strip()
if d == "[DONE]":
break
j = json.loads(d)
if j.get("usage"):
usage = j["usage"]
for ch in j.get("choices", []):
delta = (ch.get("delta") or {})
piece = delta.get("content") or delta.get("reasoning_content") or ""
if piece and ttft is None:
ttft = time.time() - t0
ctok += 1
total = time.time() - t0
gen = usage.get("completion_tokens") or ctok
dec = gen / (total - (ttft or 0)) if total > (ttft or 0) else 0
print(json.dumps({"salt": salt, "prompt_tokens": usage.get("prompt_tokens"),
"completion_tokens": gen, "ttft_s": round(ttft or 0, 2),
"total_s": round(total, 1), "decode_tps": round(dec, 2)}))
if __name__ == "__main__":
import sys
main(sys.argv[1] if len(sys.argv) > 1 else "x")
+6 -6
View File
@@ -62,7 +62,7 @@ SERVICE_ORDER = (
"lidarr",
"readarr",
"frank-large",
"frank-small",
"frank-secondary",
"frank-primary",
)
@@ -94,7 +94,7 @@ DANKCLAW_SERVICES = {
"caddy",
"homepage",
"frank-large",
"frank-small",
"frank-secondary",
}
DANKSERVER_SERVICES = {
@@ -400,12 +400,12 @@ def services_for_path(path: str, target: str) -> set[str]:
# change to it redeploying alertmanager is the only path that
# reaches it on the host.
_add(services, "alertmanager")
if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
# llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-small only. The ik_llama.cpp
if name in {"frank-secondary.env", "frank-secondary.service", "serve-frank-secondary.sh"}:
# llama.cpp frank-secondary (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-secondary only. The ik_llama.cpp
# trio that lived here for two days is in scripts/.AGENTTRASH/,
# which is archive material and plans nothing.
_add(services, "frank-small")
_add(services, "frank-secondary")
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
# CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
# serve files here the same way: a root unit on a host-built
+223 -121
View File
@@ -90,7 +90,7 @@ service_is_allowed() {
dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
dankclaw:frank-large|dankclaw:frank-small|\
dankclaw:frank-large|dankclaw:frank-secondary|\
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
@@ -322,6 +322,12 @@ deploy_homepage() {
if [[ -d shared/homepage/fonts ]]; then
run_privileged cp -r shared/homepage/fonts /srv/homepage/
fi
if [[ -d shared/homepage/img ]]; then
run_privileged cp -r shared/homepage/img /srv/homepage/
fi
if [[ -d shared/homepage/sfx ]]; then
run_privileged cp -r shared/homepage/sfx /srv/homepage/
fi
local deployed_at
deployed_at="$(git log -1 --format=%cI)"
printf 'window.DEPLOYED_AT = "%s";\n' "$deployed_at" > /tmp/deployed.js
@@ -935,15 +941,24 @@ deploy_plex() {
fi
systemctl --user restart plex.service
# Readiness gate: /identity answers MediaContainer XML once PMS is up.
#
# HTTPS, NOT HTTP. PMS drops plain HTTP off-loopback: the probe's
# connection arrives on the container's eth0, so an http:// request is
# accepted and then answered with nothing (curl 52, Empty reply) against a
# perfectly healthy server. Measured 2026-10-02 — the gate burned its full
# 600s and failed a plex whose https://127.0.0.1:32400/identity answered
# 200 in 44 ms the whole time. The same law is why the Caddyfile dials the
# plex upstream over TLS (tests/caddyfile.bats pins it).
# -k: PMS presents its own self-signed cert on 32400.
#
# Reality check from the 2026-09 cutover: the FIRST container boot of a
# years-old library ran ~10 min (PMS pre-binds 32400 during boot and
# empty-replies until the HTTP layer is ready), so the gate allows 600s.
# years-old library ran ~10 min, so the gate still allows 600s.
# Fail the deploy (pending state preserved) rather than mark it deployed blind.
for ((i = 1; i <= 120; i++)); do
curl -fsS --max-time 3 http://127.0.0.1:32400/identity >/dev/null && return 0
curl -fsSk --max-time 3 https://127.0.0.1:32400/identity >/dev/null && return 0
sleep 5
done
echo "FAIL: plex did not answer /identity within 600s" >&2
echo "FAIL: plex did not answer /identity over HTTPS within 600s" >&2
return 1
}
@@ -1077,20 +1092,37 @@ deploy_anubis() {
wait_container_healthy anubis 60 || return 1
}
# frank-large is the stock llama.cpp prebuilt (unsloth b11030, ROCm
# gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2
# drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU,
# as a ROOT unit since the 2026-09-19 cutover: the runtime is a prebuilt
# binary, so there is no container image to install and no generated unit
# to dodge. relink_unit points /etc/systemd/system/frank-large.service at
# the repo file and boot persistence is that unit's [Install] section.
# frank-large is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
# through scripts/frank-large.env) serving the Quat3rnion ABLITERATED
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint —
# qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, ~62 GiB
# resident) + the REQUIRED n-gram lookup table + the vision tower —
# MTP spec decode (HALOGEN_SPEC_ADAPT=0), 2 slots x 200192, image input —
# on the Strix Halo iGPU. It stays a ROOT unit because the serve script
# drives `podman run` directly (the frank-primary pattern): relink_unit
# points /etc/systemd/system/frank-large.service at the repo file, boot
# persistence is that unit's [Install], and LimitMEMLOCK=infinity is
# fleet parity. 2026-10-06 cutover from the halogen 27B-dense shape —
# rollback = revert the swap commit's trio; the 27B payload stays on
# disk (~/models/halogen-qwen3.8-27b).
#
# It replaces the Kairic Edge runtime (ciru-ai/ROCmFPX), which had
# replaced the unsloth llama.cpp prebuilt serving
# unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS. The port gate is the same as
# every cutover since 2026-09-16: nothing starts until 7770 is released.
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}"
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
# The gates mirror serve-frank-large.sh's predicates EXACTLY (the
# 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
# each .hgn payload (checkpoint, n-gram table, vision tower) at the exact
# byte count, the tokenizer directory holding every named file as a
# regular (not symlinked) file — a green deploy against a missing
# artifact would leave a restart-looping unit and a dark endpoint. The
# port gate is the same as every cutover since 2026-09-16: nothing
# starts until 7770 is released.
LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-66687678464}"
LLM_LARGE_NGRAM="${LLM_LARGE_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
LLM_LARGE_NGRAM_BYTES="${LLM_LARGE_NGRAM_BYTES:-51200246144}"
LLM_LARGE_VISION="${LLM_LARGE_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
LLM_LARGE_VISION_BYTES="${LLM_LARGE_VISION_BYTES:-897916416}"
LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
retire_halogen_large_quadlet() {
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1123,34 +1155,65 @@ wait_port_free() {
}
deploy_llm_large() {
# Refuse rather than restart-loop. A unit whose runtime or payload is
# missing never becomes healthy, and a green deploy would leave the
# endpoint dark with nothing in the deploy log to say why.
local missing=0 f
for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
if [[ ! -f "$f" ]]; then
echo "ERROR: payload absent at $f" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large.sh" >&2
missing=1
# Refuse rather than restart-loop; every gate below has a twin in
# serve-frank-large.sh and they must agree (2026-10-01 lesson).
# Capture podman output before grepping: `podman images | grep -q`
# races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
if ! grep -qF "${LLM_LARGE_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
echo "ERROR: halogen image absent: $LLM_LARGE_IMAGE" >&2
echo "frank-large: pull it: podman pull ${LLM_LARGE_IMAGE%@*}" >&2
return 1
fi
if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
return 1
fi
if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
return 1
fi
for gate in "$LLM_LARGE_NGRAM:$LLM_LARGE_NGRAM_BYTES:n-gram table" \
"$LLM_LARGE_VISION:$LLM_LARGE_VISION_BYTES:vision tower"; do
IFS=: read -r gpath gbytes gname <<<"$gate"
if [[ ! -f "$gpath" ]]; then
echo "ERROR: halogen $gname absent at $gpath" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
return 1
fi
if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
return 1
fi
done
if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
return 1
fi
for tf in $LLM_LARGE_TOK_FILES; do
if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
return 1
fi
done
[[ "$missing" -eq 0 ]] || return 1
retire_halogen_large_quadlet
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
# the port, or the wait deadlocks on the server it just stopped serving.
# Then sweep a stray manual run of the SAME model -- that is how a bench
# harness leaves the port busy. Match the model path, not the binary name:
# frank-small on this host is also a llama-server and must not be killed.
# The pattern never appears in the deploying shell's own command line, so
# -f is safe here (it is not in `bash deploy-remote.sh`).
# The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
# the container) before waiting on the port. The stray sweep is keyed
# on the FULL container name, which appears in the argv of a manual
# `podman run --name frank-large-halogen` / `podman wait` but NOT in
# this deployer's own command line (`bash deploy-remote.sh
# frank-large`) — so it cannot shoot itself in the foot (the fleet
# pkill law), and it can never match frank-secondary's llama-server.
run_privileged systemctl stop frank-large.service 2>/dev/null || true
pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true
# Nothing may start until the predecessor has released the port. On
# 2026-09-16 the cutover failed exactly here: the old server logged
# "Deactivated successfully", the next engine tried to bind 7770 a second
# later and reported "Address already in use". Gate on the port, not on
# hope.
pkill -f 'frank-large-halogen' 2>/dev/null || true
podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
# Nothing may start until the predecessor has released the port
# (2026-09-16: "Deactivated successfully" was not "port free").
wait_port_free 7770 || return 1
relink_unit frank-large
run_privileged systemctl daemon-reload
@@ -1159,109 +1222,132 @@ deploy_llm_large() {
wait_for_health frank-large http://127.0.0.1:7770/health 1200
}
# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
# ROOT unit: the runtime is a native CUDA binary, so there is no container
# image to install and no generated unit to avoid. relink_unit points
# /etc/systemd/system/frank-small.service at the repo file and boot persistence
# is that unit's [Install] section.
# deploy_llm_secondary — frank-secondary is llama.cpp (master build, CUDA sm_86) on
# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
# ~/GitPackages/llama-master/build-cuda, so there is no container image and
# no generated unit to avoid. relink_unit points
# /etc/systemd/system/frank-secondary.service at the repo file and boot
# persistence is that unit's [Install] section.
#
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
# not shared: frank-large's engine runs on the gfx1151 iGPU.
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
# The projector IS the vision capability here: the model is natively
# multimodal and the Q6_K file carries no vision tensors — the publisher ships
# its own BF16 projector in the same repo, so a deploy that starts without
# this file silently ships a text-only endpoint. Checked unconditionally, like
# frank-large's.
LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
deploy_llm_small() {
if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
echo "frank-small: refusing to start a unit that cannot load." >&2
deploy_llm_secondary() {
# Refuse rather than restart-loop: a unit whose runtime or payload is
# missing never becomes healthy, and a green deploy would leave the
# endpoint dark with nothing in the log to say why.
if [[ ! -x "$LLM_SECONDARY_RUNTIME" ]]; then
echo "ERROR: llama.cpp runtime absent at $LLM_SECONDARY_RUNTIME" >&2
echo "frank-secondary: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
echo "frank-secondary: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
echo "frank-secondary: && cmake --build build-cuda --target llama-server) first." >&2
return 1
fi
relink_unit frank-small
for f in "$LLM_SECONDARY_MODEL_FILE" "$LLM_SECONDARY_MMPROJ"; do
if [[ ! -f "$f" ]]; then
echo "ERROR: frank-secondary payload file absent at $f" >&2
echo "frank-secondary: run scripts/download-frank-secondary.sh first;" >&2
echo "frank-secondary: refusing to start a unit that cannot load." >&2
return 1
fi
done
relink_unit frank-secondary
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-small.service
run_privileged systemctl restart frank-small.service
wait_for_health frank-small http://127.0.0.1:7771/health 900
run_privileged systemctl enable frank-secondary.service
run_privileged systemctl restart frank-secondary.service
wait_for_health frank-secondary http://127.0.0.1:7771/health 900 || return 1
# False-green hardening (frank-large's lesson): the port answering
# proves nothing about WHO answers — the unit must be active with a
# live MainPID, or a stray SGLang holdover is masquerading as deployed.
local main_pid
main_pid="$(run_privileged systemctl show -p MainPID --value frank-secondary.service)"
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
echo "FAIL: :7771 answered but frank-secondary.service has no MainPID — a stray process holds the port" >&2
return 1
fi
echo "frank-secondary: health answered by unit MainPID $main_pid"
}
# deploy_llm_primary — frank-primary is the R9V radiance-vLLM fork (docker-
# pinned prebuilt, gfx1201, driven through the docker->podman shim) serving
# the unsloth Qwen3.8-Flash-Next Q8_0 payload (6 GGUF shards) with the MTP4
# drafter on the two R9700s as a ROOT unit whose serve script execs the
# fork's launcher through rootless podman as dank. Same port and alias as
# the endpoints it replaced (the 2026-09-27 cutover from uncensored
# IQ4_XS); the harness models.yml id is unchanged. The uncensored state dir
# (/home/dank/r9v-state/uncensored) stays on disk as the rollback target.
# deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode,
# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
# ROOT unit whose serve script drives `podman run` directly (the engine is
# the image entrypoint; the image lives in dank's rootless podman storage).
# 2026-10-05 cutover from FreeToken (pre-cutover tag
# frank-primary-freetoken-pr587-20261005); before that, the tcclaviger vLLM
# fork (tag frank-primary-vllm-shapeG-20261003). Same port and alias; the
# harness models.yml id is unchanged.
#
# Gates: the on-host R9V setup state must exist and be ready (it carries the
# pinned image digest and every payload path), the pinned runtime image must
# be loadable from dank's rootless storage by its OCI manifest digest, the
# PLE payload must match its pinned byte count, and every GGUF shard + the
# mmproj must exist (per-file sha256 verification was setup's job at fetch
# time, never re-hashed here).
LLM_PRIMARY_STATE=/home/dank/r9v-state/q8xl/setup.json
LLM_PRIMARY_CONTAINER=frank-primary-r9v
# Gates: the image must exist under its digest ref in dank's rootless
# storage (FRANK_PRIMARY_IMAGE), the .rad container file must exist at the
# env's size floor (FRANK_PRIMARY_MODEL_FILE >= FRANK_PRIMARY_RAD_MIN_MIB —
# a truncated convert is a guaranteed long-timeout boot), and the unit env
# must be complete. Per-file sha256 verification is the fetch script's job
# at download time and rad-convert/rad-info's job at build time, never
# re-hashed here.
LLM_PRIMARY_CONTAINER=frank-primary
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
deploy_llm_primary() {
# Refuse rather than restart-loop: a unit whose image or payload is
# missing never becomes healthy, and a green deploy would leave the
# endpoint dark with nothing in the log to say why.
if [[ ! -f "$LLM_PRIMARY_STATE" ]]; then
echo "ERROR: R9V setup state absent: $LLM_PRIMARY_STATE" >&2
echo "frank-primary: run the profile setup on this host first" >&2
local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
local image model_dir model_file rad_min mib
if [[ ! -f "$env_file" ]]; then
echo "ERROR: frank-primary env file absent: $env_file" >&2
return 1
fi
local state_json image_digest model_dir ple_path ple_bytes
state_exports="$(python3 - "$LLM_PRIMARY_STATE" <<'PY'
import json, shlex, sys
state = json.load(open(sys.argv[1]))
config = state.get("config") or {}
if state.get("ready") is not True:
sys.exit("R9V setup state is not ready")
for key in ("R9V_IMAGE", "R9V_MODEL_DIR", "R9V_PLE_PATH", "R9V_PLE_EXPECTED_BYTES",
"R9V_TARGET_REL", "R9V_TARGET_SHARD2_REL", "R9V_TARGET_SHARD3_REL",
"R9V_TARGET_SHARD4_REL", "R9V_TARGET_SHARD5_REL", "R9V_TARGET_SHARD6_REL",
"R9V_MMPROJ_REL"):
print(f"{key}={shlex.quote(config.get(key, ''))}")
PY
)" || return 1
eval "$state_exports"
if [[ -z "$R9V_IMAGE" || -z "$R9V_MODEL_DIR" || -z "$R9V_PLE_PATH" ]]; then
echo "ERROR: R9V setup state is missing image/model/PLE paths" >&2
image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
model_file="$(sed -n 's/^FRANK_PRIMARY_MODEL_FILE=//p' "$env_file" | tail -1)"
rad_min="$(sed -n 's/^FRANK_PRIMARY_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
return 1
fi
if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
echo "ERROR: pinned R9V runtime image absent from rootless storage: $R9V_IMAGE" >&2
echo "frank-primary: load the image bundle (./r9v setup ...)" >&2
# The image gate is the digest ref itself: a name@sha256 ref exists in
# storage only if THAT manifest was pulled. One command — no
# pipefail/grep race (the 2026-09-30 lesson).
if ! podman image exists "$image"; then
echo "ERROR: runtime image absent from storage: $image" >&2
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest" >&2
return 1
fi
if [[ -n "$R9V_PLE_EXPECTED_BYTES" ]]; then
ple_bytes="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
if [[ "$ple_bytes" != "$R9V_PLE_EXPECTED_BYTES" ]]; then
echo "ERROR: PLE payload wrong size: $R9V_PLE_PATH (expected $R9V_PLE_EXPECTED_BYTES, found $ple_bytes)" >&2
echo "frank-primary: re-run the PLE derivation (./r9v setup ...)" >&2
return 1
fi
if [[ ! -f "$model_dir/$model_file" ]]; then
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
return 1
fi
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
if [[ "$mib" -lt "$rad_min" ]]; then
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
echo "frank-primary: rebuild it (scripts/build-frank-primary-rad.sh --force)" >&2
return 1
fi
local rel
for rel in "$R9V_TARGET_REL" "${R9V_TARGET_SHARD2_REL:-}" "${R9V_TARGET_SHARD3_REL:-}" \
"${R9V_TARGET_SHARD4_REL:-}" "${R9V_TARGET_SHARD5_REL:-}" "${R9V_TARGET_SHARD6_REL:-}" \
"$R9V_MMPROJ_REL"; do
[[ -z "$rel" ]] && continue
if [[ ! -f "$R9V_MODEL_DIR/$rel" ]]; then
echo "ERROR: payload file missing: $R9V_MODEL_DIR/$rel" >&2
echo "frank-primary: re-run the profile setup to fetch/verify the payload" >&2
return 1
fi
done
# A stale container with the same name makes `podman run --name` fail at
# start; remove any leftover from a failed previous attempt.
podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
# A stopped unit can leave its container running: systemd's cgroup kill
# races conmon's --rm cleanup, and the orphan holds the port and GPUs.
# Stop the unit, then remove the container BEFORE waiting on the port.
# Stop the unit, then remove the containers BEFORE waiting on the port.
# The legacy identities (the retired R9V container, the ad hoc
# experiment container qwen-fp8, the retired glm-probe) hold the same
# GPUs and die here too.
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
local ctr
for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
podman rm -f "$ctr" >/dev/null 2>&1 || true
done
# Nothing starts until the port is released: the lucebox cutover
# (2026-09-26) and the frank-large cutover (2026-09-16) both failed
# exactly here when the next engine bound 7770 a second too early.
@@ -1270,10 +1356,26 @@ PY
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-primary.service
run_privileged systemctl restart frank-primary.service
# Cold start: the launcher's preflight doctor plus payload page-in
# (~93 GiB cold; expert cache fills after /health). Parity with every
# predecessor on this port.
wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
# Cold start: .rad load + PLE pin + expert host-pool upload (the
# publisher's compose healthcheck allows a 15-minute start_period).
# Parity with every predecessor on this port. The HTTP listener binds
# AFTER the full load (main.cpp: engine.start() then engine.serve()),
# so during load the port REFUSES connections — a bare curl is honest
# here, unlike FreeToken's 200-while-loading body. (The MainPID check
# below still guards the stray-port case.)
local i ok=0
for ((i=1; i<=900; i++)); do
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
echo "frank-primary engine healthy after approximately $((i * 2))s"
ok=1
break
fi
sleep 2
done
if [[ "$ok" != 1 ]]; then
echo "FAIL: frank-primary /health never answered 200 in 1800s" >&2
return 1
fi
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
# port cannot tell WHO answers — a stray probe holding :7770 satisfies
# it while the unit is stopped. The deploy is only green if the unit
@@ -1472,7 +1574,7 @@ deploy_service() {
seerr) deploy_seerr ;;
marreta) deploy_marreta ;;
frank-large) deploy_llm_large ;;
frank-small) deploy_llm_small ;;
frank-secondary) deploy_llm_secondary ;;
frank-primary) deploy_llm_primary ;;
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
+27 -2
View File
@@ -141,7 +141,7 @@ valid_service() {
dankclaw:alertmanager|\
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
dankclaw:frank-large|dankclaw:frank-small|\
dankclaw:frank-large|dankclaw:frank-secondary|\
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
@@ -196,6 +196,21 @@ else
printf ' - %s\n' "${PLAN[@]}"
fi
# --- frankbook e2e trigger --------------------------------------------------
# The *arr/qBittorrent e2e units walk links that terminate on FrankBook-Pro
# (192.168.0.112), which sleeps when idle. The [3/3] e2e gate therefore runs
# the frankbook Playwright project ONLY when something in its blast radius
# changed since origin/main: the frankbook spec itself, the frankbook-pro
# stack, the homepage registry, or the Caddyfile serving the arr vhosts.
# Every other deploy skips it (tests/e2e/run.sh excludes the project by
# default). See tests/e2e/specs/19-frankbook.spec.ts.
FB_E2E_TRIGGER=0
while IFS= read -r fb_path; do
case "$fb_path" in
tests/e2e/specs/19-frankbook.spec.ts|frankbook-pro/*|shared/homepage/services.json|dankclaw/Caddyfile) FB_E2E_TRIGGER=1 ;;
esac
done < <(git diff --name-only "$BASE_REVISION" "$HEAD_REVISION")
REMOTE="${REMOTE_USER}@${REMOTE_HOST}"
REMOTE_ARGS=("$TARGET" "$HEAD_REVISION" "$REMOTE_DIR" "$REMOTE_SHELL")
if [[ "${#EXPLICIT_SERVICES[@]}" -eq 0 ]]; then
@@ -209,6 +224,11 @@ if [[ "$DRY_RUN" -eq 1 ]]; then
printf '[dry-run] Runner arguments:'
printf ' %q' "${REMOTE_ARGS[@]}"
printf '\n'
if [[ "$FB_E2E_TRIGGER" -eq 1 ]]; then
echo "[dry-run] e2e gate: frankbook project INCLUDED (frankbook trigger paths changed)."
else
echo "[dry-run] e2e gate: frankbook project EXCLUDED (no frankbook trigger paths)."
fi
exit 0
fi
@@ -250,7 +270,12 @@ if [[ "$SKIP_E2E" -eq 1 ]]; then
echo "[3/3] e2e gate SKIPPED (--skip-e2e): live fleet NOT verified." >&2
else
echo "[3/3] Running Playwright e2e gate against the live fleet..."
if bash "$REPO_ROOT/tests/e2e/run.sh"; then
E2E_ENV=()
if [[ "$FB_E2E_TRIGGER" -eq 1 ]]; then
echo " frankbook trigger paths changed: including the frankbook e2e project."
E2E_ENV=(E2E_INCLUDE_FRANKBOOK=1)
fi
if env "${E2E_ENV[@]}" bash "$REPO_ROOT/tests/e2e/run.sh"; then
echo "e2e gate PASSED."
else
echo "=== DEPLOY INCOMPLETE: e2e gate FAILED on $TARGET ===" >&2
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env python3
"""derive-protected-experts.py — rewrite the recipe's protected-expert section
from a calibration recording. RUNS ON DANKSERVER (venv: numpy).
The law (docs/MOE-W4.md "Protected experts", restated in the recipe):
an expert whose share of its layer's down-input energy is at least 3% AND
at least three times its share of the routing is kept bf16; the count must
be EVEN on each rank's half of the layer (the four-bit GEMM takes the rest
as two equal tables by parity) — an odd half gains its next most energetic
non-qualifying expert.
Data: the calibration dir's per-expert down Grams,
gram.r<rank>.blk.L.ffn_down_exps.<e>.bin — 32-byte RADGRAM2 header
(magic, k, rank, rows, domain) then k*k f32, named with the checkpoint's
GLOBAL expert numbering (arch/common/rad_block_moe_fp8.h: ge =
expert_base + e). rows (header) is the expert's routing count; the trace
of the Gram is its down-input energy. Rank halves: 0-255, 256-511
(contiguous expert_base split at TP=2).
Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
[--min-route-mult 3.0]
"""
import argparse
import glob
import os
import re
import struct
import sys
import numpy as np
NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
def read_gram(path):
"""(k, rank, rows, trace) from a RADGRAM2 file (full matrix on disk)."""
with open(path, "rb") as f:
hdr = f.read(32)
magic, k, rank, rows, domain = struct.unpack("<8sIIQI", hdr[:28])
if magic != b"RADGRAM2":
raise SystemExit(f"not a RADGRAM2 file: {path}")
n = k * k
a = np.frombuffer(f.read(n * 4), dtype=np.float32)
if a.size != n:
raise SystemExit(f"short gram file: {path}")
tr = float(a.reshape(k, k).diagonal().sum())
return k, rank, int(rows), tr
def main():
ap = argparse.ArgumentParser()
ap.add_argument("calib_dir")
ap.add_argument("--min-share", type=float, default=0.03)
ap.add_argument("--min-route-mult", type=float, default=3.0)
a = ap.parse_args()
# layer -> global expert -> (rank, rows, energy)
data = {}
for path in sorted(glob.glob(os.path.join(
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
m = NAME_RE.match(os.path.basename(path))
if not m:
continue
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
k, _, rows, tr = read_gram(path)
data.setdefault(layer, {})[expert] = (rank, rows, tr)
if not data:
raise SystemExit("no per-expert down grams found")
lines, report = [], []
total = 0
for layer in sorted(data):
experts = data[layer]
tot_e = sum(v[2] for v in experts.values())
tot_r = sum(v[1] for v in experts.values())
if tot_e <= 0 or tot_r <= 0:
report.append(f"blk.{layer}: empty recording, skipped")
continue
qual, pool = [], []
for e, (rank, rows, tr) in experts.items():
es, rs = tr / tot_e, rows / max(tot_r, 1)
rec = (e, rank, es, rs)
if es >= a.min_share and es >= a.min_route_mult * rs:
qual.append(rec)
elif es > 0:
pool.append(rec)
chosen = {r[0]: (r[2], r[3]) for r in qual}
for half in (0, 1): # rank halves by the contiguous expert_base split
n_half = sum(1 for e in chosen if e // 256 == half)
if n_half % 2:
pool.sort(key=lambda r: -r[2])
for cand in pool:
if cand[1] == half and cand[0] not in chosen:
chosen[cand[0]] = (cand[2], cand[3])
report.append(
f"blk.{layer}: half {half} odd -> added e{cand[0]} "
f"(energy {cand[2]*100:.2f}%)")
break
for e in sorted(chosen):
es, rs = chosen[e]
lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight"
f" cast dtype=bf16")
report.append(f"blk.{layer} e{e:3d} (rank {e // 256}): "
f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
f"ratio {es/max(rs,1e-12):6.1f}")
total += len(chosen)
print("\n".join(lines))
print("\n".join(report), file=sys.stderr)
print(f"total protected: {total}", file=sys.stderr)
if __name__ == "__main__":
main()
+119
View File
@@ -0,0 +1,119 @@
#!/usr/bin/env bash
# download-frank-large-flashnext.sh — fetch frank-large's flash-server v2
# payload: the Quat3rnion abliterated Swift1.5 checkpoint of peonist's v2
# .hgn, the REQUIRED lookup table and the optional vision tower from
# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
# card pins), and the flat tokenizer the two repos share. All public, no
# token — and per the fleet secret law, none would ever touch argv.
#
# The fetch law, same as every sibling script: curl -f (an access error
# must not become a "successful" 153-byte artifact), exact byte count from
# the Hub tree API, LFS sha256 verification, --verify-only to re-check an
# existing payload without downloading.
#
# File manifest (verified against the Hub tree API on 2026-10-06):
# qwen38-flash-next-v2-swift15-abliterated.hgn 66,687,678,464 B sha256
# 9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
# (Quat3rnion main; compacted r2 of the card's 89,775,868,032 B
# append-only original — byte-identical live tensors, dropped padding)
# qwen38-flash-next-ngram.hgn 51,200,246,144 B sha256
# 3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
# (peonist @5cc17cea; REQUIRED — the engine reads it beside the
# checkpoint)
# qwen38-flash-next-vision.hgn 897,916,416 B sha256
# d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
# (peonist @e053f488; optional, enables image input)
# tokenizer/* @e053f488 — byte-identical to the 27B-era tokenizer
# already on disk (tokenizer.json LFS
# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3);
# fetched fresh rather than symlinked, per the no-symlink law.
set -euo pipefail
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
ABLIT="https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated/resolve/main"
PEONIST="https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next/resolve"
NGRAM_REV="5cc17cea1a10b502b4a5db41d8d2b1a0f22dba26"
TOK_REV="e053f488b120b99ed2525e6ac99f68c51b3b6179"
FILES=(qwen38-flash-next-v2-swift15-abliterated.hgn
qwen38-flash-next-ngram.hgn
qwen38-flash-next-vision.hgn
tokenizer/tokenizer.json
tokenizer/chat_template.jinja
tokenizer/generation_config.json
tokenizer/merges.txt
tokenizer/vocab.json
tokenizer/tokenizer_config.json)
SIZES=(66687678464
51200246144
897916416
12809320
8952
202
3353259
6722759
17928)
SHAS=(9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3
- - - - - -)
# Fetch source per file: repo base + revision (empty = ABLIT @main).
SRCS=("$ABLIT"
"$PEONIST/$NGRAM_REV"
"$PEONIST/$TOK_REV"
"$PEONIST/$TOK_REV"
"$PEONIST/$TOK_REV"
"$PEONIST/$TOK_REV"
"$PEONIST/$TOK_REV"
"$PEONIST/$TOK_REV"
"$PEONIST/$TOK_REV")
VERIFY_ONLY=0
[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
mkdir -p "$DEST"
for i in "${!FILES[@]}"; do
f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"; base="${SRCS[$i]}"
path="$DEST/$f"
mkdir -p "$(dirname "$path")"
if [ "$VERIFY_ONLY" -eq 0 ]; then
# A file already at the published size skips the fetch entirely:
# resuming it would ask the CDN for a range past EOF (416 -> curl
# error under set -e), and re-pulling 62 GiB to prove a point is
# not what --verify-only is for. sha256 below still decides.
if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
echo "HAVE $f ($want bytes) — skipping fetch"
else
echo "FETCH $f -> $path"
# HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
# the Range header across redirects — a naive -C - then
# "resumes" by pulling the ~1 KB redirect body (measured
# 2026-10-05 on the 27B fetch: file frozen 14 MiB short, curl
# exit 0). Resolve the effective URL first, then resume
# against the CDN, which honors Range.
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$base/$f") \
|| { echo "cannot resolve $base/$f" >&2; exit 1; }
curl -fL --retry 5 -C - -o "$path" "$url"
fi
fi
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
got="$(stat -c %s "$path")"
[ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
echo "OK $f ($got bytes)"
done
echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
for i in "${!FILES[@]}"; do
f="${FILES[$i]}"; sha="${SHAS[$i]}"
if [ "$sha" = "-" ]; then
echo "OK $f (plain git blob, size-checked)"
continue
fi
got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
[ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
echo "OK $f sha256"
done
echo "DOWNLOAD_DONE $DEST"
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env bash
# download-frank-large-halogen.sh — fetch frank-large's halogen payload:
# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
# secret law, none would ever touch argv).
#
# The fetch law, same as every sibling script: curl -f (an access error
# must not become a "successful" 153-byte artifact), exact byte count from
# the model card, LFS sha256 verification, --verify-only to re-check an
# existing payload without downloading.
set -euo pipefail
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
FILES=(qwen3.8-27b-p1w4d-d2.hgn
tokenizer/tokenizer.json
tokenizer/vocab.json
tokenizer/merges.txt
tokenizer/chat_template.jinja
tokenizer/tokenizer_config.json)
# Byte sizes as published by the Hub on 2026-10-05, plus sha256 (the LFS
# oid) where the Hub stores an LFS object — plain git blobs carry no oid
# and are size-checked only. Re-check with:
# curl -s "https://huggingface.co/api/models/peonist-ai/halogen-qwen3.8-27b/tree/main?recursive=true"
SIZES=(35865565184 12809320 6722759 3353259 8952 17928)
SHAS=(274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 - - - -)
VERIFY_ONLY=0
[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
mkdir -p "$DEST"
for i in "${!FILES[@]}"; do
f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"
path="$DEST/$f"
mkdir -p "$(dirname "$path")"
if [ "$VERIFY_ONLY" -eq 0 ]; then
# A file already at the published size skips the fetch entirely:
# resuming it would ask the CDN for a range past EOF (416 -> curl
# error under set -e), and re-pulling 33.4 GiB to prove a point is
# not what --verify-only is for. sha256 below still decides.
if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
echo "HAVE $f ($want bytes) — skipping fetch"
else
echo "FETCH $f -> $path"
# HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
# the Range header across redirects — a naive -C - then
# "resumes" by pulling the ~1 KB redirect body (measured
# 2026-10-05: file frozen 14 MiB short, curl exit 0). Resolve
# the effective URL first, then resume against the CDN, which
# honors Range.
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$BASE/$f") \
|| { echo "cannot resolve $BASE/$f" >&2; exit 1; }
curl -fL --retry 5 -C - -o "$path" "$url"
fi
fi
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
got="$(stat -c %s "$path")"
[ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
echo "OK $f ($got bytes)"
done
echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
for i in "${!FILES[@]}"; do
f="${FILES[$i]}"; sha="${SHAS[$i]}"
if [ "$sha" = "-" ]; then
echo "OK $f (plain git blob, size-checked)"
continue
fi
got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
[ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
echo "OK $f sha256"
done
echo "DOWNLOAD_DONE $DEST"
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env bash
# Download kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF for
# frank-large (Dankclaw iGPU). Public repo — no auth header, and never a token
# on the command line (it is world-readable in /proc/<pid>/cmdline).
#
# The payload is the repo's Q6 AGENT rung:
# * Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf — ftype 114, 23.15 GiB.
# Q6_0_ROCMFPX tensor type (102) + the AGENT routing, which keeps the
# output-head and attention tensors at plain Q8_0 — the variant the card
# names for MTP serving ("Take Q6 AGENT (114) if you want protected
# heads"). Like every ROCmFPX file it loads on nothing but the fork.
#
# The other two files frank-large needs are NOT re-downloaded:
# * mtp-Qwen3.8-27B-Q4_0.gguf — the uncensored repo redistributes the
# ALIGNED repo's draft head byte for byte (identical Hub LFS oid
# 051a1764…), so it is hardlinked out of the aligned payload dir when that
# dir holds the verified bytes, and fetched from this repo otherwise.
# * mmproj-Qwen3.8-27B-BF16.gguf — this repo ships a Q8_0 projector; the
# fleet keeps the BF16 one, which is the projector that was verified 4/4
# on spatial ground truth on this host. Hardlinked the same way, fetched
# from the ALIGNED repo if it is not on disk.
#
# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
#
# Usage:
# scripts/download-frank-large-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
set -euo pipefail
REPO="kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF"
ALIGNED_REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
BASE="https://huggingface.co/${REPO}/resolve/main"
ALIGNED_BASE="https://huggingface.co/${ALIGNED_REPO}/resolve/main"
SOURCE_DIR="${SOURCE_DIR:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP}"
# name bytes sha256 base_url may_link — may_link=1 means the same bytes may
# already exist in SOURCE_DIR (hardlink them instead of re-fetching).
FILES=(
"Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf 24856607008 e966deb3d1a5a1b9ca7765126292f2d822f3dccdedb3b4872e673dec1cddc199 $BASE 0"
# The draft head is redistributed byte-identical by both repos; the BF16
# projector only exists in the aligned repo.
"mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e $BASE 1"
"mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8 $ALIGNED_BASE 1"
)
matches() { # matches FILE SIZE SHA — byte count AND LFS sha256
[[ -f "$1" ]] && [[ "$(stat -c%s "$1")" == "$2" ]] \
&& sha256sum -c --quiet <(printf '%s %s\n' "$3" "$1")
}
mkdir -p "$DEST"
status=0
fetch() { # fetch NAME SIZE SHA BASE_URL
local name="$1" size="$2" sha="$3" url="$4" f="$DEST/$1"
echo "FETCHING $name"
curl -fsSL --fail-with-body -o "$f.part" "$url/$name"
mv "$f.part" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
return 1
fi
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $name"
return 1
fi
echo "VERIFIED $name"
}
for entry in "${FILES[@]}"; do
read -r name size sha url may_link <<<"$entry"
f="$DEST/$name"
if matches "$f" "$size" "$sha"; then
echo "OK $name"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $name"
status=1
continue
fi
# Shared bytes: prefer the local copy (same filesystem, so a hardlink costs
# nothing) over re-downloading gigabytes that hash identically.
if [[ "$may_link" == 1 && -n "${SOURCE_DIR:-}" ]] && matches "$SOURCE_DIR/$name" "$size" "$sha"; then
echo "LINKING $name from $SOURCE_DIR"
ln -f "$SOURCE_DIR/$name" "$f" 2>/dev/null || cp -f "$SOURCE_DIR/$name" "$f"
if matches "$f" "$size" "$sha"; then
echo "VERIFIED $name"
continue
fi
echo "SHA FAIL $name (linked copy)"
status=1
continue
fi
if ! fetch "$name" "$size" "$sha" "$url"; then
status=1
fi
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
# command line (it is world-readable in /proc/<pid>/cmdline).
#
# The payload is the repo's highest-quality target plus its MTP draft head and
# vision projector:
# * Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf — ftype 115, 187 tensors at plain
# Q8_0 and 311 at GGML_TYPE_Q8_0_ROCMFPX (103). The AGENT routing keeps the
# output-head tensors at full Q8_0 (187 vs 2 in the non-AGENT build), which
# is the variant to take when serving with the MTP head.
# * mtp-Qwen3.8-27B-Q4_0.gguf — the draft head. Q4_0, NOT Q8_0: the head's
# per-step cost is its projection over a 248K vocabulary, so the cheaper
# head is faster end to end at equal acceptance (author: +6.0% at n-max 1,
# +10.2% at n-max 2).
# * mmproj-Qwen3.8-27B-BF16.gguf — the vision tower (verified 4/4 on spatial
# ground truth with -fa on).
#
# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
#
# The repo's own SHA256SUMS covers only the ROCmFP4 files, so the hashes below
# come from the Hub LFS oids (api/models/<repo>/tree/main).
#
# Usage:
# scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
set -euo pipefail
REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
BASE="https://huggingface.co/${REPO}/resolve/main"
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
# name bytes sha256
FILES=(
"Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf 28214425952 f3ef2bdbf56e244fff14b72114dc2aa300b84a4cb4eaf54f2e5e4da1cfcae10b"
"mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e"
"mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8"
)
mkdir -p "$DEST"
status=0
for entry in "${FILES[@]}"; do
read -r name size sha <<<"$entry"
f="$DEST/$name"
have=0
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
have=1
fi
fi
if (( have )); then
echo "OK $name"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $name"
status=1
continue
fi
echo "FETCHING $name"
tmp="$f.part"
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $name"
status=1
continue
fi
echo "VERIFIED $name"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env bash
# Download the Q4_K_XL frank-large payload on Dankclaw:
# unsloth/Qwen3.8-27B-GGUF -> Qwen3.8-27B-UD-Q4_K_XL.gguf
#
# bash scripts/download-frank-large-q4.sh
# bash scripts/download-frank-large-q4.sh --verify-only
#
# Integrity: the file is checked against its declared byte count and sha256
# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
# the repo is public and ungated (gated: false), and a token on the curl
# command line is world-readable via /proc/<pid>/cmdline. If this ever needs
# a token, put it in a mode-600 file and pass it with `curl -K`.
#
# Layout after a successful run (MODEL_DIR):
# Qwen3.8-27B-UD-Q4_K_XL.gguf 17559178144 (16.35 GiB)
#
# WHY Q4 AND NOT THE Q8 SWIFT PAYLOAD: this is the vanilla (non-abliterated)
# Qwen3.8-27B at UD-Q4_K_XL, ~13 GiB smaller than the Swift UD-Q8_K_XL. Every
# GiB of weights saved is a GiB of f16 KV the endpoint can hold instead: the
# KV costs 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim
# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
#
# The speculative drafter is UNCHANGED and still fetched by its own script
# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
# appear here at all — the frank-large seam greps THIS file for it to prove
# the drafter was not folded into the payload fetch.
#
# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
# spec decode here is the block-diffusion drafter, not MTP.
set -euo pipefail
MODEL_DIR="/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL"
REL="Qwen3.8-27B-UD-Q4_K_XL.gguf"
SIZE=17559178144
SHA=3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e
URL="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${REL}"
mode="download"
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
if [[ "$mode" == "download" ]]; then
mkdir -p "$MODEL_DIR"
if [[ -s "$MODEL_DIR/$REL" ]]; then
echo "already present: $MODEL_DIR/$REL"
else
echo "fetching $REL (17559178144 bytes) — this is ~16.4 GiB"
# -f so an HTTP error is a failure, not a 153-byte "artifact".
# -C - so an interrupted run resumes instead of restarting.
curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
fi
fi
echo "verifying $MODEL_DIR against declared size + sha256"
python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
import hashlib, os, sys
model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
dest = os.path.join(model_dir, rel)
if not os.path.isfile(dest):
print(f" MISSING {rel}")
sys.exit(1)
actual = os.path.getsize(dest)
if actual != size:
print(f" SIZE MISMATCH {rel}: {actual} != {size}")
sys.exit(1)
h = hashlib.sha256()
with open(dest, "rb") as f:
for chunk in iter(lambda: f.read(1 << 24), b""):
h.update(chunk)
if h.hexdigest() != sha:
print(f" SHA256 MISMATCH {rel}: {h.hexdigest()}")
sys.exit(1)
print(f" OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
PY
echo "Q4_K_XL payload complete: $MODEL_DIR"
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env bash
# Download the Swift-1.5 frank-large candidate on Dankclaw:
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
#
# bash scripts/download-frank-large-swift.sh
# bash scripts/download-frank-large-swift.sh --verify-only
#
# Integrity: the file is checked against its declared byte count and sha256
# (the LFS oid from the Hub, 2026-09-28). NO AUTH HEADER IS USED: the repo is
# public and ungated, and a token on the curl command line is world-readable
# via /proc/<pid>/cmdline. If this ever needs a token, put it in a mode-600
# file and pass it with `curl -K`.
#
# Layout after a successful run (MODEL_DIR):
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf 31457990272
#
# This is the CANDIDATE payload, not the deployed one. frank-large still
# serves scripts/download-frank-large.sh's output; do not repoint
# frank-large.env at MODEL_DIR until the gufo load test passes.
set -euo pipefail
MODEL_DIR="/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL"
REL="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
SIZE=31457990272
SHA=9076f99a2b1c0d232d5c169bb117e919ca21bafbe092be8fc8d9c4d645ed09c9
URL="https://huggingface.co/ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/${REL}"
mode="download"
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
if [[ "$mode" == "download" ]]; then
mkdir -p "$MODEL_DIR"
if [[ -s "$MODEL_DIR/$REL" ]]; then
echo "already present: $MODEL_DIR/$REL"
else
echo "fetching $REL (31457990272 bytes) — this is ~29 GiB"
# -f so an HTTP error is a failure, not a 153-byte "artifact".
# -C - so an interrupted run resumes instead of restarting.
curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
fi
fi
echo "verifying $MODEL_DIR against declared size + sha256"
python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
import hashlib, os, sys
model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
dest = os.path.join(model_dir, rel)
if not os.path.isfile(dest):
print(f" MISSING {rel}")
sys.exit(1)
actual = os.path.getsize(dest)
if actual != size:
print(f" SIZE MISMATCH {rel}: {actual} != {size}")
sys.exit(1)
h = hashlib.sha256()
with open(dest, "rb") as f:
for chunk in iter(lambda: f.read(1 << 24), b""):
h.update(chunk)
if h.hexdigest() != sha:
print(f" SHA256 MISMATCH {rel}: {h.hexdigest()}")
sys.exit(1)
print(f" OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
PY
echo "swift candidate payload complete: $MODEL_DIR"
+113
View File
@@ -0,0 +1,113 @@
#!/usr/bin/env bash
# download-frank-primary-swift15.sh — payload fetch/verify for the planned
# frank-primary swap to the Swift 1.5 finetune: ukisai/Swift1.5-Qwen3.8-Flash-Next
# (BF16 source, 149 files, 360,037,355,038 bytes = 335.3 GiB: 131 safetensors
# shards + weight-map index + tokenizer + vision preprocessing configs + chat
# template + licenses). Key set verified identical to the official
# Qwen/Qwen3.8-Flash-Next base (0 added / 0 missing), so the qwen4exp recipes
# apply verbatim; the checkpoint ships BF16 with NO quantization_config.
# Repo is UNGATED — no token needed (never put one on the command line).
#
# bash scripts/download-frank-primary-swift15.sh --fetch resume/complete
# bash scripts/download-frank-primary-swift15.sh verify only
#
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
# listing/resume downloader, curl+jq only): per-file size checks against
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
# Verify (default) is the AGENTS.md law in three parts: every listed file
# present with the listed size, the shard count matches the weight-map
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
# only the public API.
set -euo pipefail
REPO="ukisai/Swift1.5-Qwen3.8-Flash-Next"
DEST="/home/dank/models/ukisai/Swift1.5-Qwen3.8-Flash-Next"
EXPECTED_SHARDS=131
mode="verify"
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if [[ "$mode" == "fetch" ]]; then
# HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$DEST" main
fi
echo "verifying local payload: $DEST"
fail=0
INDEX="$DEST/model.safetensors.index.json"
if [[ ! -f "$INDEX" ]]; then
echo " BAD weight-map index missing: $INDEX" >&2
exit 1
fi
python3 - "$REPO" "$DEST" <<'PY' || fail=1
import hashlib, json, os, sys, urllib.request
repo, dest = sys.argv[1], sys.argv[2]
def listing():
"""The tree API, following the Link cursor exactly like hfget.sh."""
out, api, pages = [], (
f"https://huggingface.co/api/models/{repo}/tree/main"
"?recursive=true&expand=true"), 0
while api:
with urllib.request.urlopen(api) as r:
out += json.load(r)
api = None
link = r.headers.get("Link", "")
for part in link.split(","):
if 'rel="next"' in part:
api = part[part.find("<") + 1:part.find(">")]
pages += 1
if pages > 200:
raise SystemExit("listing did not terminate")
return out
files = [e for e in listing() if e.get("type") == "file"]
sizes = {e["path"]: e.get("size", 0) for e in files}
lfs = {e["path"]: e["lfs"]["oid"] for e in files if e.get("lfs")}
bad = 0
for path, want in sorted(sizes.items()):
p = os.path.join(dest, path)
if not os.path.isfile(p):
print(f" BAD {path}: missing")
bad += 1
elif os.path.getsize(p) != want:
print(f" BAD {path}: {os.path.getsize(p)} bytes, listing says {want}")
bad += 1
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(sizes) - bad}/{len(sizes)} files present at listed size")
if bad:
raise SystemExit(1)
# Shard set against the weight-map index.
wm = json.load(open(os.path.join(dest, "model.safetensors.index.json")))["weight_map"]
shards = sorted(set(wm.values()))
print(f" index maps {len(wm)} tensors across {len(shards)} shards")
if len(shards) != 131:
print(f" BAD shard count {len(shards)}, want 131")
raise SystemExit(1)
# Content: every LFS-tracked file hashed against the Hub oid. The safetensors
# shards dominate the bytes; tokenizer/config files ride the same check when
# the Hub tracks them in LFS (they are plain blobs here, already size-checked).
todo = [p for p in shards if p in lfs]
for i, path in enumerate(todo):
h = hashlib.sha256()
with open(os.path.join(dest, path), "rb") as f:
for chunk in iter(lambda: f.read(1 << 24), b""):
h.update(chunk)
if h.hexdigest() != lfs[path]:
print(f" BAD {path}: sha256 mismatch (want {lfs[path]})")
bad += 1
if (i + 1) % 16 == 0:
print(f" ...hashed {i + 1}/{len(todo)}")
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(todo) - bad}/{len(todo)} shards sha256-match the Hub")
if bad:
raise SystemExit(1)
PY
[[ "$fail" -eq 0 ]] || exit 1
echo "payload verified: 131 shards + index + tokenizer + configs, sizes and sha256 checked"
+102 -83
View File
@@ -1,92 +1,111 @@
#!/usr/bin/env bash
# Fetch the frank-primary payload: the Qwen3.8-Flash-Next Q8_0 target (six
# split parts), the shared Q8_0 MTP drafter head and the BF16 vision
# projector.
# download-frank-primary.sh — payload fetch/verify for frank-primary:
# the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint (131 safetensors
# shards + weight-map index + tokenizer + vision preprocessing configs,
# ~172.8 GB). 2026-10-05: replaces the orcarouter uncensored payload of the
# FreeToken era (that script lived here; its repo was gated and needed
# HF_TOKEN — this one is UNGATED and needs no token at all).
#
# bash scripts/download-frank-primary.sh
# bash scripts/download-frank-primary.sh --verify-only
# bash scripts/download-frank-primary.sh --fetch resume/complete
# bash scripts/download-frank-primary.sh verify only
#
# The repo (unsloth/Qwen3.8-Flash-Next-GGUF) is PUBLIC and anonymous: no HF
# token is needed, and none is accepted on the command line (a -H auth
# header is world-readable in /proc/<pid>/cmdline -- the 2026-09-16 leak).
#
# WHY THESE FILES:
# * Target: Q8_0/ -- unsloth's Q8_0 of Qwen3.8-Flash-Next (125B total /
# 6B active MoE + the 51B n-gram embedding table). Q_8 is the
# operator's requested quant (2026-09-26 swap off the lucebox DS4.1
# engine). The six parts ARE the payload: part 1's header advertises
# file_count=6, so they are never concatenated (concatenation trips
# "invalid split file name").
# * Drafter: MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -- the shared
# MTP head (1 layer). "shared" means it borrows the target's embedding
# tensors instead of carrying its own copy: 2.6 GiB instead of 3.9,
# and unsloth's own MTP guide names it as the recommended head. It is
# loaded via -md (this build never autoloads drafters or projectors).
# * Vision: mmproj-BF16.gguf -- the BF16 projector, RDNA4-native per the
# frank-small precedent. Loaded via --mmproj explicitly.
#
# The script is resumable (curl -C -) and checks every artifact's exact byte
# count AND sha256 against the Hub's declared values, so a truncated or
# mis-fetched file fails here instead of load-crashing the server. Sizes and
# hashes were captured from the resolve endpoint's x-linked-etag/x-linked-size
# on 2026-09-26.
#
# Total: 191,919,144,448 bytes (178.74 GiB), of which 175.30 GiB is the
# target itself. Destination keeps the Hub layout so the serve script can
# name paths exactly as they appear upstream.
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
# listing/resume downloader, curl+jq only): per-file size checks against
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
# Verify (default) is the AGENTS.md law in three parts: every listed file
# present with the listed size, the shard count matches the weight-map
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
# only the public API.
set -euo pipefail
REPO="unsloth/Qwen3.8-Flash-Next-GGUF"
DEST="/home/dank/models/Qwen3.8-Flash-Next-GGUF"
REPO="Qwen/Qwen3.8-Flash-Next-FP8"
DEST="/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8"
EXPECTED_SHARDS=131
# path|bytes|sha256
MANIFEST=(
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf|10946624|2dabcbb53ca537a7947bc7d20414fd464eeaf4d66d43021b5b2556cc87544ad2"
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00002-of-00006.gguf|682434912|494ca4ed3dbf97bc28da88af3890b8877b9032f909812d00c0526a9ca5e91d2e"
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf|54400261312|34efd79a80a1ce540a517a5d56171924b66ce1c38b04c904f17ad6d8ef17cf20"
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00004-of-00006.gguf|49446841216|bfa634025fabbd2658bf7694bc80b90e571699c768723f844c934c7ef06c691a"
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00005-of-00006.gguf|49668930400|232a8f14cc0fa4262e7efe8593774b136fe40909e39c7a020342ddaa27259a97"
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00006-of-00006.gguf|34015618784|538a93bca918064983409a41187ad4c68640f9aced6f29564da8f551bf86d7a5"
"MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf|2786568256|5ff54097406a905cf3a724c709124ceb0e3e10235ee862298969e91c96fa96e6"
"mmproj-BF16.gguf|907542944|2e788f8c511d8093c7b43cb87b2fd7e14228340318057f8fb20c86df2efe2355"
)
mode="verify"
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
mode="download"
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
if [[ "$mode" == "download" ]]; then
for entry in "${MANIFEST[@]}"; do
path="${entry%%|*}"
mkdir -p "$DEST/$(dirname "$path")"
echo "fetching $path"
# -f: a 401/403 must FAIL the fetch, not write the error body as
# the artifact (the 2026-09-16 gated-repo incident). -C -: resume
# partial files.
curl -fL -C - -o "$DEST/$path" --retry 10 --retry-delay 5 \
--retry-max-time 21600 \
"https://huggingface.co/$REPO/resolve/main/$path"
done
if [[ "$mode" == "fetch" ]]; then
# HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$DEST" main
fi
echo "verifying sizes and sha256"
ok=1
for entry in "${MANIFEST[@]}"; do
path="${entry%%|*}"; rest="${entry#*|}"
size="${rest%%|*}"; sha256="${rest#*|}"
file="$DEST/$path"
act="$(stat -c%s "$file" 2>/dev/null || echo 0)"
if [[ "$act" -ne "$size" ]]; then
echo " BAD-SIZE $file ($act bytes, want $size)" >&2
ok=0
continue
fi
act_sha="$(sha256sum "$file" | cut -d' ' -f1)"
if [[ "$act_sha" != "$sha256" ]]; then
echo " BAD-SHA $file ($act_sha)" >&2
ok=0
continue
fi
echo " OK $path ($act bytes)"
done
[[ "$ok" -eq 1 ]] || exit 1
echo "all artifacts present, sized and hashed correctly"
echo "verifying local payload: $DEST"
fail=0
INDEX="$DEST/model.safetensors.index.json"
if [[ ! -f "$INDEX" ]]; then
echo " BAD weight-map index missing: $INDEX" >&2
exit 1
fi
python3 - "$REPO" "$DEST" <<'PY' || fail=1
import hashlib, json, os, sys, urllib.request
repo, dest = sys.argv[1], sys.argv[2]
def listing():
"""The tree API, following the Link cursor exactly like hfget.sh."""
out, api, pages = [], (
f"https://huggingface.co/api/models/{repo}/tree/main"
"?recursive=true&expand=true"), 0
while api:
with urllib.request.urlopen(api) as r:
out += json.load(r)
api = None
link = r.headers.get("Link", "")
for part in link.split(","):
if 'rel="next"' in part:
api = part[part.find("<") + 1:part.find(">")]
pages += 1
if pages > 200:
raise SystemExit("listing did not terminate")
return out
files = [e for e in listing() if e.get("type") == "file"]
sizes = {e["path"]: e.get("size", 0) for e in files}
lfs = {e["path"]: e["lfs"]["oid"] for e in files if e.get("lfs")}
bad = 0
for path, want in sorted(sizes.items()):
p = os.path.join(dest, path)
if not os.path.isfile(p):
print(f" BAD {path}: missing")
bad += 1
elif os.path.getsize(p) != want:
print(f" BAD {path}: {os.path.getsize(p)} bytes, listing says {want}")
bad += 1
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(sizes) - bad}/{len(sizes)} files present at listed size")
if bad:
raise SystemExit(1)
# Shard set against the weight-map index.
wm = json.load(open(os.path.join(dest, "model.safetensors.index.json")))["weight_map"]
shards = sorted(set(wm.values()))
print(f" index maps {len(wm)} tensors across {len(shards)} shards")
if len(shards) != 131:
print(f" BAD shard count {len(shards)}, want 131")
raise SystemExit(1)
# Content: every LFS-tracked file hashed against the Hub oid. The safetensors
# shards dominate the bytes; tokenizer/config files ride the same check when
# the Hub tracks them in LFS (they are plain blobs here, already size-checked).
todo = [p for p in shards if p in lfs]
for i, path in enumerate(todo):
h = hashlib.sha256()
with open(os.path.join(dest, path), "rb") as f:
for chunk in iter(lambda: f.read(1 << 24), b""):
h.update(chunk)
if h.hexdigest() != lfs[path]:
print(f" BAD {path}: sha256 mismatch (want {lfs[path]})")
bad += 1
if (i + 1) % 16 == 0:
print(f" ...hashed {i + 1}/{len(todo)}")
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(todo) - bad}/{len(todo)} shards sha256-match the Hub")
if bad:
raise SystemExit(1)
PY
[[ "$fail" -eq 0 ]] || exit 1
echo "payload verified: 131 shards + index + tokenizer + configs, sizes and sha256 checked"
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env bash
# Download the frank-secondary payload for llama.cpp on the RTX 3090:
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the
# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively
# multimodal, so this file is the endpoint's eyes). Public repo — no auth
# header, and never a token on the command line (it is world-readable in
# /proc/<pid>/cmdline).
#
# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized
# file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a
# success, then byte count and sha256. The Hub exposes no LFS oid for this
# repo (Xet storage), so the sha256 values below were computed locally after
# the first verified fetch — byte counts match the Hub's declared sizes.
# A sha of "-" means "first fetch: size-verified, sha printed for pinning".
# Idempotent: complete files are skipped.
#
# 2026-10-07 (Frank): the payload moved to Qwen3.5-9B-Uncensored Q6_K at
# 2 x 200,704 (qwen35 hybrid: 32 layers, 8 full-attention + 24 Gated
# DeltaNet, no MTP tensors in this file). Rollback payloads stay on disk:
# /home/dank/models/Gemma4-26B-A4B/ (Q4_K_M, the 2026-10-07 era being
# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/.
#
# Usage:
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
set -euo pipefail
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K}"
# repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo)
FILES=(
"HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|7359259008|c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986"
"HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|921704448|05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934"
)
status=0
for entry in "${FILES[@]}"; do
IFS='|' read -r repo hubname destname size sha <<<"$entry"
url="https://huggingface.co/${repo}/resolve/main/${hubname}"
f="$DEST/$destname"
have=0
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
if [[ "$sha" == "-" ]]; then
have=1
elif sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
have=1
fi
fi
if (( have )); then
echo "OK $destname"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $destname"
status=1
continue
fi
echo "FETCHING $destname (from $repo)"
tmp="$f.part"
curl -fsSL --fail-with-body -o "$tmp" "$url"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if [[ "$sha" == "-" ]]; then
echo "VERIFIED $destname (size only) sha256=$(sha256sum "$f" | cut -d' ' -f1)"
continue
fi
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $destname"
status=1
continue
fi
echo "VERIFIED $destname"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
# fleet-probes-swift15.sh — RUNS ON DANKSERVER. The cutover contract probes.
set -uo pipefail
B=http://localhost:7770/v1
echo "== vision (solid red 512px, expect 'Red')"
python3 - <<'PY'
import base64, json, urllib.request, struct, zlib
def png(color):
w = h = 512
raw = b"".join(b"\x00" + bytes(color) * w for _ in range(h))
def chunk(t, d):
c = t + d
return struct.pack(">I", len(d)) + c + struct.pack(">I", zlib.crc32(c))
return (b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
+ chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b""))
b64 = base64.b64encode(png((255, 0, 0))).decode()
body = json.dumps({"model": "frank-archon", "max_tokens": 256, "temperature": 0,
"messages": [{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "data:image/png;base64," + b64}},
{"type": "text", "text": "What color is this image? Answer with one word."}]}]}).encode()
r = urllib.request.Request("http://localhost:7770/v1" + "/chat/completions",
data=body, headers={"Content-Type": "application/json"})
d = json.load(urllib.request.urlopen(r, timeout=300))
m = d["choices"][0]["message"]
ans = (m.get("content") or m.get("reasoning_content") or "")
print("VISION ANSWER:", ans[:80].replace("\n", " "))
PY
echo "== tools (expect tool_calls)"
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' -d '{
"model":"frank-archon","temperature":0,"max_tokens":128,
"messages":[{"role":"user","content":"What is 23 times 47? Use the calculator tool."}],
"tools":[{"type":"function","function":{"name":"calculator","description":"evaluate arithmetic",
"parameters":{"type":"object","properties":{"expr":{"type":"string"}},"required":["expr"]}}}]}' \
| python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print("TOOL_CALLS:", m.get("tool_calls"))'
echo "== reasoning_effort fold (xhigh/medium/low/high all 200 on froggeric v22.5)"
for lvl in xhigh medium low high; do
code=$(curl -s -o /dev/null -w "%{http_code}" -m 120 $B/chat/completions \
-H 'Content-Type: application/json' \
-d "{\"model\":\"frank-archon\",\"max_tokens\":8,\"reasoning_effort\":\"$lvl\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}")
echo " reasoning_effort=$lvl -> HTTP $code"
done
echo "== reasoning_content channel"
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"frank-archon","max_tokens":64,"messages":[{"role":"user","content":"Is 97 prime? Think briefly."}]}' \
| python3 -c 'import json,sys; m=json.load(sys.stdin)["choices"][0]["message"]; print("has reasoning_content:", bool(m.get("reasoning_content")), "| content:", (m.get("content") or "")[:60].replace(chr(10)," "))'
echo "== response_format json_object"
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"frank-archon","max_tokens":64,"response_format":{"type":"json_object"},"messages":[{"role":"user","content":"Return JSON {\"ok\": true}"}]}' \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print("JSON MODE:", (d["choices"][0]["message"].get("content") or "")[:60])'
echo "PROBES_DONE"
+153 -31
View File
@@ -1,36 +1,158 @@
# frank-large — stock llama.cpp (mainline, the unsloth b11030 prebuilt)
# serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU
# (gfx1151).
# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
# Strix Halo iGPU (gfx1151), host port 7770.
#
# These are the tunables read by scripts/serve-frank-large.sh. The flag
# list is owned by the serve script; this file only sets fleet values.
# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
# a root unit, driven by scripts/serve-frank-large.sh. This CUTS OVER the
# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
# rollback = revert the swap commit's trio; the 27B payload stays on disk at
# /home/dank/models/halogen-qwen3.8-27b/.
#
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
# exception because the serve script itself consumes it for --host.
# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2
# .hgn loads ONLY in the flash-server line (the old halogen engine is for
# the 27B p1w4d file; the v2 quickstart pins flash-server). What the move
# buys (card + live probe evidence): MTP spec decode (53-57 t/s greedy,
# 3-10x serial), ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than
# the censored 27B (905 mean), the ABLITERATED base (refusals should NOT
# return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint —
# red-PNG probe verified on the card; the 27B engine ignored its encoder),
# tool calls verified, and the native 262,144 context.
#
# SHAPE — the knobs that are NOT free, with the arithmetic:
# HALOGEN_CTX=262144 is the PER-SLOT window and the per-conversation hard
# ceiling — a prompt over it is a 400 naming the limit, never a silent
# truncation. RAISED from 200192 on 2026-10-07: the harness clamps
# max_tokens to contextWindow − est − 4096, its chars/4 est UNDER-counts
# the rendered prompt by ~3 % on code/agent traffic, and compaction only
# fires at 85 % — so with window == engine ceiling, every request over
# ~146k est (65-85 % of the window) was a guaranteed 400. Measured
# 2026-10-07: est 152,938, server 157,388, request 43,158 vs room 42,804
# — over by 354, margin 4,096 < the 4,450 under-count.
# Harness contract (models.yml): contextWindow 217088, maxTokens 45056.
# Compaction fires at 85 % = 184,524 est ≈ 190k measured (Frank's 180k
# target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 %
# est-error ≈ 219k <= 262144 — the 45k gap between harness window and
# engine ceiling absorbs the estimate error with ~7x margin (robust to a
# 10 % under-count). Compaction rides on frank-secondary: input ≈
# threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot.
# HALOGEN_KV_POOL_POSITIONS=524288 = KV_SLOTS(2) x CTX — the explicit
# form of the engine's default, so the seam test can pin the product.
# KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
# drafter is NOT slot-locked on this engine (batching and speculation
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
# quickstart ships 2 slots WITH the head), so the second slot is
# concurrent capacity, not a speculation tax.
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
# mmap'd page cache beside the checkpoint, not a pinned charge — the
# card's own quickstart arithmetic does not bill it):
# weights ~62 GiB resident + KV 14.0 + arena (HALOGEN_MAX_TOK=8192 =>
# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
# = ~86 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
# honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
# ~10-15. ~15 GiB margin. The boot ledger is the referee — read it
# after deploy and re-size if the engine prints a clamp.
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
# turns the MTP head off for whole requests after low acceptance and
# HALVES decode as a result. Frank accepted the head always-on trade
# (temp 1.0 traffic costs acceptance; the card still ships the recipe).
#
# CLIENT CONTRACT deltas vs the 27B shape (harness side lives in
# ~/.omp/agent/models.yml):
# * SERVER-SIDE SAMPLER EXISTS NOW: HALOGEN_TEMPERATURE/TOP_P/TOP_K set
# the card's thinking recipe (1.0 / 0.95 / 20) as engine defaults, so
# an absent temperature is no longer greedy. models.yml extraBody
# carries the same values — consistent, not load-bearing.
# * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
# max_tokens; the harness sends at most its maxTokens 45056. Requests
# over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
# queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
# worst case needs > 6550 s; 7200 honors it. Do not move one alone.
# * max_tokens INCLUDES thinking; a reasoning reply over a small budget
# returns empty content with finish_reason "length".
# * Prompt cache is ON by default (PROMPT_CACHE=2): a follow-up turn at
# 100k context re-prefills ~2 s instead of ~88 s cold. No pin needed —
# unlike the 27B engine, auto-sizing fits this box's ledger.
# * Thinking lands in reasoning_content (same as the 27B engine); the
# v2 chat template rejects reasoning_effort outside xhigh|medium|low,
# so the models.yml fold (high->xhigh) still applies.
# * Vision: OpenAI image_url content parts only (the fleet's standard
# contract). >=256 px probe images or the model answers "video".
# * response_format json_object/json_schema unsupported in this line
# (no constrained decoding) — unchanged from the 27B engine.
# * The engine's token protocol has no auth and binds loopback INSIDE
# the container; only the frontend port (8731) is published. Same as
# before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
# amdgpu.noretry=0 (check at deploy).
# The payload (scripts/download-frank-large.sh): the serve script refuses to
# start without the main GGUF and the DFlash2 drafter.
MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
# is what was qualified). 0.16.2 = the version the model card names as its
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
# Container identity: kept from the 27B era (unit ExecStop names it).
FRANK_LARGE_CONTAINER=frank-large-halogen
# Old quadlet-era container identity; the serve script clears stale names.
FRANK_LARGE_LEGACY_CONTAINERS=frank-large
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
# default); the engine port (8730) is never published.
FRANK_LARGE_PUBLISH=7770:8731
# The stock llama.cpp runtime (the unsloth b11030 prebuilt).
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
# append-only original (byte-identical live tensors, padding dropped:
# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
FRANK_LARGE_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
FRANK_LARGE_CHECKPOINT_BYTES=66687678464
FRANK_LARGE_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
# The n-gram lookup table: REQUIRED (the engine reads it beside the
# checkpoint; the card's quickstart names it explicitly). From
# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
FRANK_LARGE_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
FRANK_LARGE_NGRAM_BYTES=51200246144
FRANK_LARGE_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
FRANK_LARGE_VISION_TOWER=qwen38-flash-next-vision.hgn
FRANK_LARGE_VISION_BYTES=897916416
FRANK_LARGE_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer,
# mounted whole at /tokenizer. The two repos ship the same one; fetched
# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
# the 27B-era copy; the rest are plain git blobs — size-checked only). The
# serve gate and the deployer refuse a missing file or ANY symlink: an HF
# cache snapshot symlinks each file into blobs/, and those links dangle
# inside the container mount.
FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
# Fleet endpoint: the same port and alias the prior deployments served, so
# no client, probe or models.yml entry changes address.
HOST=0.0.0.0
PORT=7770
# One stream, full 262144 context (the model's native window).
CONTEXT_SIZE=262144
PARALLEL_SLOTS=1
# DFlash2 spec decode (a block-diffusion drafter) at depth 4. The drafter
# has a sliding window of 2048 (small draft KV, ~40 MiB).
SPEC_DRAFT_N_MAX=4
# Prompt-cache budget: 8 GiB. The model is a 27B hybrid (16 full attention
# + 48 Gated DeltaNet linear attention); its DeltaNet state is smaller than
# a primarily-DeltaNet model, so the fleet's 12.8 GiB DeltaNet floor does
# not apply.
CACHE_RAM=8192
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
# stays at the image default, which is what was qualified) ------------------
# Served model id — Protoss name (wire rename 2026-10-09); the service, unit,
# scripts and this file keep the frank-large name. OpenWebUI api_configs
# model_ids for the :7770 backend MUST name this value.
HALOGEN_MODEL_ID=frank-dragoon
HALOGEN_CTX=262144
HALOGEN_KV_POOL_POSITIONS=524288
HALOGEN_KV_SLOTS=2
HALOGEN_HOST_RESERVE_GIB=20
# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
# depth 2 is the documented "best all-round" default. HALOGEN_DRAFTER /
# HALOGEN_DFLASH_* from the 27B engine DO NOT EXIST here.
HALOGEN_SPEC_ADAPT=0
# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
# the GPU, the timeout bounds how long the NEXT client waits. Do not move
# one alone (see the client-contract note above).
HALOGEN_MAX_TOK=8192
HALOGEN_MAX_TOKENS_CAP=65536
HALOGEN_QUEUE_TIMEOUT=7200
# Server-side sampler = the card's thinking recipe (the engine now HAS one;
# requests may still override per-call). Matches models.yml extraBody.
HALOGEN_TEMPERATURE=1.0
HALOGEN_TOP_P=0.95
HALOGEN_TOP_K=20
# Card's belt-and-braces: the engine pins weights by default
# (HALOGEN_FLASH_PIN_TRUNK=1; only the retired w4b build honored 0).
HALOGEN_WEIGHTS_LOCK=1
+13 -2
View File
@@ -1,5 +1,5 @@
[Unit]
Description=stock llama.cpp (unsloth b11030, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, fp8 KV, 262144 ctx / 1 slot, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
After=network-online.target
Wants=network-online.target
@@ -13,8 +13,19 @@ Group=dank
LimitMEMLOCK=infinity
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
Restart=on-failure
# systemd stop alone cannot stop the container child; stop it explicitly
# (frank-primary pattern). The engine drains on SIGTERM.
ExecStop=/usr/bin/podman stop -t 60 frank-large-halogen
TimeoutStartSec=1200
TimeoutStopSec=120
# always, not on-failure: an engine that exits 0 after an internal fault
# would otherwise leave the endpoint down until someone notices.
Restart=always
RestartSec=10
# The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set
# on a box that also hosts frank-secondary, ComfyUI and the Caddy/observability
# stack; make it a late OOM candidate, not the first.
OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
+167 -46
View File
@@ -1,47 +1,168 @@
# frank-primary.env — R9V radiance-vLLM production overrides (dankserver :7770).
# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
# dankserver :7770.
#
# The full tuned engine shape (tiered-expert placement, CPU offload budgets
# per rank, MTP4 drafter, TP=2, qwen3_coder tool parser, 131072 model len,
# 80-slot tiered expert cache, PLE pinned residency 54.4 GB) lives on-host
# in the R9V setup state — /home/dank/r9v-state/q8xl/setup.json — written by
# the q8 profile's setup run (ready: true; doctor PASS, artifacts
# sha256-verified at fetch time). serve-frank-primary.sh layers that state
# first and THIS FILE second, so the keys below win. Never copy tuned values
# from the state into this file — the state is the single source of truth
# for engine shape.
R9V_HOME=/home/dank/GitPackages/radiance-vllm-qwen4exp
R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json
# Fleet identity: LAN bind, the single frank-primary port, the harness's
# model id, our own container name (the launcher default r9v-qwen38-flash-next
# is the scratch identity).
R9V_HOST_BIND=0.0.0.0
R9V_HOST_PORT=7770
R9V_SERVED_MODEL_NAME=frank-primary
R9V_CONTAINER_NAME=frank-primary-r9v
R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0
# PLE registration (54.4 GB mmap -> HIP) can exceed the engine's 600 s
# VLLM_PLE_OFFLOAD_READY_TIMEOUT default under page-cache churn (measured
# 740 s on 2026-09-28, right after a bench teardown; a production boot
# failed on it the same morning). launch.sh 2de0357 forwards this to the
# engine; 1800 s covers the slowest observed registration with margin.
R9V_PLE_READY_TIMEOUT=1800
# The Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS
# package ships inject as F32), so the shard-uniformity check rejects the
# fused projection at load. This dev-overlay adapter dequantizes a quantized
# inject shard in-stream (fork commit 363d606): opt-in mount, uncensored
# payloads pass through unchanged.
R9V_DEV_QWEN4_EXP_ADAPTER_PY=/home/dank/GitPackages/radiance-vllm-qwen4exp/runtimes/dev-overrides/qwen4_exp.py
# The runtime overlay's env.always injects R9V_FULL_MUTABLE_CACHE=1 — the
# qualified 160-slot uncensored IQ4_XS cache geometry — which the q8
# profile's 80-slot tiered cache fails with a baseline mismatch. This flips
# that injection off for this payload (launch.sh passes it after the overlay
# envs; docker last-wins).
R9V_FULL_MUTABLE_CACHE=0
# The Qwen-Sharp template (the frank-large fleet law): folds reasoning
# efforts (high/xhigh/max -> xhigh, minimal/low -> low) and defaults
# preserve_thinking true. --chat-template wins over the tokenizer dir's
# metadata/chat_template.jinja; request-level kwargs still win per request.
# Braces in values get the same single-quote law as parentheses (source
# parses assignment values).
R9V_CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/qwen-sharp-v22.4.0.jinja
R9V_CHAT_TEMPLATE_KWARGS='{"preserve_thinking":true}'
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
# frank-primary-vllm-shapeG-20261003).
#
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
# gfx1201, drives `podman run` directly from the digest-pinned
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
# --host-pool-mib 12288) remains the documented fallback if the native
# container proves unbootable.
#
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
# staging dir whose config.json drops `quantization_config`: the descriptor
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
# containers carry. Per-weight encodings are data, not the descriptor —
# removing the key changes nothing about what the weights hold. Built by
# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
#
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
# then engine.serve()) — during load the port refuses connections, so a
# bare curl /health gate is honest. /health == /ping: 200 JSON while
# serving, 503 only while stopping (core/server/admin.cpp:385).
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
# the chat format. Both need no parser flags.
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
# repeat_last_n, chat_template_kwargs, reasoning_effort,
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
# models.yml's extraBody set is fully inside it. Unknown params are
# refused (400), which is why nothing unverified rides the wire.
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
# publisher's throughput peak for this model. Stated explicitly with
# --num-speculative-tokens 3.
# - The fused TP all-reduce is declared over max-num-seqs x
# (1 + --num-speculative-tokens) rows; 3 x 3 = 9 rows fits either wire.
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
# Acceptance on the greedy harness is deterministic (the control
# reproduced the recorded 0.755 exactly), so the drop is a real numerics
# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
# The vram-pair translation itself is exact (1.2.0 boot ledger took
# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
# file already uses the pair, which BOTH pins accept (rollback symmetry).
FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
# The .rad container FILE (mounted under /models) and the state dir for the
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
# .rad files; the raw HF download and its staging dir live beside it and
# are never mounted.
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
FRANK_PRIMARY_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
# cache must ride a host mount — serve script mounts it, see the runbook's
# "Tuning levers").
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
# name across systemctl and podman ps. The runtime changes; the name does
# not. The legacy list retires every predecessor identity that could hold
# the GPUs or the port.
FRANK_PRIMARY_CONTAINER=frank-primary
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
# Served model id — Protoss name (wire rename 2026-10-09). The container,
# unit, scripts and this file keep the frank-primary name above; only the
# alias on the wire moved. Radiance validates the requested model name, so
# every client (harness, bench scripts, probes) MUST send this value.
FRANK_PRIMARY_SERVED_NAME=frank-archon
FRANK_PRIMARY_PORT=7770
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
# repo's root chat_template.jinja, vendored here; the checkpoint's own
# template is baked into the .rad by rad-convert and this overrides it).
# Radiance derives the reasoning/tool-call reply format from it.
FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
# wire is libr4d's own, not RCCL.
FRANK_PRIMARY_TP=2
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
# engine default.
FRANK_PRIMARY_TP_WIRE=wht6
# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
# start"); raise only after a real-traffic look at slot occupancy and the
# KV pool's headroom line. The recurrent-state cost scales with it (54
# MiB/copy/card, 2 copies with speculation).
FRANK_PRIMARY_MAX_SEQS=3
# Served window. Native is 262,144; 200,000 is the publisher's production
# value (compose) and what models.yml's contextWindow must quote — a longer
# prompt is refused by the engine, never truncated.
FRANK_PRIMARY_MAX_MODEL_LEN=200000
FRANK_PRIMARY_PLACEMENT=expert_tiered
# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
# publisher shape's ~29 GiB — this pool is the cost of native precision.
# The boot ledger prints the pool's coverage ("the pool backs N% of the
# worst case ... M full-length sessions"); resize from that line, not from
# arithmetic.
FRANK_PRIMARY_HOST_POOL_MIB=40960
# VRAM left unclaimed per card (publisher default 96 MiB).
FRANK_PRIMARY_GPU_HEADROOM_MIB=96
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
# Engine 1.2.0 REMOVED --expert-vs-cache-ratio; these two knobs replace
# it. The values transcribe the live 1.0.13 boot ledger at ratio 0.82:
# vram_weights 25.46 GiB, vram_kv 3.46 GiB per card -> 26071 / 3543.
# Bracket arms scale the KV term linearly in (1-r) from the 0.82 anchor
# with weights fixed: ratio 0.90 -> KV 1968, ratio 0.95 -> KV 984. The
# boot ledger arbitrates; re-derive after any payload change.
FRANK_PRIMARY_VRAM_WEIGHTS_MIB=26071
FRANK_PRIMARY_VRAM_KV_MIB=3543
# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
# 2 KV heads, 256 head_dim, K+V).
FRANK_PRIMARY_KV_DTYPE=fp8
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
# flag states it so the ledger records the intent).
FRANK_PRIMARY_SPEC_TOKENS=2
FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
FRANK_PRIMARY_PREFIX_HOST_MIB=4096
FRANK_PRIMARY_PREFIX_DISK_MIB=65536
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
FRANK_PRIMARY_NGRAM_PLACEMENT=ram
# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
# plan, so the serve script refuses to start until both are raised. This is
# the floor it enforces (in MiB), with ~8 GiB of margin.
FRANK_PRIMARY_TTM_MIN_MIB=140000
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
# multi-flag values as one string; the serve script expands it unquoted on
# purpose.
FRANK_PRIMARY_EXTRA_ARGS=
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
# guaranteed long-timeout boot instead of a fast failure). The converted
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
# tail) - floor sits just under the payload.
FRANK_PRIMARY_RAD_MIN_MIB=110000
+48 -28
View File
@@ -1,44 +1,64 @@
# frank-primary.service — R9V radiance-vLLM endpoint (unsloth
# Qwen3.8-Flash-Next Q8_0, MTP4, TP=2, tiered experts, 6-shard payload) on
# the 2x R9700 pair, dankserver :7770. 2026-09-27 cutover from the
# uncensored IQ4_XS payload (its state dir stays on disk as rollback).
# frank-primary.service — radiance (StillDeadcode, Apache-2.0) endpoint
# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
# frank-primary-freetoken-pr587-20261005 is the pre-cutover state; the
# retired unit/env/serve trio and Dockerfile are in that tag).
#
# Replaces the llama.cpp frank-primary unit (same port, same served alias;
# harness models.yml unchanged in id, retuned for the 131072 window). The
# serve script drives the R9V fork's own launcher through the docker->podman
# shim and then waits on the container, so a crashed engine is a nonzero
# ExecStart and Restart=on-failure is real.
#
# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
# of GiB of host RAM (the UVA expert host copy plus the 16 GiB PLE pinned
# reserve). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
# session on this host carries 8 MiB and a user quadlet is capped by the user
# manager's own hard limit — raising that means restarting user@1000.service
# and bouncing every user container on the host. A root unit grants
# LimitMEMLOCK=infinity to the dank process via PID1, and the image stays in
# dank's rootless podman storage (the shim drives podman as dank).
# Same port, same served alias, same container identity as the unit name;
# the harness models.yml id is unchanged. The serve script drives
# `podman run` directly: the digest-pinned stilldeadcode/radiance image
# (287 MiB, kernels prebuilt for gfx1201) has the engine as its
# entrypoint. The image lives in dank's ROOTLESS podman storage.
# WHY A ROOT UNIT (not a rootless quadlet): radiance pins ~132 GiB of HOST
# RAM (2 x 40 GiB expert pools + the 47.7 GiB PLE ngram table + the prefix
# host tier). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
# session on this host carries 8 MiB and a user quadlet is capped by the
# user manager's own hard limit — raising that means restarting
# user@1000.service and bouncing every user container on the host. A root
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
# engine still runs out of dank's rootless podman storage.
[Unit]
Description=frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 on 2x R9700
Description=frank-primary — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
After=network-online.target
Wants=network-online.target
# Never let the start-rate limiter give up after a crash loop (a failed LLM
# endpoint that systemd refuses to retry is worse than a retrying one).
StartLimitIntervalSec=0
[Service]
User=dank
Group=dank
# The tiered-expert offloader pins host RAM; an RLIMIT_MEMLOCK smaller than
# the pinned set fails the load mid-start. PID1 grants infinity here —
# parity with the fleet's other LLM units.
# The engine pins ~132 GiB of host RAM (expert pools + PLE table); an
# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
# PID1 grants infinity here — parity with the fleet's other LLM units.
LimitMEMLOCK=infinity
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
# systemd stop: wait alone cannot stop the container child; stop it
# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
ExecStop=/usr/bin/podman stop -t 120 frank-primary-r9v
# Boot cost: the launcher's preflight doctor (~1 min) after a warm page
# cache; cold (post-reboot) payload page-in of ~175 GiB dominates.
TimeoutStartSec=1800
Restart=on-failure
# explicitly. No graceful drain contract is claimed (the OpenAI front end
# holds no queue to drain — in-flight decodes are lost, which a stop
# during a bench window accepts; radiance takes SIGTERM on the engine
# process and podman reaps the container).
ExecStop=/usr/bin/podman stop -t 120 frank-primary
# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
# publisher's compose healthcheck allows a 15-minute start_period; the
# vLLM-era budget (1800 s) carried forward.
# Restart=always, NOT on-failure (the 2026-10-01 vLLM lesson carries over:
# a clean API-server exit after a worker fault reads as success to
# systemd). `always` restarts on any container death; an explicit
# `systemctl stop` still wins, so bench windows and deploys are unaffected.
Restart=always
RestartSec=10
# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without
# an explicit TimeoutStopSec systemd's 90 s default cuts the stop short,
# kills it, and records 'Failed with result timeout' (measured 2026-10-01
# 13:27:17). Must exceed the podman grace with margin.
TimeoutStopSec=180
# The engine pins tens of GiB of host RAM (expert rows + PLE table); it is
# the last process on this host that should be chosen by the kernel OOM
# killer.
OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
+62
View File
@@ -0,0 +1,62 @@
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
# :7771.
#
# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
# MTP disabled (SPEC_TYPE=none), log verbosity 4.
#
# Architecture — read from the GGUF's own header (HTTP range read, not the
# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
# F32×177 — standard types only, so the stock master build loads it (no
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
#
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
# where the 26B MoE fit only one.
#
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
# The verbosity-4 boot ledger is the referee.
#
# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
# Served model id — the harness/family name (Protoss rename completed to the
# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
# this day). The SERVICE, unit, scripts and this env file keep the
# frank-secondary name; only the alias on the wire changed. Consumers that
# key on the served id: OpenWebUI (model row, access_grant, api_configs
# model_ids) and the harness models.yml.
ALIAS=frank-zealot
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
# answered with a 400, never truncated.
CTX_SIZE=401408
PARALLEL=2
BATCH_SIZE=4096
UBATCH_SIZE=2048
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
SPEC_TYPE=none
NGL=99
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
# and graph capture land in the journal, which is what the VRAM-fit forensics
# read. Levels above 4 flood the journal.
VERBOSITY=4
+19
View File
@@ -0,0 +1,19 @@
[Unit]
Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=dank
Group=dank
# CUDA pins its staging buffers, so the process needs the memlock budget even
# though the weights themselves are copied into VRAM.
LimitMEMLOCK=infinity
Environment="FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
-118
View File
@@ -1,118 +0,0 @@
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
#
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
# cmake --build build-cuda --target llama-server -j
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
# attention, and without the flag quantized-KV prefill collapses ~12x on
# CUDA (see the CUDA lesson in AGENTS.md).
#
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
# per-layer token embeddings (PLE) inside the file, and a sliding-window
# attention pattern (attention.sliding_window = 1024) -- most layers cache
# only a 1024-position window, which is why the KV pool is a fraction of
# the full-ctx x all-layers number.
#
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
# is passed via --model-draft with --spec-type draft-mtp, so the draft
# context loads the assistant model. The same --spec-type value chains the
# draftless ngram-mod matcher, which attaches unconditionally; two
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
# is the only legal shape that carries both.
#
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
# which would shrink each slot below the full 262144 window once
# parallel > 1.
#
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
# params; it carries an audio encoder too). The fleet's image path lives
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
# WORKS: input_audio content parts transcribe verbatim (verified
# 2026-09-17 with a 16k mono WAV).
#
# Sampling: greedy by default (the fleet contract); a client that sends a
# temperature gets its own, and the harness applies per-model overrides
# from models.yml.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
# FA_ALL_QUANTS=ON.
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
# Model package. The MTP sidecar file is required for speculative decoding.
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
# Fleet identity and network. 7771 is this service's own endpoint on
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
# what clients ask for -- the harness entry in models.yml names it.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
# pool; --parallel divides it). See the SIZING note above for the fit
# arithmetic; a fifth slot does not fit.
CTX_SIZE=1048576
PARALLEL=4
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0
# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
# model, so the >12.8 GiB CACHE_RAM floor does not apply.
CACHE_RAM=2048
# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
# tried. Idle VRAM is 23,300 MiB at this geometry.
BATCH_SIZE=8192
UBATCH_SIZE=1024
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
NUM_THREADS=8
THREAD_BATCH=8
# Every layer on the 3090, with flash attention (required for q8 KV).
NGL=999
FLASH_ATTN=on
# Speculative decoding: the MTP sidecar (draft context) chained with the
# draftless ngram-mod matcher. Draft depth and busy-wait level are made
# explicit here (both were the implicit build defaults: 3 and 50).
# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
# costs more attention re-read at long context than the extra accepted
# token earns. Keep 3/50.
SPEC_TYPE=draft-mtp,ngram-mod
SPEC_DRAFT_N_MAX=3
POLL_LEVEL=50
# Greedy default (fleet contract); reasoning is extracted into
# reasoning_content with an unlimited budget (the model decides).
TEMPERATURE=0.0
REASONING_FORMAT=deepseek
REASONING_BUDGET=-1
-20
View File
@@ -1,20 +0,0 @@
[Unit]
Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=dank
Group=dank
# CUDA pins its staging buffers, so the process needs the memlock budget even
# though the weights themselves are copied into VRAM. Parity with the halogen
# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
LimitMEMLOCK=infinity
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
+107
View File
@@ -0,0 +1,107 @@
#!/bin/sh
# hfget.sh -- fetch a Hugging Face repo with curl and jq, and nothing else.
#
# The engine and its tooling do not depend on Python, so `huggingface-cli` is not assumed. The API
# gives a file listing as JSON and every file has a stable resolve/ URL, which is all a downloader
# needs.
#
# scripts/hfget.sh <repo-id> [dest-dir] [revision]
#
# Set HF_TOKEN for a gated or private repo. Re-running resumes: a file whose local size already
# matches the listing is skipped and a short one is continued with `curl -C -`, so an interrupted
# transfer of a 300 GB checkpoint costs only what it had not finished.
#
# WEIGHT DUPLICATES ARE SKIPPED. Many repos carry the same tensors several times over -- safetensors
# beside the older pickle .bin, an ONNX export, a GGUF conversion -- and on a large checkpoint each
# copy is hundreds of gigabytes of the same numbers. When the listing holds any .safetensors, every
# other weight format is left behind.
#
# HFGET_JOBS concurrent transfers, default 4. These are long sequential reads: a few streams
# saturate the link and more only makes the destination seek.
set -e
# ---- child mode: one file, named by "<size><tab><path>" in $1, everything else inherited -------
# sh cannot export a function, so parallelism is one re-entry of this script per file under
# `xargs -P` rather than a worker pool.
if [ -n "$HFGET_CHILD" ]; then
size=${1%% *}; path=${1#* }
out=$HFGET_DEST/$path
if [ -f "$out" ]; then
cur=$(wc -c < "$out")
if [ "$cur" = "$size" ]; then echo " = $path"; exit 0; fi
echo " + $path (resuming at $cur of $size)"
else
echo " > $path"
fi
mkdir -p "$(dirname "$out")"
if [ -n "$HF_TOKEN" ]; then
curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors -C - \
-H "Authorization: Bearer $HF_TOKEN" \
-o "$out" "https://huggingface.co/$HFGET_REPO/resolve/$HFGET_REV/$path"
else
curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors -C - \
-o "$out" "https://huggingface.co/$HFGET_REPO/resolve/$HFGET_REV/$path"
fi
got=$(wc -c < "$out")
[ "$got" = "$size" ] || { echo "hfget: $path is $got bytes, listing says $size" >&2; exit 1; }
exit 0
fi
# ---- parent ------------------------------------------------------------------------------------
repo=$1
[ -n "$repo" ] || { echo "usage: hfget.sh <repo-id> [dest-dir] [revision]" >&2; exit 2; }
dest=${2:-$HOME/models/$repo}
rev=${3:-main}
jobs=${HFGET_JOBS:-4}
command -v jq >/dev/null || { echo "hfget: jq is required" >&2; exit 2; }
echo "hfget: listing $repo@$rev"
# THE LISTING IS PAGINATED AND THE PAGE SIZE IS SMALL. A large checkpoint runs to a hundred-odd
# shards and the API returns fifty entries with a `Link: ...; rel="next"` header. Taking only the
# first page is the worst possible failure here: every file it did fetch is the right length, so
# the size check passes and the converter is handed a checkpoint missing two thirds of its
# tensors. Follow the cursor until there is no next link.
api="https://huggingface.co/api/models/$repo/tree/$rev?recursive=true&expand=true"
listing='[]'
page=0
while [ -n "$api" ]; do
hdr=$(mktemp "${TMPDIR:-/tmp}/hfhdr.XXXXXX")
if [ -n "$HF_TOKEN" ]; then
body=$(curl -sSL --fail-with-body -D "$hdr" -H "Authorization: Bearer $HF_TOKEN" "$api")
else
body=$(curl -sSL --fail-with-body -D "$hdr" "$api")
fi
[ "$(printf '%s' "$body" | jq -r 'type')" = "array" ] || {
rm -f "$hdr"
echo "hfget: $(printf '%s' "$body" | jq -r '.error // .message // .')" >&2
echo "hfget: a gated or private repo needs HF_TOKEN in the environment" >&2; exit 1; }
listing=$(printf '%s\n%s' "$listing" "$body" | jq -s 'add')
api=$(tr -d '\r' < "$hdr" | sed -n 's/^[Ll]ink: *<\([^>]*\)>; *rel="next".*/\1/p' | head -1)
rm -f "$hdr"
page=$((page + 1))
[ "$page" -gt 200 ] && { echo "hfget: listing did not terminate after 200 pages" >&2; exit 1; }
done
echo "hfget: $page page(s) of listing"
have_st=$(printf '%s' "$listing" | jq '[.[] | select(.path | endswith(".safetensors"))] | length')
sel='.[] | select(.type == "file")'
[ "$have_st" -gt 0 ] && sel="$sel"' | select((.path | test("\\.(bin|pth|msgpack|h5|onnx|onnx_data|gguf)$")) | not)'
list=$(mktemp "${TMPDIR:-/tmp}/hfget.XXXXXX")
trap 'rm -f "$list"' EXIT INT TERM
printf '%s' "$listing" | jq -r "$sel"' | "\(.size)\t\(.path)"' > "$list"
n=$(wc -l < "$list")
[ "$n" -gt 0 ] || { echo "hfget: the listing has no files" >&2; exit 1; }
need=$(awk -F'\t' '{s += $1} END {printf "%.1f", s / 1073741824}' "$list")
mkdir -p "$dest"
free=$(df -BG --output=avail "$dest" | tail -1 | tr -dc '0-9')
echo "hfget: $n file(s), $need GiB -> $dest"
[ "$have_st" -gt 0 ] && echo "hfget: safetensors present, pickle weight formats skipped"
echo "hfget: $free GiB free on the destination filesystem"
awk -v n="$need" -v f="$free" 'BEGIN { if (n > f) { print "hfget: THE DOWNLOAD DOES NOT FIT"; exit 1 } }'
export HFGET_CHILD=1 HFGET_DEST="$dest" HFGET_REPO="$repo" HFGET_REV="$rev"
self=$(cd "$(dirname "$0")" && pwd)/$(basename "$0")
tr '\n' '\0' < "$list" | xargs -0 -P "$jobs" -I{} sh "$self" "{}" \
|| { echo "hfget: incomplete -- re-run to resume" >&2; exit 1; }
echo "hfget: complete -- $(du -sh "$dest" | cut -f1) in $dest"
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
# kld-gate-swift15.sh — RUNS ON DANKSERVER. Score the quantized candidate
# against the BF16 reference and apply the pre-approved gate:
# mean KL <= 0.03 AND top-1 agreement >= 95%.
# Same engine (1.0.13) and same boot shape as the reference RECORD
# (run-swift15-bf16.sh kld): disk-tier, no speculation, deterministic, fp8 KV.
# --kld-ref reads its tokens from the reference (no --kld-corpus here — the
# (run-swift15-bf16.sh kld): 1.0.13, tp 2, disk-tier, fp8 KV, no speculation,
# deterministic, max-model-len 16384, vram-kv 3072, 4 slots. Two deviations
# are forced by the w4 file: MNBT 2048 (the w4 kernels stream from the file
# tier ONLY at MNBT<=2048 — at the recording's 32768 the engine demands a
# widened form: "DID NOT FIT ... stored form is not the container's"), and
# pool 40960 (the recording's 12 GiB pool predates the w4 tables; at 12288 the
# gate_up slab cannot be cached). Chunk size is softmax-invariant, so the
# MNBT delta is float-order noise riding on the KL floor.
set -uo pipefail
PIN=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
C=swift15gate
LOG=/home/dank/logs/swift15_kldgate.log
KLD_REF_DIR=/home/dank/corpora/kld-ref
KLD_OUT=/home/dank/radiance-state/kld-gate.json
[ -d "$KLD_REF_DIR" ] || { echo "missing reference: $KLD_REF_DIR"; exit 2; }
: > "$KLD_OUT"
cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
/home/dank/models/tpl.jinja 2>/dev/null || true
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
podman rm -f "$C" >/dev/null 2>&1 || true
podman run --rm --name "$C" \
--network host --init \
--security-opt seccomp=unconfined \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v /home/dank/models:/models:ro \
-v /home/dank/radiance-state:/data \
-v "$KLD_REF_DIR:/data/kld-ref:ro" \
"stilldeadcode/radiance@$PIN" \
--model /models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad \
--tp 2 --placement expert_tiered --weights-disk-tier \
--host-pool-mib 40960 --deterministic --gpu-headroom-mib 768 \
--num-speculative-tokens 0 \
--kv-cache-dtype fp8 --max-model-len 16384 \
--ngram-placement ram \
--override-chat-template /models/tpl.jinja \
--kld-ref /data/kld-ref --kld-out /data/kld-gate.json \
--vram-kv-mib 3072 \
--max-num-seqs 4 --max-num-batched-tokens 2048 \
2>&1 | tee "$LOG"
rc=${PIPESTATUS[0]}
[ "$rc" -eq 0 ] || { echo "KLD GATE BOOT FAILED rc=$rc; log: $LOG"; exit "$rc"; }
python3 - "$KLD_OUT" "$LOG" <<'PY'
import json, sys, re
out, log = sys.argv[1], sys.argv[2]
def num(pat):
m = re.findall(pat, open(log, errors="replace").read(), re.I)
return float(m[-1]) if m else None
kl = top1 = None
try:
d = json.load(open(out))
def find(obj, keys):
if isinstance(obj, dict):
for k, v in obj.items():
if any(s in k.lower() for s in keys) and isinstance(v, (int, float)):
return float(v)
for v in obj.values():
r = find(v, keys)
if r is not None: return r
return None
kl = find(d, ("mean_kl", "mean kl", "kl_mean", "mean"))
top1 = find(d, ("top1", "top_1", "top-1", "agreement"))
if top1 is not None and top1 <= 1.0:
top1 *= 100.0
except Exception as e:
print("report JSON unusable (%s); falling back to log" % e)
if kl is None:
kl = num(r"mean[_ ]kl[^0-9]*([0-9]*\.?[0-9]+(?:e[-+][0-9]+)?)")
if top1 is None:
top1 = num(r"top[-_ ]?1[^0-9]*([0-9]*\.?[0-9]+)")
if kl is None or top1 is None:
print("KLD GATE PARSE FAILED: mean_kl=%s top1=%s — inspect %s / %s" % (kl, top1, out, log))
sys.exit(2)
print("KLD GATE: mean KL = %.4f (max 0.03) | top-1 = %.2f%% (min 95.0)" % (kl, top1))
ok = (kl <= 0.03) and (top1 >= 95.0)
print("KLD GATE " + ("PASS" if ok else "FAIL"))
sys.exit(0 if ok else 1)
PY
gate=$?
echo "kld-gate complete rc=$gate; log: $LOG report: $KLD_OUT"
exit "$gate"
+40
View File
@@ -0,0 +1,40 @@
#!/bin/bash
# kld-then-calib.sh — RUNS ON DANKSERVER. Campaign chain: wait for the KLD
# record to finish, verify it, then boot the 1.2.0 calibration server and
# push the 10M-token corpus. One GPU, so the phases serialize; prod is
# restored by each run script's trap.
set -u
KLOG=/home/dank/logs/swift15-kld-driver.log
CLOG=/home/dank/logs/swift15-calib-driver.log
S=/home/dank/serverAppConfigs-swift15/scripts
# Phase 1: KLD record (SKIP_KLD=1 when the record is already complete —
# kld.json's rows count is the receipt; a mid-run script swap can kill the
# wrapper with rc=127 after the engine exited clean).
if [ "${SKIP_KLD:-0}" != 1 ]; then
while ! grep -q "exited rc=" "$KLOG" 2>/dev/null; do sleep 60; done
RC=$(grep -o "exited rc=[0-9]*" "$KLOG" | tail -1)
if [ "$RC" != "exited rc=0" ]; then echo "KLD FAILED ($RC); calib NOT started"; exit 1; fi
fi
N=$(find /home/dank/corpora/kld-ref -type f | wc -l)
SZ=$(du -sm /home/dank/corpora/kld-ref | cut -f1)
echo "KLD OK files=$N size_mib=$SZ"
[ "$N" -ge 5 ] && [ "$SZ" -ge 1000 ] || { echo "kld-ref looks thin; calib NOT started"; exit 1; }
grep -q '"rows": 50235' /home/dank/corpora/kld-ref/kld.json || {
echo "kld.json rows != 50235; calib NOT started"; exit 1; }
# Phase 2: calibration serve (1.2.0) + corpus push.
nohup bash "$S/run-swift15-bf16.sh" calib > "$CLOG" 2>&1 &
CP=$!
for _ in $(seq 1 160); do
sleep 15
curl -sf localhost:7770/health >/dev/null 2>&1 && break
done
if ! curl -sf localhost:7770/health >/dev/null 2>&1; then
echo "CALIB BOOT FAILED; see /home/dank/logs/swift15_bf16.log"; kill "$CP" 2>/dev/null; exit 1
fi
echo "CALIB UP, pushing corpus"
python3 "$S/calib-push.py" --port 7770 --parallel 16 --container swift15ref --max-rounds 6
PUSH=$?
kill "$CP" 2>/dev/null
sleep 10
echo "CALIB_CHAIN_DONE push=$PUSH grams=$(find /home/dank/corpora/calib -name 'gram.r*.bin' 2>/dev/null | wc -l)"
+317
View File
@@ -0,0 +1,317 @@
#!/usr/bin/env python3
"""Build the two Swift1.5 campaign corpora (calibration + KLD).
calib.jsonl — 10M tokens of agent-shaped RAW text (45% code, 30% prose,
25% tool-call JSON), one {"prompt": text} per line, pushed to
/v1/completions with max_tokens=1 during the calibration run (the Hessian
tap is of the prompt tokens; docs/MOE-W4.md step 2). Code and prose mix
the repo checkout with HF dataset parquets (Frank's call 2026-10-06:
local material is only ~1.6M unique tokens; MOE-W4 wants 10M):
nampdn-ai/tiny-codes (code) and Salesforce/wikitext wikitext-103-raw
(prose). Tool calls are synthesized from commands harvested from the
repo's own scripts plus a fleet-ops set, with argument-shaped variants.
kld.jsonl — 1.8M tokens, one {"prompt", "score_from", "source"} per
line; each prompt is a multi-turn conversation rendered through the
Froggeric v22.5 template with a >= --kld-min-prompt (8192) token target,
and the last 256 tokens (the tail of the assistant answer) are the
scored span: score_from = n - 256. At 8k-token prompts the 1.8M corpus
is ~220 prompts = ~56k scored positions ~= 25 GB of record, matching
the published Flash-Next KLD scale (83k positions, MOE-W4.md). The KLD
reference run and every candidate run consume the SAME file (TOOLS.md
--kld-record / --kld-ref). The KLD corpus is LOCAL material only, so the
file-level disjointness rule (fnv1a64(path) % 5 == 0 -> KLD pool) is
untouched by the HF additions.
Deps: tokenizers, jinja2, pyarrow (venv /home/dank/venvs/radcorpus).
Usage: make-calib-corpus.py --repo ROOT --tokenizer-json PATH
--template PATH --out-dir DIR [--hf-code-parquet F,F]
[--hf-prose-parquet F,F] [--calib-tokens 10000000]
[--kld-tokens 1800000]
"""
import argparse
import json
import os
import random
import re
from jinja2 import Environment, FileSystemLoader
from tokenizers import Tokenizer
CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
".yaml", ".yml", ".bats", ".service", ".container", ".toml")
JSONL_EXTS = (".jsonl", ".json")
DOC_DIR = "serverAppConfigs-docs"
FLEET_CMDS = [
"systemctl status frank-primary --no-pager",
"journalctl -u frank-primary -n 100 --no-pager",
"podman ps --format '{{.Names}} {{.Status}}'",
"podman logs --tail 50 frank-primary",
"curl -s http://192.168.0.235:7770/health",
"curl -s http://192.168.0.235:7770/metrics | grep radiance_",
"df -h /home/dank | tail -1",
"free -g",
"sudo systemctl restart frank-primary",
"cd /home/dank/serverAppConfigs && git pull --ff-only",
"./scripts/deploy.sh dankserver --service frank-primary",
"./tests/run.sh 2>&1 | tail -5",
"jq . /home/dank/models/rad/index.json | head -40",
"python3 -c \"import json; print(json.load(open('config.json'))['num_hidden_layers'])\"",
"ip neigh show 192.168.0.235",
"amdgpu_top --device-info 2>/dev/null | head -20",
"grep -E 'MemAvailable' /proc/meminfo",
"sha256sum model-00001-of-00131.safetensors",
"git log --oneline -10",
"awk '{s+=$1} END {print s}' shard_sizes.txt",
]
TOOL_NAMES = ["bash", "read", "grep", "web_search", "edit", "write"]
WORKDIRS = ["/home/dank", "/home/dank/serverAppConfigs", "/home/dank/models",
"/etc/systemd/system", "/var/log"]
DESCS = ["check service state", "tail the engine ledger", "list containers",
"query health", "read metrics", "check disk", "check memory",
"restart the endpoint", "sync the checkout", "run deployer",
"run the test seams", "inspect the index", "probe the peer"]
def fnv(s: str) -> int:
h = 0xcbf29ce484222325
for b in s.encode():
h = ((h ^ b) * 0x100000001b3) & 0xFFFFFFFFFFFFFFFF
return h
def harvest_cmds(repo: str, rng: random.Random) -> list[str]:
cmds = list(FLEET_CMDS)
pat = re.compile(r"^\s*(?:sudo\s+)?([a-z][\w./-]*(?:\s+\S+){1,6})\s*$")
for root, dirs, files in os.walk(repo):
dirs[:] = [d for d in dirs if not d.startswith(".")]
for f in files:
if not f.endswith(".sh"):
continue
try:
for line in open(os.path.join(root, f), encoding="utf-8",
errors="replace"):
m = pat.match(line.rstrip())
if m and len(m.group(1)) > 12:
cmds.append("ssh dank@192.168.0.235 '"
+ m.group(1)[:90] + "'")
except OSError:
pass
rng.shuffle(cmds)
return cmds[:4000]
def tool_call(rng: random.Random, cmds: list[str]) -> str:
"""One agent turn: 2-4 tool calls with argument-shaped JSON."""
parts = []
for _ in range(rng.randint(2, 4)):
args: dict = {"command": rng.choice(cmds)}
if rng.random() < 0.4:
args["cwd"] = rng.choice(WORKDIRS)
if rng.random() < 0.3:
args["timeout"] = rng.choice([10, 30, 60, 300, 600])
if rng.random() < 0.5:
args["description"] = rng.choice(DESCS)
parts.append('<tool_call>{"name": "%s", "arguments": %s}<|im_end|>'
% (rng.choice(TOOL_NAMES),
json.dumps(args, ensure_ascii=False)))
return "\n".join(parts)
def load_parquet_texts(path: str, kind: str, rng: random.Random,
cap: int) -> list[str]:
import pyarrow.parquet as pq
d = pq.read_table(path).to_pydict()
n = len(next(iter(d.values())))
idx = range(n) if n <= cap else rng.sample(range(n), cap)
out = []
for i in idx:
if kind == "code":
if "content" in d:
frag = d["content"][i]
else:
frag = ("// %s\n### Instruction:\n%s\n### Response:\n%s"
% (d.get("file_name", [""])[i], d["prompt"][i],
d["code"][i] or d["response"][i]))
else:
frag = d["text"][i]
if frag and frag.strip():
out.append(frag[:8000])
rng.shuffle(out)
return out[:cap]
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True)
ap.add_argument("--tokenizer-json", required=True)
ap.add_argument("--template", required=True)
ap.add_argument("--out-dir", required=True)
ap.add_argument("--calib-tokens", type=int, default=10_000_000)
ap.add_argument("--kld-tokens", type=int, default=1_800_000)
ap.add_argument("--chunk-tokens", type=int, default=8192)
ap.add_argument("--score-tail", type=int, default=256)
ap.add_argument("--kld-min-prompt", type=int, default=8192)
ap.add_argument("--hf-code-parquet", default="")
ap.add_argument("--hf-prose-parquet", default="")
ap.add_argument("--seed", type=int, default=20261006)
a = ap.parse_args()
rng = random.Random(a.seed)
tokenizer = Tokenizer.from_file(a.tokenizer_json)
env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
trim_blocks=True, lstrip_blocks=True)
env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
ValueError(m))
tname = os.path.basename(a.template)
code, prose, jsonl = [], [], []
for root, dirs, files in os.walk(a.repo):
dirs[:] = [d for d in dirs if not d.startswith(".")]
for f in sorted(files):
p = os.path.relpath(os.path.join(root, f), a.repo)
if DOC_DIR in p and f.endswith(".md"):
prose.append(p)
elif p.startswith(DOC_DIR):
pass
elif f.endswith(CODE_EXT):
code.append(p)
elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
jsonl.append(p)
# file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
pools: dict[str, tuple[list, list]] = {}
for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
[p for p in lst if fnv(p) % 5 != 0])
def text_of(p: str) -> str:
path = os.path.join(a.repo, p)
raw = open(path, encoding="utf-8", errors="replace").read()
if p.endswith(JSONL_EXTS):
lines = [json.loads(l)["content"] for l in raw.splitlines()
if l.strip() and "content" in l]
raw = "\n".join(lines)
toks = tokenizer.encode(raw, add_special_tokens=False).ids
return tokenizer.decode(toks[:2400], skip_special_tokens=True)
cmds = harvest_cmds(a.repo, rng)
# fragment pools per class, corpus-split first
frags: dict[str, dict[str, list[str]]] = {}
for cls in pools:
kld_files, cal_files = pools[cls]
fr = {}
src = {"kld": kld_files, "cal": cal_files}
if cls == "tools":
fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
else:
fr["kld"] = [text_of(p) for p in src["kld"]]
fr["cal"] = [text_of(p) for p in src["cal"]]
frags[cls] = fr
# per-class fragment caps: prose rows are short (~66 tok) so need many;
# codeparrot files are long (~2.6k tok) so a small cap suffices
hfcap = {"code": 40000, "prose": 60000, "tools": 0}
for cls, spec in (("code", a.hf_code_parquet),
("prose", a.hf_prose_parquet)):
for path in [p for p in spec.split(",") if p]:
frags[cls]["cal"].extend(
load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
for cls in frags:
rng.shuffle(frags[cls]["cal"])
def toks(s: str) -> int:
return len(tokenizer.encode(s, add_special_tokens=False).ids)
def emit_calib() -> tuple[int, list[dict]]:
targets = {"code": .45, "prose": .30, "tools": .25}
rows: list[dict] = []
for cls, frac in targets.items():
want = int(a.calib_tokens * frac)
pool = frags[cls]["cal"]
if not pool:
continue
got, i, buf, nbuf = 0, 0, [], 0
while got < want:
frag = pool[i % len(pool)]
i += 1
n = toks(frag)
if nbuf + n > a.chunk_tokens and buf:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
buf, nbuf = [], 0
if got >= want:
break
buf.append(frag)
nbuf += n
if i > len(pool) * 40:
break # pool exhausted with repeats; stop
if buf and got < want:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
print(f"calib {cls}: {got} tokens, {i} fragments")
rng.shuffle(rows)
return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
for r in rows]
def emit_kld() -> tuple[int, list[dict]]:
rows, total = [], 0
classes = ["code", "prose", "tools"]
i = {c: 0 for c in classes}
guard = 0
while total < a.kld_tokens and guard < 5000:
guard += 1
msgs, srcs = [], []
for t in range(rng.choice([2, 3, 4])):
cls = classes[(len(rows) + t) % 3]
pool = frags[cls]["kld"]
if not pool:
continue
frag = pool[i[cls] % len(pool)]
i[cls] += 2
msgs.append({"role": "user",
"content": "Summarize and explain this "
f"engineering material:\n{frag}"})
msgs.append({"role": "assistant",
"content": frag[:4000]})
srcs.append(cls)
rendered = env.get_template(tname).render(
messages=msgs, add_generation_prompt=False)
n = toks(rendered)
if n < a.kld_min_prompt:
continue
rows.append({"prompt": rendered,
"score_from": n - a.score_tail,
"source": "+".join(sorted(set(srcs)))})
total += n
return len(rows), rows
os.makedirs(a.out_dir, exist_ok=True)
nc, rows = emit_calib()
with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
nk, krows = emit_kld()
with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
for r in krows:
f.write(json.dumps(r) + "\n")
scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
- r["score_from"] for r in krows)
manifest = {
"seed": a.seed, "template": tname,
"tokenizer": os.path.basename(a.tokenizer_json),
"calib_prompts": nc, "kld_prompts": nk,
"kld_scored_positions": scored,
"kld_score_tail": a.score_tail,
"disjoint_file_pools": "local KLD material only; HF code feeds calib",
"calib_mix": {"code": .45, "prose": .30, "tools": .25},
"hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
}
with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
json.dump(manifest, f, indent=1)
print(json.dumps(manifest, indent=1))
if __name__ == "__main__":
main()
@@ -0,0 +1,86 @@
# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
#
# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
#
# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
#
# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
# says so on every boot:
#
# Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
# provides neither a compiled submodule nor breakable CUDA graphs.
# Overriding to FULL_DECODE_ONLY.
#
# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
# generations at the deployed shape) shows what the missing compilation costs. Total
# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
# time of ~104 ms (9.6 tok/s with the profiler attached):
#
# 16.07 s 34.8 % 708,500 launches vectorized_elementwise_kernel (22.7 us each)
# 4.60 s 10.0 % 42,000 r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
# 3.11 s 6.7 % 174,708 elementwise_kernel_manual_unroll
# 2.94 s 6.4 % 138,252 wvSplitK_hf_sml_ (split-K GEMM)
# 4.09 s 8.9 % 83,160 topk/router (sbtopk gatherTopK + warpMergeSortTopK
# + bitonicSortKVInPlace)
# 3.01 s 6.5 % 133,920 r4d_mhc_* (hyper-connection pre/post/finish)
# 2.78 s 6.0 % 127,916 inductor triton_poi_* / triton_per_* (the only
# fusions that DO happen)
# 0.49 s 1.05% 20,664 r4d_moe_lru_gather_kernel <-- the suspect
#
# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
# that touches that mass; the router is second at 9 % but is binary-blocked (the
# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
# and GLM has 288 experts, with no HIP sources shipped to rebuild).
#
# TARGET: vllm/models/glm5next/common/model.py (inside the pinned image)
# base sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
# patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
#
# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
# no-op, so the file can stay mounted for an A/B without a second artifact.
# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
# the first probe call inside the traced region.
--- a/vllm/models/glm5next/common/model.py
+++ b/vllm/models/glm5next/common/model.py
@@ -1,12 +1,14 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+import os
from collections.abc import Iterable
from typing import ClassVar, Literal
import torch
from torch import nn
+from vllm.compilation.decorators import support_torch_compile
from vllm.config import ParallelConfig, VllmConfig
from vllm.distributed import (
get_ep_group,
@@ -623,6 +625,18 @@
)
+_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
+
+
+@support_torch_compile(
+ dynamic_arg_dims={
+ "input_ids": 0,
+ "positions": -1,
+ "intermediate_tensors": 0,
+ "inputs_embeds": 0,
+ },
+ enable_if=lambda _cfg: _GLM_COMPILE,
+)
class Glm5NextModel(nn.Module):
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
super().__init__()
@@ -0,0 +1,119 @@
# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
# inside a torch.compile region.
#
# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
# Inert (but harmless) without 01. Not part of the deployed shape.
#
# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
# profile_run, i.e. the first — cold — forward):
#
# Explanation: Dynamo does not know how to trace the builtin `posix.stat`
# ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
#
# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
# re-traces the wrapped body to build its guard even when the cache is warm — it warns
# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
#
# cold lru_cache call -> RAISE Unsupported
# WARM lru_cache call -> RAISE Unsupported <-- warming is not enough
# plain module-global read -> compiles
# uncached os.path.exists -> RAISE Unsupported
#
# So each probe must resolve ONCE into a plain value and be read as a global. The set
# below is every no-argument probe / symbol lookup reachable from a forward pass,
# grep-verified call sites:
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79 available()
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81 has_w4a16_moe()
# model_executor/kernels/mhc/r4d_dsv4.py:16 r4d_mhc_enabled()
# v1/attention/ops/rocm_aiter_mla_sparse.py:694 r4d_indexer_enabled(), mqa_max_heads()
# v1/attention/ops/rocm_aiter_mla_sparse.py:1124 r4d_indexer_enabled()
# model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry() moe_w4a16()/moe_w4a16_mt1() per MoE call
# (lib() is the transitive dependency of all of them.)
#
# TARGET: vllm/model_executor/kernels/r4d_lib.py (inside the pinned image)
# base sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
# patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
#
# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
#
# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
--- a/vllm/model_executor/kernels/r4d_lib.py
+++ b/vllm/model_executor/kernels/r4d_lib.py
@@ -361,3 +361,71 @@
@functools.lru_cache
def moe_w4a16_mt1():
return getattr(lib(), "r4d_gemm_moe_w4a16_nt_b16_m64_mt1", None)
+
+
+# FORK-LOCAL (patch: trace-safe capability probes). torch.compile traces the
+# BODY of a functools.lru_cache call even when the cache is warm, so a capability
+# probe called from a forward pass is a dynamo graph break. Measured 2026-09-30
+# on the GLM-5.3-Flash trunk under @support_torch_compile:
+# Dynamo does not know how to trace the builtin `posix.stat`
+# r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
+# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
+# (outside any tracing region), so a traced call is a dict read. The probe set is
+# every no-argument capability probe / symbol lookup reachable from a forward
+# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
+# and the draft/lm-head guards.
+_PROBE_NAMES = (
+ "available",
+ "lib",
+ "has_w4a16_moe",
+ "moe_w4a16",
+ "moe_w4a16_mt1",
+ "has_mhc_dsv4",
+ "r4d_mhc_enabled",
+ "r4d_indexer_enabled",
+ "mqa_max_heads",
+ "r4d_act_quant_enabled",
+)
+
+
+def _freeze_probe(fn):
+ """Return fn bound to its first result. Result must be a plain value."""
+
+ memo = []
+
+ def probe(*args, **kwargs):
+ if not memo:
+ memo.append(fn(*args, **kwargs))
+ return memo[0]
+
+ return probe
+
+
+def _freeze_probes() -> list[str]:
+ frozen = []
+ for name in _PROBE_NAMES:
+ fn = globals().get(name)
+ if fn is None:
+ continue
+ # Only the fast branch of a bound probe is traced; an empty memo means the
+ # first call still does I/O, so populate it here as well.
+ globals()[name] = _freeze_probe(fn)
+ frozen.append(name)
+ return frozen
+
+
+from vllm.logger import init_logger as _init_logger
+
+_probe_log = _init_logger(__name__)
+
+if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
+ _warmed, _failed = [], []
+ for _n in _freeze_probes():
+ try:
+ globals()[_n]()
+ _warmed.append(_n)
+ except Exception: # noqa: BLE001 -- probe unavailable here; recompute lazily
+ _failed.append(_n)
+ _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
+ len(_warmed), _warmed,
+ f" (deferred: {_failed})" if _failed else "")
@@ -0,0 +1,116 @@
# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
# of failing on a break.
#
# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
# yet boot the engine; nothing in scripts/ mounts it.
#
# THE BREAK: with 01+02 mounted the worker dies with
#
# RuntimeError: Graph breaks are not supported with aot compile.
# Please use torch.compile(fullgraph=True).
#
# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
# satisfy fullgraph: its indexer metadata path syncs to the host —
# models/glm5next/amd/sparse_indexer.py:263 int(positions[...].max().item()) + 1
# models/glm5next/amd/sparse_indexer.py:419 lmax = int(decode_metadata.decode_lens.max().item())
# — and a .item() inside a captured graph is not expressible.
#
# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
#
# fullgraph=True suppress_errors=False -> RAISE Unsupported
# fullgraph=True suppress_errors=True -> RAISE Unsupported <-- suppress_errors
# does NOT cover breaks
# fullgraph=False suppress_errors=False -> ok (traceable segments compiled by
# inductor, the rest runs eagerly;
# numerics unchanged)
#
# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
# so every other model keeps stock behaviour), and the run also needs
# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
#
# TARGET: vllm/compilation/wrapper.py (inside the pinned image)
# base sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
# patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
# (the file already imports os; the patch does not add a second import)
#
# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run
# on the decode forward:
#
# KeyError: 'in_proj_qkvbfg_a'
# model.py:748 Glm5NextModel.forward hidden_states, residual, post, comb = layer(...)
# model.py:456 Glm5NextDecoderLayer.forward
#
# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
# (models/glm5next/common/kda.py:217, called at :407) — and it is also a name in the
# load_weights shard table (model.py:780-785), i.e. a real submodule attribute name,
# not an arbitrary string.
#
# WHERE IT IS THROWN (established 2026-09-30 by reading the full traceback, since the
# report line is inside dynamo-rewritten code): torch/_dynamo keeps a name -> module map
# and looks it up by name — output_graph.py:1259 / :1384 register into `nn_modules`,
# :3000 does `self.nn_modules[node.target]` (a bare-name KeyError when absent), and
# :1225 documents stripping "_buffers[..]/_parameters[..]/_modules[..]" from those
# names. So the compiler is failing to RESOLVE A SUBMODULE BY NAME while tracing the
# GLM decoder layer. Whether the root cause is the fork's module tree (something
# dynamo cannot register statically) or dynamo's own registration is NOT yet settled.
#
# STATUS OF THAT BLOCKER (updated 2026-09-30, after the diagnostics below):
# * TORCHDYNAMO_VERBOSE=1 adds NO internal stack trace -> it is a plain Python
# KeyError raised in a callee dynamo INLINED into Glm5NextDecoderLayer.forward
# (the reported line 456 is that method's `def` line, which is how inlined frames
# surface).
# * It is NOT raised from dynamo's four nn_modules lookups: the probe in
# /home/dank/dynamo-probe wraps three of them and stayed silent over two runs with
# the probe confirmed installed in the API-server and worker processes.
# * It is NOT a literal string lookup: the name appears tree-wide only as the KDA
# layer's attribute (kda.py:217, used at :407) and in the load_weights shard table
# (model.py:780-785). The four `proj_name = prefix.split(".")[-1]` sites in the
# quantization utils do compute exactly this bare key, but each is guarded
# (`if proj_name not in fused_mapping: return`), so none of them can raise it.
# * It reproduces on every run of this configuration (5 of 5).
# REMAINING INSTRUMENT: name the inlined caller. Either `TORCH_LOGS=+dynamo` on a boot
# and grep the log for the key (dynamo prints inlined frames with source), or wrap the
# inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post /
# hc_fused_post_pre in layers/mhc.py) and the indexer op — with a KeyError-reporting
# probe. Until that names the caller this is a dynamo-side name-resolution failure.
# NOTE the launcher gotcha that wasted a boot earlier: the scratch launcher now
# preserves ENVARGS (`ENVARGS="${ENVARGS:-}"`), but it previously reset it, silently
# dropping outer env vars.
#
# SECOND BLOCKER, KNOWN AND NOT FATAL IN THIS CONFIG: the indexer's in-forward
# workspace acquisition (patch 05 makes it compiler-opaque) and the fused-norm
# launcher cell (patch 04 removes the call) are real breaks, but with
# VLLM_COMPILE_FULLGRAPH=0 they are tolerated — they only become fatal under
# fullgraph=True, which this trunk can never satisfy anyway.
--- a/vllm/compilation/wrapper.py
+++ b/vllm/compilation/wrapper.py
@@ -19,6 +19,15 @@
logger = init_logger(__name__)
+
+# FORK-LOCAL (patch: fullgraph opt-out). torch.compile(fullgraph=True) turns any
+# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
+# (verified in-image, torch 2.11+rocm10.0). With VLLM_COMPILE_FULLGRAPH=0 the
+# traceable segments are still compiled by inductor and the untraceable ones run
+# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
+# indexer) needs. Default 1 keeps stock behaviour for every other model.
+_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
+
R = TypeVar("R")
P = ParamSpec("P")
@@ -146,7 +155,7 @@
with aot_context:
self._compiled_callable = torch.compile(
compiled_ptr,
- fullgraph=True,
+ fullgraph=_FULLGRAPH,
dynamic=False,
backend=backend,
options=options,
@@ -0,0 +1,112 @@
# glm53-compile-04-fused-qkv-rmsnorm-optout.patch — take GLM's fused-norm JIT kernel
# out of the traced path.
#
# STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the
# decorator) to matter; not part of the deployed shape.
#
# MEASUREMENT NOTE — read the run history before trusting any claim below, because a
# launcher bug made four consecutive boots misleading. The scratch launcher
# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the
# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper
# patch while `_FULLGRAPH` still defaulted to True. With fullgraph=True *every* break
# is an error, which is why the reported failure kept moving as breaks were removed:
#
# fullgraph=False, AOT off -> breaks tolerated; dies later on the dynamo name-map
# `KeyError: 'in_proj_qkvbfg_a'`
# fullgraph=True, AOT off -> dies on whichever break comes first
# (cell -> workspace -> compiler-disable)
#
# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference.
# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True.
# Launcher fixed; the env var is set again.
#
# What it means for THIS patch: under fullgraph=True the fused-norm call was the first
# break dynamo reported, and after this patch the report moved on to the workspace
# acquisition — so it removes a real break. It is not by itself what unblocks a boot:
# with breaks genuinely tolerated the blocker is the name-map KeyError.
#
# IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family
# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the
# object this one closes over is bound after use (see below).
#
# THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30). With 01+02+03 the
# trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal
# even with VLLM_COMPILE_FULLGRAPH=0, because dynamo cannot build a resume function
# around it:
#
# torch._dynamo.exc.Unsupported: Read uninitialized cell
# Explanation: Attempted to read a cell variable that has not been populated yet.
# Developer debug context: CellVariable()
#
# User stack:
# glm5next/common/model.py:748 Glm5NextModel.forward layer(...)
# glm5next/common/model.py:526 Glm5NextDecoderLayer.forward
# glm5next/common/attention.py:590
# model_executor/layers/mla.py:208
# q_proj_input, kv_c_normed = fused_q_kv_rmsnorm(...)
# -> torch/_dynamo/polyfills/__init__.py:392 getattr_and_trace
#
# So dynamo is tracing the fork's JIT-kernel launcher framework
# (VllmTritonJitKernel + @kernel_launcher — 39 call sites) and finds a closure name
# that is not yet bound. In models/common/ops/fused_qk_rmsnorm.py the kernel object
# is bound AFTER the function that closes over it:
#
# def fused_q_kv_rmsnorm(...): # line 194
# ...
# _FUSED_Q_KV_RMSNORM_KERNEL(...) # line 211
# _FUSED_Q_KV_RMSNORM_KERNEL = FusedQKVRMSNormKernel() # line 223
#
# Two ways out, and this patch takes the cheap one:
# (a) WORKAROUND (here): stop calling the fused kernel from the traced region. The
# flag is already a constructor argument (mla.py:80, default False) that glm5next
# turns ON at attention.py:583, so it becomes an env knob.
# (b) ROOT FIX (not done): make the launcher trace-safe for all 39 sites. Any other
# @kernel_launcher-backed kernel still in the traced path can produce the same
# fatal break, so (b) is the durable fix and (a) is what unblocks iterating.
#
# COST of R4D_GLM_FUSE_QKV_RMSNORM=0: one extra small RMSNorm launch per MLA layer per
# token (q_a_layernorm + kv_a_layernorm instead of one fused kernel) — ~2 launches out
# of ~3,700 per token on this shape, against the 44 % of device time currently spent in
# unfused pointwise work. Correct trade for a compile experiment.
#
# TARGET: vllm/models/glm5next/common/attention.py (inside the pinned image)
# base sha256 362bbf0e6c50cdfa446c6551b9b1ddff425e08f88e3d4140fd8db88f8a619e0c
# patched sha256 84969993c250e9f8183a33a6e1ddd607272f48b950b560272eb33b7cdde31189
#
# APPLY: `patch -p1` from the vllm package root (verified: applies clean, reproduces
# the patched sha256), or bind-mount new/.../attention.py over site-packages.
# R4D_GLM_FUSE_QKV_RMSNORM=1 (default) restores the fused kernel.
--- a/vllm/models/glm5next/common/attention.py
+++ b/vllm/models/glm5next/common/attention.py
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+import os
import torch
import torch.nn.functional as F
from torch import nn
@@ -36,6 +37,14 @@
from vllm.utils.deep_gemm import PAGED_MQA_PAGE_SIZES
from vllm.v1.kv_cache_interface import KpoolTailSpec, MLAAttentionSpec
+
+# FORK-LOCAL (patch: keep the traced region trace-safe). R4D_GLM_FUSE_QKV_RMSNORM=0
+# replaces the fused `fused_q_kv_rmsnorm` JIT-kernel launch (one kernel) with two plain
+# RMSNorms, because dynamo cannot trace the launcher's closure ("Read uninitialized cell",
+# fatal even with fullgraph=False). The fused kernel object in
+# models/common/ops/fused_qk_rmsnorm.py is bound AFTER the function that uses it.
+_FUSE_QKV_RMSNORM = os.environ.get("R4D_GLM_FUSE_QKV_RMSNORM", "1") == "1"
+
logger = init_logger(__name__)
# Shared torch.compile config for the indexer's small-kernel leaves. The MLA
@@ -580,7 +589,7 @@
quant_config,
prefix,
skip_topk=False,
- fuse_qkv_rmsnorm=True,
+ fuse_qkv_rmsnorm=_FUSE_QKV_RMSNORM,
)
def forward(
@@ -0,0 +1,96 @@
# glm53-compile-05-workspace-acquisition-opaque.patch — the indexer's workspace
# acquisition must not be traced.
#
# STATUS: WIP — fifth of the "compile the GLM trunk" changelist. This is a break worth
# removing, but NOT part of the confirmed blocker and NOT required for the
# break-tolerant route: `torch.compiler.disable` is itself a graph break, so this patch
# helps only a hypothetical fullgraph=True route, where the workspace acquisition would
# be better RESTRUCTURED out of the forward (allocate at init/profile time) than made
# opaque. Kept because the discovery is real and the wrapper is a no-op when nothing is
# compiling.
#
# THE BREAK — CONFIRMED with AND WITHOUT the logging flag (2026-09-30):
#
# torch._dynamo.exc.Unsupported: Unsupported method call
# user stack
# glm5next/common/model.py:748 Glm5NextModel.forward
# glm5next/common/model.py:526 Glm5NextDecoderLayer.forward
# glm5next/common/attention.py:599 / :401
# glm5next/amd/sparse_indexer.py:711 forward_native
# glm5next/amd/sparse_indexer.py:132 sparse_attn_indexer_kpool
# v1/worker/workspace.py:135 get_simultaneous
# v1/worker/workspace.py:155 _ensure_workspace_size
#
# Line 132 is the PROFILE-RUN branch — the source comment says what it is:
#
# if not isinstance(attn_metadata, dict):
# # Reserve workspace for indexer during profiling run
#
# so the engine dies exactly where it sizes its memory. `_ensure_workspace_size`
# reads ContextVar and ubatch state (`_workspace_lane.get()`,
# `dbo_current_ubatch_id()`) and may allocate; none of that is traceable.
#
# FIX: the documented escape hatch — `torch.compiler.disable` around a thin wrapper,
# so dynamo stops analysing the call and executes it eagerly from inside the compiled
# region. Chosen over decorating the workspace manager's own methods because that
# would change workspace behaviour for every model, not just this one. The wrapper is
# a no-op when nothing is compiling, so it needs no env gate and is safe in the
# stock (uncompiled) path — which is why this patch can be reasoned about
# independently of whether the changelist ever ships.
#
# CAVEAT, stated plainly: a compiler-disabled region is still a graph break, so this
# needs patch 03 (VLLM_COMPILE_FULLGRAPH=0) to be tolerable at all, and whether dynamo
# can build a resume function around it is exactly what the next boot decides — the
# difference between this failure and 04's `Read uninitialized cell`.
#
# TARGET: vllm/models/glm5next/amd/sparse_indexer.py (inside the pinned image)
# base sha256 f773a99265d3f440c8fad22c8fb5a8b33f926162094cffc3a2ecdaa42763c5b1
# patched sha256 c154115afe8253a35c652b7198929ef4c1874957d333fdb3340cacf88e612b78
#
# APPLY: `patch -p1` from the vllm package root, or bind-mount over site-packages.
--- a/vllm/models/glm5next/amd/sparse_indexer.py
+++ b/vllm/models/glm5next/amd/sparse_indexer.py
@@ -35,6 +35,21 @@
from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton
from vllm.v1.worker.workspace import current_workspace_manager
+@torch.compiler.disable
+def _acquire_workspace(*shapes_and_dtypes):
+ """Acquire shared workspace buffers, untraced.
+
+ Workspace acquisition reads ContextVar/ubatch state and may allocate
+ (v1/worker/workspace.py: ``_ensure_workspace_size`` -> ``_workspace_lane.get()``,
+ ``dbo_current_ubatch_id()``), which dynamo cannot trace -- it raises
+ ``Unsupported: Unsupported method call`` and kills the engine during the profile
+ run. ``torch.compiler.disable`` keeps the call as an opaque, eagerly-executed
+ region inside the compiled graph. No-op when not compiling.
+ """
+
+ return current_workspace_manager().get_simultaneous(*shapes_and_dtypes)
+
+
logger = init_logger(__name__)
# kpool write helper: form pools from the current token batch and compress them
@@ -129,7 +144,7 @@
values_spec, scales_spec = _gather_workspace_shapes(
total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
)
- current_workspace_manager().get_simultaneous(
+ _acquire_workspace(
values_spec,
scales_spec,
((RADIX_TOPK_WORKSPACE_SIZE,), torch.uint8),
@@ -275,11 +290,10 @@
# Layout switches between FP8 (head_dim bytes + 4-byte fp32 scale) and
# MXFP4 (head_dim/2 bytes packed + head_dim/MXFP4_BLOCK_SIZE ue8m0
# scales) based on use_fp4_cache.
- workspace_manager = current_workspace_manager()
values_spec, scales_spec = _gather_workspace_shapes(
total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
)
- k_quant_full, k_scale_full = workspace_manager.get_simultaneous(
+ k_quant_full, k_scale_full = _acquire_workspace(
values_spec,
scales_spec,
)
@@ -0,0 +1,173 @@
# glm53-dflash-01-aux-hidden-states.patch — give the GLM trunk the EAGLE3 / DFlash2
# auxiliary-hidden-state interface.
#
# STATUS: WIP, and INDEPENDENT of the compile changelist (01-05). It patches the same
# file as glm53-compile-01-model-decorator.patch, so if both are ever applied, apply 01
# first. Not part of the deployed shape.
#
# WHY. `--speculative-config '{"method":"dflash","model":"/path/to/drafter", ...}'` with
# incoai/GLM-5.3-Flash-DFlash2 is accepted -- the config parses, the drafter loads -- and
# then the worker dies at init with:
#
# RuntimeError: Model does not support EAGLE3 interface
#
# raised by v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20 through
# `supports_eagle3(model)` == `isinstance(model, SupportsEagle3)` (a runtime-checkable
# Protocol). Nothing about the drafter, the method name or the payload is wrong:
# `glm5next` implements neither half of the contract.
#
# THE CONTRACT (read out of this image, not assumed):
# * DEFINITIVE (v5, after four failed boots): the member set is checked against
# the INSTANCE the runner calls -- which for GLM-5.3-Flash is
# Glm5NextForConditionalGeneration, NOT Glm5NextForCausalLM. The multimodal
# wrapper wraps the text model as self.language_model
# (init_vllm_registered_model, architectures=["Glm5NextForCausalLM"],
# glm5next/common/model.py:1113), and model_runner.py:425 calls
# set_eagle3_aux_hidden_state_layers(self.model) on the CONDITIONAL.
# Presence rule (verified in-image with stock typing, python 3.14):
# isinstance walks ALL __protocol_attrs__ -- members of SupportsEagle3 AND
# its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
# getattr_static()s each one off the instance, breaking on the first miss.
# So BOTH wrappers need all three attributes.
# * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
# "Cannot create a consistent method resolution order (MRO) for bases ..."
# (GLM's wrappers already carry four/five Protocol-derived bases). The
# attribute route avoids the MRO entirely. Also proven by simulation:
# the four-base ForCausalLM ordering cannot take SupportsEagle3 in ANY
# position.
# * Method placement: without inheriting, the Protocol's default
# set_aux_hidden_state_layers / get_eagle3_default_aux_hidden_state_layers
# do not exist on either wrapper, so both are defined explicitly --
# ForCausalLM unwraps to self.model (the EagleModelMixin trunk, v1's
# correct bodies), the conditional delegates to self.language_model
# (the pixtral.py:498 pattern).
# * interfaces.py:1673-1719 -- for reference, the inherited defaults unwrap
# `holder = getattr(parent_ref, "model", parent_ref)` and assert
# `isinstance(holder, EagleModelMixin)`, defaulting to (2, L // 2, L - 3).
# * eagle3_utils.py:26-31 -- layers come from the spec config when it carries them
# (this drafter ships target_layer_ids [5, 14, 24, 33, 42]), else from the default.
# * deepseek_v2.py:1551-1585 and v1/worker/gpu/cudagraph_utils.py:621 -- the trunk
# appends one tensor per aux layer while looping and returns the TUPLE
# (hidden_states, aux_hidden_states) only when that list is non-empty.
#
# WHAT IT ADDS for GLM: EagleModelMixin on Glm5NextModel; aux capture in the layer
# loop, where the aux state is the single-stream `hidden_states` ENTERING that layer
# (GLM's loop carries the already-summed single-stream state -- its non-mHC branch
# re-splits `residual = hidden_states` inside the layer -- so this is the equivalent
# of deepseek_v2's `hidden_states + residual`); the conditional tuple return; the
# three SupportsEagle3 member attributes on BOTH wrappers; and the two interface
# methods on both (ForCausalLM: unwrap to trunk; conditional: delegate to
# language_model).
#
# HONEST SCOPE: this makes the interface PRESENT, which is exactly what the engine
# checks -- it does not make drafts GOOD, and per the measured profile it is not a
# decode-speed fix (a verification pass pays pointwise cost proportional to its token
# count, which is why MTP4 measured flat). It is the prerequisite for driving DFlash2
# on this endpoint at all.
#
# TARGET: vllm/models/glm5next/common/model.py (inside the pinned image)
# base sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
# patched sha256 bc9913a15c8e7f88e0c36b07bf029b383cc8c2078b58d66dc91aa235f8af77f4
#
# APPLY: `patch -p1` from the vllm package root, or bind-mount the patched file over
# site-packages (which is what a scratch-launcher DFLASH_PATCH=1 does).
--- a/vllm/models/glm5next/common/model.py
+++ b/vllm/models/glm5next/common/model.py
@@ -95,6 +95,12 @@
logger = init_logger(__name__)
+from vllm.model_executor.models.interfaces import (
+ EagleModelMixin,
+ SupportsEagle3,
+)
+
+
class Glm5NextMLP(nn.Module):
def __init__(
self,
@@ -623,7 +629,7 @@
)
-class Glm5NextModel(nn.Module):
+class Glm5NextModel(nn.Module, EagleModelMixin):
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
super().__init__()
@@ -730,7 +736,14 @@
if self.is_sequence_parallel:
hidden_states = sp_shard(hidden_states)
- for layer in self._active_layers:
+ aux_hidden_states: list[torch.Tensor] = []
+ for local_idx, layer in enumerate(self._active_layers):
+ if (getattr(self, "start_layer", 0) + local_idx) in self.aux_hidden_state_layers:
+ # Auxiliary hidden state for EAGLE-3 / DFlash2 drafts: the
+ # single-stream state ENTERING this layer, before its input norm --
+ # the same semantic deepseek_v2 implements by appending
+ # hidden_states + residual at this point in its loop.
+ aux_hidden_states.append(hidden_states)
hidden_states, residual, post, comb = layer(
positions, hidden_states, residual, post, comb
)
@@ -749,6 +762,8 @@
hidden_states = sp_all_gather(hidden_states)[:full_num_tokens]
hidden_states = self.norm(hidden_states)
+ if len(aux_hidden_states) > 0:
+ return hidden_states, aux_hidden_states
return hidden_states
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
@@ -937,6 +952,10 @@
class Glm5NextForCausalLM(
nn.Module, HasInnerState, SupportsPP, MixtureOfExperts, IsHybrid
):
+ supports_eagle3: ClassVar[Literal[True]] = True
+ has_own_lm_head: bool = False
+ has_own_embed_tokens: bool = False
+
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
super().__init__()
self.model_config = vllm_config.model_config
@@ -963,6 +982,14 @@
def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
return self.model.embed_input_ids(input_ids)
+ def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
+ # EAGLE-3 / DFlash2 interface: unwrap to the trunk (the EagleModelMixin).
+ self.model.aux_hidden_state_layers = layers
+
+ def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
+ num_layers = len(self.model.layers)
+ return (2, num_layers // 2, num_layers - 3)
+
def forward(
self,
input_ids: torch.Tensor | None,
@@ -1033,6 +1060,10 @@
class Glm5NextForConditionalGeneration(
Glm4vForConditionalGeneration, HasInnerState, IsHybrid, MixtureOfExperts
):
+ supports_eagle3: ClassVar[Literal[True]] = True
+ has_own_lm_head: bool = False
+ has_own_embed_tokens: bool = False
+
# The text model (KDA + dense-MLA + MoE) is a hybrid mamba model. The
# multimodal wrapper must declare the same interfaces so vLLM treats it as
# hybrid (auto-aligns mamba/attention block sizes, sizes the mamba state
@@ -1076,6 +1107,13 @@
return Glm5NextForCausalLM.get_mamba_state_copy_func()
+ def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
+ # EAGLE-3 / DFlash2: delegate to the language model (pixtral.py:498).
+ self.language_model.set_aux_hidden_state_layers(layers)
+
+ def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
+ return self.language_model.get_eagle3_default_aux_hidden_state_layers()
+
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
super(Glm4vForConditionalGeneration, self).__init__()
config = vllm_config.model_config.hf_config
@@ -0,0 +1,317 @@
--- a/vllm/v1/core/kv_cache_utils.py
+++ b/vllm/v1/core/kv_cache_utils.py
@@ -1210,17 +1210,38 @@
for name, spec in kv_cache_spec.items()
if isinstance(spec, KpoolTailSpec)
}
+ hidden_specs = {
+ name: spec
+ for name, spec in kv_cache_spec.items()
+ if isinstance(spec, HiddenStateCacheSpec)
+ }
attn_specs = {
name: spec
for name, spec in kv_cache_spec.items()
- if not isinstance(spec, (MambaSpec, KpoolTailSpec))
+ if not isinstance(
+ spec, (MambaSpec, KpoolTailSpec, HiddenStateCacheSpec)
+ )
}
- if not mamba_specs or not all(
- type(spec) is MLAAttentionSpec for spec in attn_specs.values()
- ):
+ if not mamba_specs or not attn_specs:
+ return None
+ # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash
+ # drafter, attn_specs also carries the drafter's attention layers. The
+ # GLM MLA layers keep their own group (no page unification -- per-group
+ # pages are the glm5 design); the drafter layers must form one uniform
+ # group of their own, and the hidden-state layers get per-layer groups
+ # like the generic tail.
+ mla_specs = {
+ name: spec
+ for name, spec in attn_specs.items()
+ if type(spec) is MLAAttentionSpec
+ }
+ draft_specs = {
+ name: spec for name, spec in attn_specs.items() if name not in mla_specs
+ }
+ if not mla_specs:
+ return None
+ if draft_specs and UniformTypeKVCacheSpecs.from_specs(draft_specs) is None:
return None
-
- mla_specs = cast(dict[str, MLAAttentionSpec], attn_specs)
idx_pages = {
spec.page_size_bytes for spec in mla_specs.values() if spec.tokens_per_state > 1
}
@@ -1233,7 +1254,7 @@
mla_pages = {mla_specs[name].page_size_bytes for name in mla_names}
assert len(mla_pages) == 1
mla_page = mla_pages.pop()
- uniform_spec = UniformTypeKVCacheSpecs.from_specs(attn_specs)
+ uniform_spec = UniformTypeKVCacheSpecs.from_specs(mla_specs)
assert uniform_spec is not None
tail_group: KVCacheGroupSpec | None = None
@@ -1270,10 +1291,20 @@
for index, name in enumerate(mamba_specs):
mamba_grouped_names[index % num_groups].append(name)
+ draft_group: KVCacheGroupSpec | None = None
+ if draft_specs:
+ draft_uniform = UniformTypeKVCacheSpecs.from_specs(draft_specs)
+ assert draft_uniform is not None
+ draft_group = KVCacheGroupSpec(list(draft_specs), draft_uniform)
+ hidden_groups = [
+ KVCacheGroupSpec([name], spec) for name, spec in hidden_specs.items()
+ ]
return (
- [KVCacheGroupSpec(list(attn_specs), uniform_spec)]
+ [KVCacheGroupSpec(list(mla_specs), uniform_spec)]
+ ([tail_group] if tail_group is not None else [])
+ + ([draft_group] if draft_group is not None else [])
+ create_kv_cache_group_specs(padded_specs, mamba_grouped_names)
+ + hidden_groups
)
@@ -1289,10 +1320,17 @@
int,
list[str],
int,
+ KVCacheGroupSpec | None,
+ list[KVCacheGroupSpec],
]
| None
):
- """Recognize the GLM-5.3-Flash grouping after optional PP projection."""
+ """Recognize the GLM-5.3-Flash grouping after optional PP projection.
+
+ FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): also recognizes the
+ drafter's uniform group and the per-layer hidden-state groups, returned
+ as ``draft_group`` / ``hidden_groups`` so the pool builder and the memory
+ estimator can size their tensors beside the aliased GLM pool."""
uniform_groups = [
group
for group in kv_cache_groups
@@ -1301,17 +1339,28 @@
mamba_groups = [
group for group in kv_cache_groups if isinstance(group.kv_cache_spec, MambaSpec)
]
+ hidden_groups = [
+ group
+ for group in kv_cache_groups
+ if isinstance(group.kv_cache_spec, HiddenStateCacheSpec)
+ ]
attn_group: KVCacheGroupSpec | None = None
tail_group: KVCacheGroupSpec | None = None
+ draft_group: KVCacheGroupSpec | None = None
for group in uniform_groups:
inner = cast(UniformTypeKVCacheSpecs, group.kv_cache_spec).kv_cache_specs
if all(type(spec) is MLAAttentionSpec for spec in inner.values()):
attn_group = group
elif all(isinstance(spec, KpoolTailSpec) for spec in inner.values()):
tail_group = group
+ else:
+ draft_group = group
if attn_group is None or not mamba_groups:
return None
- if len(uniform_groups) + len(mamba_groups) != len(kv_cache_groups):
+ if (
+ len(uniform_groups) + len(mamba_groups) + len(hidden_groups)
+ != len(kv_cache_groups)
+ ):
return None
attn_uniform = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
@@ -1362,6 +1411,8 @@
idx_page,
tail_names,
tail_page,
+ draft_group,
+ hidden_groups,
)
@@ -1395,10 +1446,34 @@
return kv_cache_spec
max_page_size = max(page_sizes)
+ # GLM-5.3-Flash + a draft model: the sparse indexer k_cache page is
+ # tokens_per_state x the MLA page (same per-token bytes), and MLA layers
+ # cannot be PADDED to an arbitrary page size (sparse MLA indexes the cache
+ # in whole token rows) -- but they CAN grow their block size when the max
+ # page is an exact multiple of the MLA page. Bump the max page to a
+ # multiple of every MLA-family page so all of them take the divisible
+ # block-growth branch below instead of the padding refusal, while the
+ # draft and mamba layers simply pad to the (slightly larger) page.
+ mla_pages = {
+ spec.page_size_bytes
+ for spec in kv_cache_spec.values()
+ if isinstance(spec, MLAAttentionSpec)
+ }
+ if mla_pages:
+ mla_common = math.lcm(*mla_pages)
+ max_page_size = -(-max_page_size // mla_common) * mla_common
new_kv_cache_spec = {}
for layer_name, layer_spec in kv_cache_spec.items():
if layer_spec.page_size_bytes == max_page_size:
new_kv_cache_spec[layer_name] = layer_spec
+ elif isinstance(layer_spec, KpoolTailSpec):
+ # The kpool tail kernel pins the tail block to the kpool size
+ # (tail_kv_cache.view(...) must be [blocks, 2, index_kpool,
+ # head_dim]); growing block_size breaks that contract. Pad the
+ # physical page instead -- the tail is its own group and the pad
+ # wastes bytes only.
+ new_spec = replace(layer_spec, page_size_padded=max_page_size)
+ new_kv_cache_spec[layer_name] = new_spec
elif isinstance(layer_spec, MambaSpec):
# MambaSpec's page size is determined by its state shapes and does
# not scale with block_size, so pad the page instead. This is the
@@ -1593,8 +1668,31 @@
) -> int:
"""Return the largest cache group's bytes per block."""
if (glm5_layout := _glm5_next_tensor_layout(kv_cache_groups)) is not None:
- _, _, mla_names, idx_names, mla_page, idx_page, _, _ = glm5_layout
- return len(mla_names) * mla_page + len(idx_names) * idx_page
+ (
+ _,
+ _,
+ mla_names,
+ idx_names,
+ mla_page,
+ idx_page,
+ _,
+ _,
+ draft_group,
+ hidden_groups,
+ ) = glm5_layout
+ bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
+ if draft_group is not None:
+ d_inner = cast(
+ UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+ ).kv_cache_specs
+ bytes_per_block += sum(
+ spec.page_size_bytes for spec in d_inner.values()
+ )
+ for group in hidden_groups:
+ bytes_per_block += (
+ group.kv_cache_spec.page_size_bytes * len(group.layer_names)
+ )
+ return bytes_per_block
bytes_per_block = max(
sum(
@@ -1701,12 +1799,30 @@
idx_page,
tail_names,
_,
+ draft_group,
+ hidden_groups,
) = glm5_layout
bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
+ extra_bytes_per_block = 0
+ if draft_group is not None:
+ d_inner = cast(
+ UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+ ).kv_cache_specs
+ extra_bytes_per_block += sum(
+ spec.page_size_bytes for spec in d_inner.values()
+ )
+ for group in hidden_groups:
+ extra_bytes_per_block += (
+ group.kv_cache_spec.page_size_bytes * len(group.layer_names)
+ )
num_blocks = may_override_num_blocks(
- vllm_config, available_memory // bytes_per_block
+ vllm_config,
+ available_memory // (bytes_per_block + extra_bytes_per_block),
)
- size = bytes_per_block * num_blocks
+ # One backing allocation shared by every tensor (allocate_kv_cache
+ # asserts all tensor sizes equal): the aliased GLM pool plus the
+ # draft/hidden regions appended after it.
+ size = (bytes_per_block + extra_bytes_per_block) * num_blocks
attn_specs = cast(
UniformTypeKVCacheSpecs, attn_group.kv_cache_spec
).kv_cache_specs
@@ -1745,6 +1861,37 @@
).kv_cache_specs
add_tensor(tail_name, tail_specs[tail_name], offset)
+ # Draft + hidden-state layers get their own tensors appended after
+ # the aliased pool (no aliasing: independent block tables).
+ offset_cursor = bytes_per_block * num_blocks
+ if draft_group is not None:
+ d_inner = cast(
+ UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+ ).kv_cache_specs
+ d_page = next(iter(d_inner.values())).page_size_bytes
+ kv_cache_tensors.append(
+ KVCacheTensor(
+ size=size,
+ layers=list(draft_group.layer_names),
+ layer_stride=d_page * num_blocks,
+ block_stride=d_page,
+ offset=offset_cursor,
+ )
+ )
+ offset_cursor += d_page * num_blocks * len(draft_group.layer_names)
+ for group in hidden_groups:
+ h_page = group.kv_cache_spec.page_size_bytes
+ kv_cache_tensors.append(
+ KVCacheTensor(
+ size=size,
+ layers=list(group.layer_names),
+ layer_stride=h_page * num_blocks,
+ block_stride=h_page,
+ offset=offset_cursor,
+ )
+ )
+ offset_cursor += h_page * num_blocks * len(group.layer_names)
+
return KVCacheConfig(
num_blocks=num_blocks,
kv_cache_tensors=kv_cache_tensors,
@@ -2440,6 +2587,8 @@
idx_page,
tail_names,
_,
+ draft_group,
+ hidden_groups,
) = glm5_layout
uniform_spec = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
total_blocks = uniform_spec.max_memory_usage_pages(vllm_config)
@@ -2452,7 +2601,30 @@
)
if tail_names:
total_blocks += 1
- return total_blocks * (len(mla_names) * mla_page + len(idx_names) * idx_page)
+ pool_bytes = total_blocks * (
+ len(mla_names) * mla_page + len(idx_names) * idx_page
+ )
+ extra_bytes = 0
+ if draft_group is not None:
+ d_inner = cast(
+ UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+ ).kv_cache_specs
+ d_spec = next(iter(d_inner.values()))
+ d_blocks = cdiv(
+ d_spec.max_memory_usage_bytes(vllm_config), d_spec.page_size_bytes
+ )
+ extra_bytes += d_blocks * d_spec.page_size_bytes * len(
+ draft_group.layer_names
+ )
+ for group in hidden_groups:
+ h_spec = group.kv_cache_spec
+ h_blocks = cdiv(
+ h_spec.max_memory_usage_bytes(vllm_config), h_spec.page_size_bytes
+ )
+ extra_bytes += (
+ h_blocks * h_spec.page_size_bytes * len(group.layer_names)
+ )
+ return pool_bytes + extra_bytes
bytes_per_block = _pool_bytes_per_block(kv_cache_groups)
total_blocks = 0
@@ -0,0 +1,31 @@
--- a/vllm/model_executor/offloader/expert_activation_registry.py
+++ b/vllm/model_executor/offloader/expert_activation_registry.py
@@ -316,7 +316,27 @@
def _glm53_kv_bytes_per_token(cfg: "VllmConfig") -> int:
- return 13600
+ per_tok = 13600
+ # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash/dspark
+ # drafter, the KV spec set gains the drafter's attention layers and the aux
+ # hidden-state cache (one bf16 hidden vector per target aux layer per
+ # token). Both are full-length per-token costs the aliased glm5 pool
+ # cannot absorb, so the reserve must include them. The drafter is a 5-layer
+ # qwen3 (8 kv heads x 128 x K+V x bf16 = 4096 B/token); the aux count is
+ # the dflash target_layer_ids count (5).
+ # NOTE: use_aux_hidden_state_outputs lives on the gpu_model_runner
+ # instance (gpu_model_runner.py:623/678), not on the spec config -- it is
+ # not set yet when the expert plan runs. The spec-config METHOD is the
+ # plan-time signal for the same fact.
+ sc = getattr(cfg, "speculative_config", None)
+ if sc is not None and getattr(sc, "method", "") in (
+ "dflash",
+ "dspark",
+ "extract_hidden_states",
+ ):
+ hidden = int(cfg.model_config.hf_config.hidden_size)
+ per_tok += 5 * hidden * 2 + 4096
+ return per_tok
for _arch in ("Glm5NextForConditionalGeneration", "Glm5NextForCausalLM"):
+65
View File
@@ -0,0 +1,65 @@
# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
#
# STATUS: NOT part of the deployed frank-primary shape. It exists because the
# fork's plan under-models this model's KV, and it is the enabler for a future
# MTP attempt (the production GLM shape runs without MTP and boots fine on the
# plan's own reserve).
#
# THE BUG: the plan models KV as 13600 B/token + a small mamba term, but
# GLM-5.3-Flash also carries a per-sequence KDA state (34 linear-attention
# layers) and, under MTP, the draft model's KV. gpu_worker then CAPS the KV
# cache at that reserve, so the plan's own arithmetic can make a legal
# --max-model-len unbootable. Measured 2026-09-30 on the pinned image:
# - no MTP, max_model_len 131072: reserve 1.74 GiB >= need 1.86 GiB -> boots
# - MTP4, max_model_len 131072: reserve 1.74 GiB < need 2.07 GiB -> refuses
# - MTP4, max_model_len 262144: reserve 3.43 GiB < need 3.62 GiB -> refuses
# ("To serve at least one request with the model's max seq len ...")
# Raising max_model_len does not fix it: the cap's slope (~13,845 B/token) only
# overtakes the need's (~12,700 B/token + 0.52 GiB fixed) around 440k tokens,
# where the KV reserve would eat a third of the expert arena.
#
# THE FIX: let an explicit operator KV request raise the cap. The compile and
# graph-capture headroom still comes off the top, so the blast radius is one
# flag the operator chose.
#
# APPLY (into a host dir that is then bind-mounted read-only over the image):
# IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
# D=/home/dank/glm-patch; mkdir -p $D/vllm/v1/worker
# podman run --rm --entrypoint cat $IMAGE \
# /opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py \
# > $D/vllm/v1/worker/gpu_worker.py
# ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch )
# # then add to the podman run:
# # -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro
# # -e CLAV_KV_RESERVE_FLOOR_GIB=<needed> (or pass --kv-cache-memory-bytes)
#
# Base file sha256 (the pristine file out of the pinned image):
# 1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7
# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces
# these bytes exactly):
# 80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840
#
--- a/vllm/v1/worker/gpu_worker.py
+++ b/vllm/v1/worker/gpu_worker.py
@@ -824,6 +824,21 @@
expert_cfg = self.vllm_config.offload_config.expert
if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0:
cap = int(expert_cfg.planned_kv_reserve_bytes)
+ # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as
+ # 13600 B/token + a small mamba term, but this model also carries a
+ # per-sequence KDA state and (under MTP) the draft's KV, so the
+ # reserve can sit BELOW what max_model_len legitimately needs -- and
+ # the cap then makes a legal config unbootable (measured: need
+ # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An
+ # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB,
+ # is the operator's word: let it raise the cap. The compile/capture
+ # headroom still comes off the top.
+ _explicit = self.cache_config.kv_cache_memory_bytes
+ _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB")
+ if _explicit:
+ cap = max(cap, int(_explicit))
+ if _floor:
+ cap = max(cap, int(float(_floor) * (1 << 30)))
if self.available_kv_cache_memory_bytes > cap:
logger.info_once(
"expert offload: KV capped at the plan's reserve %s GiB; %s GiB of "
+460
View File
@@ -0,0 +1,460 @@
{%- set template_version = "qwen3.8-froggeric-v22.5" %}
{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %}
{%- set image_count = namespace(value=0) %}
{%- set video_count = namespace(value=0) %}
{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %}
{%- if preserve_reasoning is defined and preserve_reasoning is not none %}
{%- set _preserve_thinking = preserve_reasoning %}
{%- elif preserve_thinking is defined and preserve_thinking is not none %}
{%- set _preserve_thinking = preserve_thinking %}
{%- else %}
{%- set _preserve_thinking = true %}
{%- endif %}
{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %}
{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %}
{%- set _default_reasoning_effort = 'medium' %}
{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %}
{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %}
{%- set _initial_thinking = enable_thinking %}
{%- if _effort_raw in ('none', 'off') %}
{%- set _initial_thinking = false %}
{%- set _initial_effort = 'medium' %}
{%- elif _effort_raw in ('minimal', 'low') %}
{%- set _initial_effort = 'low' %}
{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %}
{%- set _initial_effort = 'xhigh' %}
{%- else %}
{%- set _initial_effort = 'medium' %}
{%- endif %}
{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %}
{%- if auto_disable_thinking_with_tools and _has_tools %}
{%- set ns_state.thinking = false %}
{%- endif %}
{%- for msg in messages %}
{%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %}
{%- if msg.content is string %}
{%- if '<|think_off|>' in msg.content %}
{%- set ns_state.thinking = false %}
{%- elif '<|think_on|>' in msg.content %}
{%- set ns_state.thinking = true %}
{%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %}
{%- set ns_state.thinking = true %}
{%- set ns_state.effort = 'xhigh' %}
{%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %}
{%- set ns_state.thinking = true %}
{%- set ns_state.effort = 'low' %}
{%- elif '<|think_medium|>' in msg.content %}
{%- set ns_state.thinking = true %}
{%- set ns_state.effort = 'medium' %}
{%- endif %}
{%- elif msg.content is iterable and msg.content is not mapping %}
{%- for item in msg.content %}
{%- if item is string %}
{%- set _item_text = item %}
{%- elif item is mapping and 'text' in item and item.text is string %}
{%- set _item_text = item.text %}
{%- else %}
{%- set _item_text = '' %}
{%- endif %}
{%- if _item_text %}
{%- if '<|think_off|>' in _item_text %}
{%- set ns_state.thinking = false %}
{%- elif '<|think_on|>' in _item_text %}
{%- set ns_state.thinking = true %}
{%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %}
{%- set ns_state.thinking = true %}
{%- set ns_state.effort = 'xhigh' %}
{%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %}
{%- set ns_state.thinking = true %}
{%- set ns_state.effort = 'low' %}
{%- elif '<|think_medium|>' in _item_text %}
{%- set ns_state.thinking = true %}
{%- set ns_state.effort = 'medium' %}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- set reasoning_instructions = '' %}
{%- if ns_state.thinking %}
{%- if ns_state.effort == 'xhigh' %}
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
{%- elif ns_state.effort == 'low' %}
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
{%- endif %}
{%- endif %}
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
{%- if content is string %}
{{- content }}
{%- elif content is iterable and content is not mapping %}
{%- for item in content %}
{%- if item is mapping %}
{%- if item.type == 'image' or 'image' in item or 'image_url' in item %}
{%- if is_system_content %}
{{- raise_exception('System message cannot contain images.') }}
{%- endif %}
{%- if do_vision_count %}
{%- set image_count.value = image_count.value + 1 %}
{%- endif %}
{%- if add_vision_id %}
{{- 'Picture ' ~ image_count.value ~ ': ' }}
{%- endif %}
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
{%- elif item.type == 'video' or 'video' in item or 'video_url' in item %}
{%- if is_system_content %}
{{- raise_exception('System message cannot contain videos.') }}
{%- endif %}
{%- if do_vision_count %}
{%- set video_count.value = video_count.value + 1 %}
{%- endif %}
{%- if add_vision_id %}
{{- 'Video ' ~ video_count.value ~ ': ' }}
{%- endif %}
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
{%- elif 'text' in item %}
{{- item.text }}
{%- else %}
{{- raise_exception('Unexpected item type in content.') }}
{%- endif %}
{%- else %}
{{- item | string }}
{%- endif %}
{%- endfor %}
{%- elif content is none or content is undefined %}
{{- '' }}
{%- else %}
{{- raise_exception('Unexpected content type.') }}
{%- endif %}
{%- endmacro %}
{%- if not messages %}
{{- raise_exception('No messages provided.') }}
{%- endif %}
{%- set head = namespace(count=0, seen_non_system=false) %}
{%- for message in messages %}
{%- set _is_sys = (message.role == 'system' or message.role == 'developer') %}
{%- if _is_sys and not head.seen_non_system %}
{%- set head.count = head.count + 1 %}
{%- else %}
{%- set head.seen_non_system = true %}
{%- endif %}
{%- endfor %}
{%- set sys_state = namespace(content='') %}
{%- for message in messages[:head.count] %}
{%- set _part = render_content(message.content, false, true) | trim %}
{%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %}
{%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %}
{%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
{%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %}
{%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
{%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
{%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %}
{%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %}
{%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %}
{%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
{%- if _part %}
{%- if sys_state.content %}
{%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %}
{%- else %}
{%- set sys_state.content = _part %}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- set _sc = sys_state.content %}
{%- set _msgs = messages[head.count:] %}
{%- if _has_tools %}
{{- '<|im_start|>system\n' }}
{%- if reasoning_instructions %}
{{- reasoning_instructions + '\n\n' }}
{%- endif %}
{{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
{%- for tool in tools %}
{{- '\n' }}
{{- tool | tojson }}
{%- endfor %}
{{- '\n</tools>' }}
{%- if _tool_format == 'json' %}
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
{%- if ns_state.thinking %}
{{- '<think>\nBrief explanation of tool call\n</think>\n' }}
{%- endif %}
{{- '<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
{%- if ns_state.thinking %}
{{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
{%- endif %}
{{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside <tool_call></tool_call> XML tags.\n' }}
{%- if ns_state.thinking %}
{{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
{%- else %}
{{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
{%- endif %}
{{- '- The <tool_call> tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
{%- else %}
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
{%- if ns_state.thinking %}
{{- '<think>\nBrief explanation of tool call\n</think>\n' }}
{%- endif %}
{{- '<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
{%- if ns_state.thinking %}
{{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
{%- endif %}
{{- '- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags.\n' }}
{%- if ns_state.thinking %}
{{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
{%- else %}
{{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
{%- endif %}
{{- '- The <tool_call> and <function> tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
{%- endif %}
{%- if _sc %}
{{- '\n\n' + _sc }}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- else %}
{%- if _sc %}
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }}
{%- elif reasoning_instructions %}
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- set _last_idx = _msgs | length - 1 %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %}
{%- for message in _msgs[::-1] %}
{%- set index = (_msgs | length - 1) - loop.index0 %}
{%- if ns.multi_step_tool and message.role == 'user' %}
{%- set _rc = render_content(message.content, false) | trim %}
{%- if not (_rc.startswith('<tool_response>') and _rc.endswith('</tool_response>')) %}
{%- set ns.multi_step_tool = false %}
{%- set ns.last_query_index = index %}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if ns.multi_step_tool %}
{%- if _last_idx > 50 %}
{%- set ns.last_query_index = _last_idx %}
{%- else %}
{%- set ns.last_query_index = 0 %}
{%- endif %}
{%- endif %}
{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %}
{%- for message in _msgs %}
{%- set is_system = (message.role == "system" or message.role == "developer") %}
{%- set content = render_content(message.content, true, is_system) | trim %}
{%- if is_system or message.role == 'user' %}
{%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %}
{%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %}
{%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
{%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %}
{%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
{%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
{%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %}
{%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %}
{%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %}
{%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
{%- endif %}
{%- if is_system %}
{{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
{%- elif message.role == 'user' %}
{%- set ns2.consecutive_failures = 0 %}
{{- '<|im_start|>user\n' + content + '<|im_end|>\n' }}
{%- elif message.role == 'assistant' %}
{%- set reasoning_content = '' %}
{%- set _explicit_reasoning = '' %}
{%- if message.reasoning_content is defined and message.reasoning_content is not none %}
{%- if message.reasoning_content is string %}
{%- set _explicit_reasoning = message.reasoning_content %}
{%- else %}
{%- set _explicit_reasoning = message.reasoning_content | string %}
{%- endif %}
{%- elif message.thinking is defined and message.thinking is not none %}
{%- if message.thinking is string %}
{%- set _explicit_reasoning = message.thinking %}
{%- else %}
{%- set _explicit_reasoning = message.thinking | string %}
{%- endif %}
{%- elif message.reasoning is defined and message.reasoning is not none %}
{%- if message.reasoning is string %}
{%- set _explicit_reasoning = message.reasoning %}
{%- else %}
{%- set _explicit_reasoning = message.reasoning | string %}
{%- endif %}
{%- endif %}
{%- if _explicit_reasoning %}
{%- set _lead_end = '' %}
{%- if content.startswith('<think>') and '</think>' in content %}
{%- set _lead_end = '</think>' %}
{%- elif content.startswith('<thinking>') and '</thinking>' in content %}
{%- set _lead_end = '</thinking>' %}
{%- elif content.startswith('</think>') %}
{%- set _lead_end = '</think>' %}
{%- elif content.startswith('</thinking>') %}
{%- set _lead_end = '</thinking>' %}
{%- endif %}
{%- if _lead_end %}
{%- set content = content.split(_lead_end)[-1].lstrip('\n') %}
{%- endif %}
{%- set reasoning_content = _explicit_reasoning %}
{%- else %}
{%- set _think_end = '' %}
{%- if content.startswith('</think>') %}
{%- set _think_end = '</think>' %}
{%- elif content.startswith('</thinking>') %}
{%- set _think_end = '</thinking>' %}
{%- elif '\n</think>' in content %}
{%- set _think_end = '\n</think>' %}
{%- elif '\n</thinking>' in content %}
{%- set _think_end = '\n</thinking>' %}
{%- elif '\n</ think>' in content %}
{%- set _think_end = '\n</ think>' %}
{%- elif '\n</think >' in content %}
{%- set _think_end = '\n</think >' %}
{%- elif content.startswith('<think>') and '</think>' in content %}
{%- set _think_end = '</think>' %}
{%- elif content.startswith('<thinking>') and '</thinking>' in content %}
{%- set _think_end = '</thinking>' %}
{%- endif %}
{%- if _think_end %}
{%- if 'thinking' in _think_end %}
{%- set _think_start = '<thinking>' %}
{%- else %}
{%- set _think_start = '<think>' %}
{%- endif %}
{%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %}
{%- if _think_start in reasoning_content %}
{%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %}
{%- endif %}
{%- set content = content.split(_think_end)[-1].lstrip('\n') %}
{%- endif %}
{%- endif %}
{%- set reasoning_content = reasoning_content | trim %}
{%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %}
{{- '<|im_start|>assistant\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
{%- else %}
{{- '<|im_start|>assistant\n' + content }}
{%- endif %}
{%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined and tool_call.function is not none %}
{%- set tc = tool_call.function %}
{%- else %}
{%- set tc = tool_call %}
{%- endif %}
{%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %}
{%- if _tool_format == 'json' %}
{%- if loop.first %}
{%- if content | trim %}
{{- '\n\n' }}
{%- endif %}
{%- else %}
{{- '\n' }}
{%- endif %}
{%- set _args = '{}' %}
{%- if tc.arguments is defined and tc.arguments is not none %}
{%- if tc.arguments is mapping %}
{%- set _args = tc.arguments | tojson %}
{%- elif tc.arguments is string %}
{%- if tc.arguments %}
{%- set _args = tc.arguments %}
{%- endif %}
{%- else %}
{%- set _args = tc.arguments | tojson %}
{%- endif %}
{%- endif %}
{{- '<tool_call>\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n</tool_call>' }}
{%- else %}
{%- if loop.first %}
{%- if content | trim %}
{{- '\n\n<tool_call>\n<function=' + tc_name + '>\n' }}
{%- else %}
{{- '<tool_call>\n<function=' + tc_name + '>\n' }}
{%- endif %}
{%- else %}
{{- '\n<tool_call>\n<function=' + tc_name + '>\n' }}
{%- endif %}
{%- if tc.arguments is defined and tc.arguments is not none %}
{%- if tc.arguments is mapping %}
{%- for args_name, args_value in tc.arguments.items() %}
{{- '<parameter=' + args_name + '>\n' }}
{%- if args_value is string %}
{%- set _av = args_value %}
{%- else %}
{%- set _av = args_value | tojson %}
{%- endif %}
{%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %}
{{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }}
{%- else %}
{{- _av }}
{%- endif %}
{{- '\n</parameter>\n' }}
{%- endfor %}
{%- else %}
{%- if tc.arguments is string %}
{%- set _raw_args = tc.arguments %}
{%- else %}
{%- set _raw_args = tc.arguments | tojson %}
{%- endif %}
{%- if _raw_args %}
{%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %}
{{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }}
{%- else %}
{{- _raw_args }}
{%- endif %}
{%- endif %}
{%- endif %}
{%- endif %}
{{- '</function>\n</tool_call>' }}
{%- endif %}
{%- endfor %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message.role == 'tool' %}
{%- set _content_lower = content | lower %}
{%- set _content_head = _content_lower[:120] %}
{%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %}
{%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %}
{%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %}
{%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %}
{%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %}
{%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %}
{%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %}
{%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %}
{%- else %}
{%- set ns2.consecutive_failures = 0 %}
{%- endif %}
{%- if ns2.prev_role != 'tool' %}
{{- '<|im_start|>user' }}
{%- endif %}
{%- set _is_json_payload = (_tool_format == 'json' and content | trim | length > 0 and (content | trim)[:1] in ('{', '[')) %}
{%- if not _is_json_payload and max_tool_response_chars > 0 and content | length > max_tool_response_chars %}
{%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %}
{%- endif %}
{{- '\n<tool_response>\n' + content }}
{%- if ns2.consecutive_failures >= 2 %}
{{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }}
{%- elif ns2.consecutive_failures == 1 %}
{{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }}
{%- endif %}
{{- '\n</tool_response>' }}
{%- if loop.last %}
{{- '<|im_end|>\n' }}
{%- else %}
{%- set _next_role = _msgs[loop.index0 + 1].role %}
{%- if _next_role != 'tool' %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- else %}
{{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }}
{%- endif %}
{%- set ns2.prev_role = message.role %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- if not ns_state.thinking %}
{{- '<think>\n\n</think>\n\n' }}
{%- else %}
{{- '<think>\n' }}
{%- endif %}
{%- endif %}
@@ -0,0 +1,105 @@
# qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
# (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
# base) as a radiance container for frank-primary: EVERY routed expert -- gate_up AND down --
# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
# ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
# gate_up w4): the served kernel is w4nl64a8h for the whole expert set, bf16 experts beside it.
# The rest is qwen4exp-w5dn64-i8-hc8m.recipe verbatim: int8 trunk linears, block-fp8 hc mixers
# and MTP fc, the fixed-scale n-gram table, the 2-bit draft head.
#
# Why down is the calibrated half: the calibration tap rides the down projection, so its GPTQ
# rounding propagates against real down-input Hessians; gate_up shares the same w4 table
# uncalibrated. The published w5dn64 comparison (mean KL 0.0789 -> 0.0717, top-1 92.85% ->
# 93.34%, decode +4%, prefill -19%) is the 5-bit upper bound; this tier trades part of that
# quality back for prefill.
#
# CALIB=DIR rad-convert <Swift1.5-Qwen3.8-Flash-Next checkpoint> \
# --recipe scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe -o qwen4exp-swift15-w4dn8-i8-hc8m.rad
#
# DIR is the calibration directory the bf16 container wrote (RADIANCE_CALIB_DIR, 10M tokens of
# traffic-shaped corpus). Everything not named here is served as the checkpoint holds it (bf16:
# embeddings, the vision tower, the QSA indexer); the first rule that matches a weight decides it.
# The bf16 container is NOT passed with --reuse: the peak-disk arithmetic does not fit (checkpoint +
# container + container + output > purge-freed space), and --reuse only saves a re-read.
# The protected experts, cast to bf16 ahead of both expert rules. THE NAMES BELOW ARE THE BASE
# CHECKPOINT'S (from its 1M-token recording, qwen4exp-w5dn64-i8-hc8m.recipe) -- a warm start only.
# The Swift 1.5 finetune's routing distribution is not assumed equal: the derivation step rewrites
# this section from Swift's OWN calibration recording before this recipe is ever run, by the base's
# law -- an expert whose share of its layer's down-input energy is at least 3% and at least three
# times its share of the routing, an even count on each rank's half of the layer (the four-bit GEMM
# takes the rest as two equal tables by parity) by adding that half's next most energetic expert.
# PROTECTED-BEGIN (rewritten from Swift's OWN grams by derive-protected-experts.py)
blk.1.ffn_*_exps.271.weight cast dtype=bf16
blk.1.ffn_*_exps.355.weight cast dtype=bf16
blk.2.ffn_*_exps.279.weight cast dtype=bf16
blk.2.ffn_*_exps.413.weight cast dtype=bf16
blk.34.ffn_*_exps.146.weight cast dtype=bf16
blk.34.ffn_*_exps.153.weight cast dtype=bf16
blk.34.ffn_*_exps.407.weight cast dtype=bf16
blk.34.ffn_*_exps.453.weight cast dtype=bf16
blk.37.ffn_*_exps.288.weight cast dtype=bf16
blk.37.ffn_*_exps.377.weight cast dtype=bf16
blk.38.ffn_*_exps.406.weight cast dtype=bf16
blk.38.ffn_*_exps.420.weight cast dtype=bf16
blk.42.ffn_*_exps.37.weight cast dtype=bf16
blk.42.ffn_*_exps.90.weight cast dtype=bf16
blk.42.ffn_*_exps.302.weight cast dtype=bf16
blk.42.ffn_*_exps.388.weight cast dtype=bf16
blk.44.ffn_*_exps.315.weight cast dtype=bf16
blk.44.ffn_*_exps.350.weight cast dtype=bf16
blk.46.ffn_*_exps.290.weight cast dtype=bf16
blk.46.ffn_*_exps.419.weight cast dtype=bf16
blk.47.ffn_*_exps.119.weight cast dtype=bf16
blk.47.ffn_*_exps.143.weight cast dtype=bf16
blk.47.ffn_*_exps.399.weight cast dtype=bf16
blk.47.ffn_*_exps.445.weight cast dtype=bf16
# PROTECTED-END
# (derive-protected-experts.py rewrote this section from Swift's own grams)
# Every other routed expert, trunk and MTP layer alike: u4 codes into the w4nl table, rotated by
# the 128-wide Hadamard, an E4M3 scale a group of 64 under the fixed 2^-13 searched for least
# error as stored, chosen by GPTQ against $CALIB and swept three times by coordinate descent.
# The calibration Grams are down-input Hessians, so down is the calibrated half of the table;
# gate_up has no recording of its own and quantises uncalibrated (said on stderr, as the base
# recipe says). The MTP layer has no recording either. Expert mass: gate_up 150.00 GiB -> ~37.6,
# down 75.00 GiB -> ~18.8: ~56 GiB of experts, ~28 GiB a rank at TP=2 -- well inside the pool.
blk.*.ffn_*_exps.*.weight gptq table=w4nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
# Every connection's two mixing matrices -- the 96 trunk connections, the final mixer and the MTP
# head's three -- as E4M3 rows with an f32 scale a group of each row. A 128x128 block cannot cover
# them: lowrank is 320, 2.5 blocks. Down is [320, 10240] and takes a scale every 128 columns; up is
# [10240, 320] and takes four scales a row, 80 columns each.
*_hc_down.weight rtn codes=fp8_e4m3 group=128 scale=f32
*_hc_up.weight rtn codes=fp8_e4m3 group=80 scale=f32
# The MTP head's fc_hidden and fc_embedding, the same rows. They only ever change the drafts.
mtp.fc_hidden.weight rtn codes=fp8_e4m3 group=128 scale=f32
mtp.fc_embedding.weight rtn codes=fp8_e4m3 group=128 scale=f32
# The trunk's block linears as int8, a scale per 128 columns of a row searched for least error.
# The QSA indexer's projection is not one of them: its top-k is a hard choice between blocks, and
# keeping it as the checkpoint holds it costs 20 MB.
blk.*.attn_qg.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.attn_k.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.attn_v.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.attn_output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ssm_inz.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ssm_out.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ffn_gate_up_shexp.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ffn_down_shexp.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
# The lm_head, int8 with a bf16 scale a row per 128 columns (logits_gemm_i8): at E4M3 it costs
# ~0.6 points of top-1 on its own.
output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
# The n-gram table at one fixed E4M3 scale; qwen4exp-w4.recipe says where the value comes from.
# The value sits on a three-octave plateau (2^-13..2^-10, rel_l2 0.02662) with 1.7x headroom over
# the base table's absmax; the prep phase verifies Swift's PLE absmax lands inside the plateau
# before this rule is trusted.
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
# The MTP head's 2-bit draft head. INERT for this checkpoint: the HF weight map has no
# mtp.draft_head.weight (same as the base), kept for shape parity with the family.
mtp.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env bash
# run-swift15-bf16.sh — RUNS ON DANKSERVER. Boot the no-recipe bf16 Swift1.5
# container for the two offline campaign phases. The engine owns the GPUs, so
# the production unit must be stopped first (this script does it and restores
# on exit, autoresearch pattern: stop, never mask — relinked unit).
#
# kld --kld-record the reference logits from the bf16 model
# (runs the corpus and EXITS; engine writes DIR and leaves)
# calib serve with the calibration tap; a separate driver
# (scripts/calib-push.py) pushes the corpus; the engine writes the
# Hessians at RADIANCE_CALIB_TOKENS and the tap switches itself off.
# Stop this script after calib-push prints TAP_DONE.
#
# Shape = the vendor's bf16 calibration lines (GUIDE.md §4.3:528 + MOE-W4.md
# "calibrate the checkpoint as it ships"): expert_tiered + weights-disk-tier,
# no speculation, deterministic. Large prefill steps because a disk-tier step
# costs the same at 1 token as at 32K. The container declared --max-ctx 4096;
# --max-model-len 16384 exceeds it on purpose (the KLD prompts run to 12,007
# tokens, calib to 8,242) — the boot ledger arbitrates the QSA buffers; if
# declare refuses, re-convert with a wider --max-ctx.
set -uo pipefail
MODE="${1:?usage: run-swift15-bf16.sh kld|calib}"
PIN_1013=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
PIN_120=sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa
RAD=/home/dank/models/rad/qwen4exp-swift15-bf16.rad
C=swift15ref
LOG=/home/dank/logs/swift15_bf16.log
KLD_REF_DIR=/home/dank/corpora/kld-ref
CALIB_DIR=/home/dank/corpora/calib
KLD_CORPUS=/home/dank/corpora/out/kld.jsonl
restore() { sudo systemctl start frank-primary >/dev/null 2>&1 || true; }
trap restore EXIT
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
podman rm -f "$C" >/dev/null 2>&1 || true
ARGS=(--model /models/rad/qwen4exp-swift15-bf16.rad
--tp 2 --placement expert_tiered --weights-disk-tier
--host-pool-mib 12288 --deterministic --gpu-headroom-mib 768
--num-speculative-tokens 0
--kv-cache-dtype fp8 --max-model-len 16384
--ngram-placement ram
--override-chat-template /models/tpl.jinja)
case "$MODE" in
kld)
# 1.0.13: the engine that will serve the candidate — the reference is
# recorded on the same engine that scores it. The weights slab DERIVES as
# card - headroom - kv, and the disk tier refuses under 9.57 GiB (4.21 of
# layer buffers + 5.36 fixed), so kv is explicit SMALL: recording is
# prefill-only, self-batched at MNBT 32768 = two 16k prompts at a time;
# 4 slots x 16k fp8 (29 KiB/pos, 1.7 GiB) + headroom fits 3072.
IMAGE="stilldeadcode/radiance@$PIN_1013"
mkdir -p "$KLD_REF_DIR"
ARGS+=(--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
--vram-kv-mib 3072
--max-num-seqs 4 --max-num-batched-tokens 32768)
;;
calib)
# 1.2.0: the calibration tap (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS)
# lives in the 1.2.0 qwen4exp plugin; 1.0.13 has only the DIR half with an
# unparameterisable threshold. Grams are bf16 activations — engine-invariant.
# Same KV ceiling as kld: the disk-tier weights floor (9.57 GiB) plus
# ~18 GiB of fixed per-rank claims cap the pool near 3.5 GiB on a 32 GiB
# card — pool minus the 2.17 GiB fixed holds ~237k fp8 tokens a rank, so
# the vendor's big-step calibration shape is affordable: MOE-W4.md:244
# measures 640 tok/s at a 32k step against 930 at 55k, and the doc's own
# run drives MNBT 65536. 8 slots x 16k = 131k resident tokens fits the
# pool; the step batches 65k. KLD's observed rate confirmed step size,
# not slot count, is the throughput lever.
IMAGE="stilldeadcode/radiance@$PIN_120"
mkdir -p "$CALIB_DIR"
ARGS+=(--port 7770 --vram-kv-mib 3072
--max-num-seqs 8 --max-num-batched-tokens 65536
--prefix-cache-dir /data/calib-kvcache)
;;
probe)
IMAGE="stilldeadcode/radiance@$PIN_1013"
ARGS+=(--port 7770 --max-num-seqs 8 --max-num-batched-tokens 4096)
;;
*) echo "unknown mode: $MODE" >&2; exit 2 ;;
esac
# The prod template copy, readable from the container: mount the models tree
# ro plus the state dir rw (KLD record + prefix tier land there).
cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
/home/dank/models/tpl.jinja 2>/dev/null || true
KLD_MOUNT=()
if [ "$MODE" = kld ]; then
# Nested under the /data mount, so these must come after it (line order
# in podman run below): the corpus read-only, the record dir writable.
KLD_MOUNT=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
-v "$KLD_REF_DIR:/data/kld-ref")
fi
EXTRA_ENV=()
if [ "$MODE" = calib ]; then
EXTRA_ENV=(-v "$CALIB_DIR:/data/calib"
-e RADIANCE_CALIB_DIR=/data/calib
-e RADIANCE_CALIB_TOKENS=10000000)
fi
podman run --rm --name "$C" \
--network host --init \
--security-opt seccomp=unconfined \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v /home/dank/models:/models:ro \
-v /home/dank/radiance-state:/data \
"${KLD_MOUNT[@]}" \
"${EXTRA_ENV[@]}" \
"$IMAGE" "${ARGS[@]}" 2>&1 | tee "$LOG"
rc=${PIPESTATUS[0]}
echo "swift15-bf16 $MODE exited rc=$rc; log: $LOG"
exit "$rc"
+185 -92
View File
@@ -1,107 +1,200 @@
#!/usr/bin/env bash
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on
# Dankclaw's Strix Halo iGPU (gfx1151), using stock llama.cpp
# (mainline, the unsloth b11030 prebuilt).
# serve-frank-large.sh — frank-large endpoint: halogen-flash-server 0.16.2
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
# digest-pinned in frank-large.env) serving the Quat3rnion abliterated
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint on
# Dankclaw's Strix Halo iGPU, host :7770. 2026-10-06 cutover from the
# halogen 27B-dense shape — revert the swap commit's trio to roll back
# (the 27B payload stays on disk).
#
# Runtime: /home/dank/GitPackages/unsloth-llama-b11030/llama-server
# (mainline, ROCm gfx1151 prebuilt). The build supports the qwen3_5
# architecture and the DFlash2 drafter (llama_model_qwen35,
# llama_model_dflash, common_speculative_impl_draft_dflash).
# The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
# pattern): the image starts engine + OpenAI frontend in one container.
# The frontend listens on 8731 inside (HALOGEN_API_PORT default) and is
# published as host 7770; the engine's own token port (8730, no auth)
# stays unpublished on the container's loopback. HALOGEN_CHECKPOINT
# selects the file inside the /models mount; the n-gram lookup table
# (HALOGEN_NGRAM_TABLE, the card sets it explicitly though the documented
# fallback is "beside the checkpoint") and the vision tower
# (HALOGEN_VISION_TOWER) ride in the same directory.
#
# Shape: one stream, full 262144 context (the model's native window), all
# layers on the iGPU (--n-gpu-layers 999). The 27B dense model (31.46 GiB GGUF) + the
# DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fits the
# iGPU's GTT aperture (~114 GiB).
# Gates mirror the deployer EXACTLY (the 2026-10-01 crash-loop lesson —
# serve gate and deploy gate disagreeing = green deploy, red unit):
# checkpoint/n-gram/vision byte-exact, tokenizer directory present with
# every named file a FLAT REGULAR FILE (a HF cache snapshot's symlinks
# into blobs/ dangle inside the mount), GPU device nodes present.
#
# Spec method: DFlash2 (a block-diffusion drafter) at depth 4. The drafter
# has a sliding window of 2048 (small draft KV, ~40 MiB).
#
# KV: q8_0 quantized (--cache-type-k q8_0 --cache-type-v q8_0). The unsloth b11030
# build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/
# q5_1); q8_0 is the highest-precision 8-bit option and preserves the ~8 GiB footprint.
#
# Text-only (no mmproj) — matching the prior frank-large.
#
# The Qwen3.8 family template keeps prior thinking blocks across turns when
# preserve_thinking is set; pass --jinja and the kwarg so multi-turn agents
# do not lose the trail. The template gates reasoning_effort to {xhigh,
# medium, low} — the harness folds its levels onto that set in models.yml
# (reasoningEffortMap).
#
# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
# it; anything not set there falls through to the defaults below.
# Why containers instead of a bare binary: the payload is .hgn and only
# the flash-server line reads it; it ships as a container; the root unit
# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
# exactly like frank-primary's.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
source "$ENV_FILE"
set +a
fi
# --- defaults (used when the env file does not set them) ------------------
: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}"
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
: "${HOST:=0.0.0.0}"
: "${PORT:=7770}"
: "${CONTEXT_SIZE:=262144}"
: "${PARALLEL_SLOTS:=1}"
: "${SPEC_DRAFT_N_MAX:=4}"
: "${CACHE_RAM:=8192}"
TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/qwen-sharp-v22.4.0.jinja}"
SERVER_BIN="$RUNTIME_DIR/llama-server"
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
if [[ ! -x "$SERVER_BIN" ]]; then
echo "runtime not found at $SERVER_BIN" >&2
if [ ! -f "$ENV_FILE" ]; then
echo "frank-large: env file not found: $ENV_FILE" >&2
exit 1
fi
for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
if [[ ! -f "$f" ]]; then
echo "missing asset $f (run scripts/download-frank-large.sh)" >&2
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
: "${FRANK_LARGE_IMAGE:?FRANK_LARGE_IMAGE must name the digest-pinned image}"
: "${FRANK_LARGE_CONTAINER:?FRANK_LARGE_CONTAINER must name the container}"
: "${FRANK_LARGE_PUBLISH:?FRANK_LARGE_PUBLISH must be host:container}"
: "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
: "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
: "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
: "${FRANK_LARGE_NGRAM_TABLE:?FRANK_LARGE_NGRAM_TABLE must name the lookup table}"
: "${FRANK_LARGE_NGRAM_BYTES:?FRANK_LARGE_NGRAM_BYTES must pin the table size}"
: "${FRANK_LARGE_VISION_TOWER:?FRANK_LARGE_VISION_TOWER must name the tower}"
: "${FRANK_LARGE_VISION_BYTES:?FRANK_LARGE_VISION_BYTES must pin the tower size}"
: "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-dragoon)}"
: "${HALOGEN_CTX:?HALOGEN_CTX must be set (per-slot window)}"
: "${HALOGEN_KV_POOL_POSITIONS:?HALOGEN_KV_POOL_POSITIONS must be set}"
: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set}"
: "${HALOGEN_SPEC_ADAPT:?HALOGEN_SPEC_ADAPT must be set (0 = head always on)}"
# Capture podman's output BEFORE grepping: under `set -o pipefail` the
# `podman images | grep -q` pattern races (grep exits on first match,
# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
# 2026-09-30 deploy refusal).
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
echo "frank-large: image $FRANK_LARGE_IMAGE not in podman storage;" >&2
echo " pull it: podman pull ${FRANK_LARGE_IMAGE%@*}" >&2
exit 1
fi
# Size-exact gates on all three payload files: a truncated curl exits 0 on
# some failures and a partial .hgn loads far enough to look alive, so the
# byte counts from the Hub tree API are the contract (fleet fetch law).
ck() {
# ck <env-name> <path> <expected-bytes>
if [ ! -f "$2" ]; then
echo "frank-large: $1 $2 missing;" >&2
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
exit 1
fi
local sz
sz="$(stat -c %s "$2")"
if [ "$sz" != "$3" ]; then
echo "frank-large: $1 is $sz bytes, expected $3 (partial download?)" >&2
exit 1
fi
}
ck checkpoint "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT" "$FRANK_LARGE_CHECKPOINT_BYTES"
ck n-gram-table "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_NGRAM_TABLE" "$FRANK_LARGE_NGRAM_BYTES"
ck vision-tower "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_VISION_TOWER" "$FRANK_LARGE_VISION_BYTES"
# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
# snapshot symlinks each file into a sibling blobs/, and those symlinks
# dangle inside the container mount — so the directory must be real and
# every named file must be a regular file, not a link.
TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
exit 1
fi
for TF in $FRANK_LARGE_TOKENIZER_FILES; do
if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
exit 1
fi
done
# --- env (the unsloth build's ROCm setup) --------------------------------
export HIP_VISIBLE_DEVICES=0
export HSA_OVERRIDE_GFX_VERSION=11.5.1
export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/llvm/lib:${LD_LIBRARY_PATH:-}"
# GPU access is mandatory but the node paths are variables so the seam can
# drive the gate on a box without an AMD GPU.
DEV_KFD="${FRANK_LARGE_DEV_KFD:-/dev/kfd}"
DEV_DRI="${FRANK_LARGE_DEV_DRI:-/dev/dri}"
if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
exit 1
fi
# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
# The engine refuses to pin when MemAvailable is below the checkpoint plus
# a 16 GiB floor: pinned pages cannot be swapped, and under the floor the
# n-gram lookup pages in from disk on every long prompt. A RAM-short host
# (a co-tenant spike, a sibling engine dying) crash-loops the unit — each
# boot dies in ~5 s at the pin gate — until the RAM clears. The 2026-10-06
# cutover hit this: 8 boots, MemAvailable 74.77 GiB vs the 78.11 GiB needed,
# self-resolved in ~2 min when the dying 27B engine freed RAM. Check here and
# WAIT for the RAM to clear, so a transient dip boots once instead of
# crash-looping; if it never clears, refuse loudly.
MEMINFO="${FRANK_LARGE_MEMINFO:-/proc/meminfo}"
PIN_KB=$(( FRANK_LARGE_CHECKPOINT_BYTES / 1024 ))
NEED_KB=$(( PIN_KB + 16 * 1048576 ))
ram_avail_kb() { awk '/^MemAvailable:/{print $2; exit}' "$MEMINFO"; }
MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
WAIT_S="${FRANK_LARGE_RAM_WAIT:-600}"
GIB() { awk -v kb="$1" 'BEGIN{printf "%.2f", kb/1048576}'; }
echo "frank-large: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
deadline=$(( $(date +%s) + WAIT_S ))
while [ "$(date +%s)" -lt "$deadline" ]; do
sleep 5
MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -ge "$NEED_KB" ]; then
break
fi
done
if [ -z "$MEMAVAIL_KB" ] || [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
echo "frank-large: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
exit 1
fi
echo "frank-large: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
fi
for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
podman rm -f "$CTR" >/dev/null 2>&1 || true
done
# --- the serve command ---------------------------------------------------
ARGS=(
--host "$HOST"
--port "$PORT"
--model "$MODEL_FILE"
--alias frank-large
--ctx-size "$CONTEXT_SIZE"
--parallel "$PARALLEL_SLOTS"
--n-gpu-layers 999
--jinja
--metrics
--chat-template-file "$TEMPLATE_FILE"
--chat-template-kwargs '{"preserve_thinking":true}'
--cache-type-k q8_0
--cache-type-v q8_0
--cache-ram "$CACHE_RAM"
# DFlash2 spec decode (a block-diffusion drafter)
--spec-type draft-dflash
--model-draft "$DRAFTER_FILE"
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
--spec-draft-n-min 0
--spec-draft-device ROCm0
# Greedy sampling (the DFlash2 drafter needs greedy verification)
--temp 0
--top-p 1.0
--top-k 0
--min-p 0.0
# Reasoning (thoughts land in reasoning_content)
--reasoning auto
--reasoning-format deepseek
--reasoning-budget -1
)
LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
mkdir -p "$(dirname "$LOG")"
# --ipc=host for the udev-granted supplementary group, seccomp unconfined
# for the ROCm ioctl surface, and unbounded memlock (the unit's
# LimitMEMLOCK=infinity is inherited, so runc never has to RAISE it). NO
# --ulimit nofile: Dankclaw's hard limit is 524288, rootless runc cannot
# raise a hard limit and dies with "error setting rlimit type 7:
# operation not permitted" (measured 2026-10-05 cutover); the model card
# and both sibling serve scripts pass no nofile either. No comment lines
# inside the backslash chain — a mid-chain # terminates the continuation
# (the 2026-10-05 exit-127 crash loop on the sibling serve script).
podman run --name "$FRANK_LARGE_CONTAINER" \
--publish "$FRANK_LARGE_PUBLISH" \
--ipc=host \
--ulimit memlock=-1:-1 \
--device "$DEV_KFD" --device "$DEV_DRI" \
--group-add keep-groups \
--security-opt seccomp=unconfined \
-e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
-e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
-e HALOGEN_NGRAM_TABLE="/models/$FRANK_LARGE_NGRAM_TABLE" \
-e HALOGEN_VISION_TOWER="/models/$FRANK_LARGE_VISION_TOWER" \
-e HALOGEN_TOKENIZER=/tokenizer \
-e HALOGEN_CTX="$HALOGEN_CTX" \
-e HALOGEN_KV_POOL_POSITIONS="$HALOGEN_KV_POOL_POSITIONS" \
-e HALOGEN_KV_SLOTS="$HALOGEN_KV_SLOTS" \
-e HALOGEN_HOST_RESERVE_GIB="${HALOGEN_HOST_RESERVE_GIB:-20}" \
-e HALOGEN_SPEC_ADAPT="$HALOGEN_SPEC_ADAPT" \
-e HALOGEN_MAX_TOK="${HALOGEN_MAX_TOK:-8192}" \
-e HALOGEN_MAX_TOKENS_CAP="${HALOGEN_MAX_TOKENS_CAP:-65536}" \
-e HALOGEN_QUEUE_TIMEOUT="${HALOGEN_QUEUE_TIMEOUT:-7200}" \
-e HALOGEN_TEMPERATURE="${HALOGEN_TEMPERATURE:-1.0}" \
-e HALOGEN_TOP_P="${HALOGEN_TOP_P:-0.95}" \
-e HALOGEN_TOP_K="${HALOGEN_TOP_K:-20}" \
${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
-v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
-v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
"$FRANK_LARGE_IMAGE" \
> "$LOG" 2>&1
exec "$SERVER_BIN" "${ARGS[@]}"
# FOREGROUND `podman run`: the engine's stdout/stderr are this script's
# output, so the boot ledger (kv pool, prompt cache, MTP head, any clamp)
# lands in $LOG as documented. podman run returns the container's exit
# code, so a crash is a nonzero ExecStart and the unit's Restart=always
# stays real; systemd stop goes through the unit's ExecStop and the run
# client forwards SIGTERM to the engine, which drains on it.
+172 -108
View File
@@ -1,37 +1,33 @@
#!/usr/bin/env bash
# serve-frank-primary.sh — frank-primary endpoint (R9V radiance-vLLM fork,
# unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 drafter, 2x R9700, dankserver
# :7770; 2026-09-27 cutover from the uncensored IQ4_XS payload, whose state
# dir stays on disk as the rollback target).
# serve-frank-primary.sh — frank-primary endpoint: radiance (StillDeadcode,
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
# NATIVE block-FP8 precision from a rad-convert container
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
# one card; git tag frank-primary-freetoken-pr587-20261005 is the
# pre-cutover state).
#
# The engine is the upstream R9V launcher (scripts/launch.sh in the fork
# checkout), NOT a hand-frozen podman argv: launch.sh owns the resolved
# docker/podman args (binds, overlays, device passthrough, publish lines)
# and the fork's qualification was measured through it. This script's job:
#
# 1. layer the on-host R9V setup state (the 104-key tuned shape the
# profile setup qualified) under this repo's frank-primary.env
# overrides (bind/port/alias/container);
# 2. gate on the pinned runtime image (by its OCI manifest digest, the
# podman RepoDigest), the exact-size PLE payload, and the GGUF shards;
# 3. exec the launcher, then wait on the container: launch.sh --detach
# exits as soon as the container is created, so without the wait the
# unit would be "green" while the engine crashes — the wait is what
# keeps systemd's Restart=on-failure self-healing real.
# The run line below IS the engine argv: the image's entrypoint is the
# engine and its tools sit beside it on PATH. Unlike every predecessor
# there is no per-boot VRAM planner to second-guess: the boot ledger in the
# log states every budget before anything is allocated (weights vs KV
# split, the expert host pool's coverage of the worst case, the PLE pin,
# the P2P answer per pair) — read it after every resize. First-boot shape:
# 2 decode slots (Frank's call), 40 GiB/rank pinned expert pool, fp8 KV,
# MTP depth 3, prefix cache host 4 GiB + disk 64 GiB, PLE pinned in RAM.
#
# The root unit (scripts/frank-primary.service, User=dank,
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert UVA
# host copy and the PLE pinned residency (54.4 GB on the q8 payload). A
# plain shell session also works (proven 2026-09-27: podman rootless
# inherits usable limits here), but the unit is the fleet-parity
# boot-persistent shape.
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 40 GiB expert pools +
# the 47.7 GiB PLE table + 4 GiB prefix host tier — ~132 GiB of pinned
# pages, which is ALSO why this script refuses to start when ttm's
# pages_limit/pages_max cap is under FRANK_PRIMARY_TTM_MIN_MIB (see below).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
if [ ! -f "$ENV_FILE" ]; then
echo "serve-frank-primary: env file missing: $ENV_FILE" >&2
echo "ERROR: frank-primary env file absent: $ENV_FILE" >&2
exit 1
fi
set -a
@@ -39,104 +35,172 @@ set -a
source "$ENV_FILE"
set +a
: "${R9V_HOME:?R9V_HOME must name the fork checkout}"
: "${R9V_STATE_FILE:?R9V_STATE_FILE must name the setup state}"
: "${R9V_HOST_BIND:?R9V_HOST_BIND must be set}"
: "${R9V_HOST_PORT:?R9V_HOST_PORT must be set}"
: "${R9V_SERVED_MODEL_NAME:?R9V_SERVED_MODEL_NAME must be set}"
: "${R9V_CONTAINER_NAME:?R9V_CONTAINER_NAME must be set}"
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the radiance image}"
: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the .rad dir}"
: "${FRANK_PRIMARY_MODEL_FILE:?FRANK_PRIMARY_MODEL_FILE must name the .rad file}"
: "${FRANK_PRIMARY_STATE_DIR:?FRANK_PRIMARY_STATE_DIR must name the /data state dir}"
: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
: "${FRANK_PRIMARY_PORT:?FRANK_PRIMARY_PORT must be the port}"
: "${FRANK_PRIMARY_TEMPLATE:?FRANK_PRIMARY_TEMPLATE must name the chat template}"
: "${FRANK_PRIMARY_TP:?FRANK_PRIMARY_TP must be set}"
: "${FRANK_PRIMARY_TP_WIRE:?FRANK_PRIMARY_TP_WIRE must be set}"
: "${FRANK_PRIMARY_MAX_SEQS:?FRANK_PRIMARY_MAX_SEQS must be set}"
: "${FRANK_PRIMARY_MAX_MODEL_LEN:?FRANK_PRIMARY_MAX_MODEL_LEN must be set}"
: "${FRANK_PRIMARY_PLACEMENT:?FRANK_PRIMARY_PLACEMENT must be set}"
: "${FRANK_PRIMARY_HOST_POOL_MIB:?FRANK_PRIMARY_HOST_POOL_MIB must be set}"
: "${FRANK_PRIMARY_GPU_HEADROOM_MIB:?FRANK_PRIMARY_GPU_HEADROOM_MIB must be set}"
: "${FRANK_PRIMARY_VRAM_WEIGHTS_MIB:?FRANK_PRIMARY_VRAM_WEIGHTS_MIB must be set}"
: "${FRANK_PRIMARY_VRAM_KV_MIB:?FRANK_PRIMARY_VRAM_KV_MIB must be set}"
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set}"
: "${FRANK_PRIMARY_SPEC_TOKENS:?FRANK_PRIMARY_SPEC_TOKENS must be set}"
: "${FRANK_PRIMARY_MAX_BATCHED_TOKENS:?FRANK_PRIMARY_MAX_BATCHED_TOKENS must be set}"
: "${FRANK_PRIMARY_PREFIX_HOST_MIB:?FRANK_PRIMARY_PREFIX_HOST_MIB must be set}"
: "${FRANK_PRIMARY_PREFIX_DISK_MIB:?FRANK_PRIMARY_PREFIX_DISK_MIB must be set}"
: "${FRANK_PRIMARY_NGRAM_PLACEMENT:?FRANK_PRIMARY_NGRAM_PLACEMENT must be set}"
: "${FRANK_PRIMARY_TTM_MIN_MIB:?FRANK_PRIMARY_TTM_MIN_MIB must be set}"
: "${FRANK_PRIMARY_RAD_MIN_MIB:?FRANK_PRIMARY_RAD_MIN_MIB must be set}"
LAUNCHER="$R9V_HOME/scripts/launch.sh"
if [ ! -x "$LAUNCHER" ]; then
echo "serve-frank-primary: R9V launcher missing: $LAUNCHER" >&2
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
# in storage only if THAT manifest was pulled — a retagged look-alike or a
# newer latest does not satisfy it. `podman image exists` takes a
# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
# storage (root unit, User=dank — the fleet's frank-primary shape).
if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
echo "ERROR: radiance image absent: $FRANK_PRIMARY_IMAGE" >&2
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
exit 1
fi
# The docker->podman shim is REQUIRED: the launcher and its doctor speak
# docker (OCI-manifest-digest image IDs, LogConfig fields) that podman's
# native CLI answers differently. The shim carries those laws.
SHIM_DIR="$R9V_HOME/.podman-shim"
if [ ! -x "$SHIM_DIR/docker" ]; then
echo "serve-frank-primary: docker->podman shim missing: $SHIM_DIR/docker" >&2
# The payload gate: the .rad container file, sized. A truncated convert
# would otherwise buy a guaranteed long-timeout boot instead of this fast
# failure. The staging dir / raw checkpoint are NOT this script's concern
# (scripts/build-frank-primary-rad.sh built the file; its rad-info output
# is the record).
RAD="$FRANK_PRIMARY_MODEL_DIR/$FRANK_PRIMARY_MODEL_FILE"
if [ ! -f "$RAD" ]; then
echo "ERROR: .rad container missing: $RAD" >&2
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
exit 1
fi
export PATH="$SHIM_DIR:$PATH"
# --- layer 1: the qualified setup state (production overrides never come
# from here; keys already defined in the env file are skipped) -------------
STATE_FILE="$R9V_STATE_FILE"
if [ ! -f "$STATE_FILE" ]; then
echo "serve-frank-primary: R9V setup state missing: $STATE_FILE" >&2
echo " run the profile setup first (tools/setup_profile.py setup --state-dir ...)" >&2
exit 1
fi
STATE_EXPORTS="$(python3 - "$STATE_FILE" <<'PY'
import json, shlex, sys, os
state = json.load(open(sys.argv[1]))
config = state.get("config") or {}
if state.get("ready") is not True:
sys.exit("R9V setup state is not ready; rerun setup before serving")
env_keys = {k for k in os.environ if k.startswith("R9V_")}
for key in sorted(config):
if key.startswith("R9V_") and key not in env_keys:
print(f"export {key}={shlex.quote(config[key])}")
PY
)"
eval "$STATE_EXPORTS"
: "${R9V_IMAGE:?state must carry R9V_IMAGE}"
: "${R9V_MODEL_DIR:?state must carry R9V_MODEL_DIR}"
: "${R9V_PLE_PATH:?state must carry R9V_PLE_PATH}"
# --- gates: never serve a half-finished payload or a missing image --------
# Runtime image by pinned OCI manifest digest (podman keys images by config
# digest but preserves the manifest digest in RepoDigests — the shim's ID law).
if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
echo "serve-frank-primary: pinned R9V image absent: $R9V_IMAGE" >&2
echo " load the pinned image bundle into podman rootless storage" >&2
RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
if [ "$RAD_MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_PRIMARY_RAD_MIN_MIB" >&2
echo "frank-primary: the convert was truncated — rebuild it" >&2
exit 1
fi
# PLE payload at its pinned byte count (stat -c %s, never wc -c), plus the
# six target GGUF shards (mradermacher-style split parts) and the mmproj
# the launcher mounts.
PLE_BYTES="${R9V_PLE_EXPECTED_BYTES:-}"
if [ -n "$PLE_BYTES" ]; then
ACTUAL="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
if [ "$ACTUAL" != "$PLE_BYTES" ]; then
echo "serve-frank-primary: PLE payload wrong size: $R9V_PLE_PATH" >&2
echo " expected $PLE_BYTES, found $ACTUAL" >&2
exit 1
fi
else
if [ ! -f "$R9V_PLE_PATH" ]; then
echo "serve-frank-primary: PLE payload missing: $R9V_PLE_PATH" >&2
exit 1
fi
if [ ! -f "$FRANK_PRIMARY_TEMPLATE" ]; then
echo "ERROR: chat template missing: $FRANK_PRIMARY_TEMPLATE" >&2
exit 1
fi
for rel in "$R9V_TARGET_REL" "${R9V_TARGET_SHARD2_REL:-}" "${R9V_TARGET_SHARD3_REL:-}" \
"${R9V_TARGET_SHARD4_REL:-}" "${R9V_TARGET_SHARD5_REL:-}" "${R9V_TARGET_SHARD6_REL:-}" \
"$R9V_MMPROJ_REL"; do
if [ -z "$rel" ]; then continue; fi
if [ ! -f "$R9V_MODEL_DIR/$rel" ]; then
echo "serve-frank-primary: payload file missing: $R9V_MODEL_DIR/$rel" >&2
exit 1
fi
done
mkdir -p "$FRANK_PRIMARY_STATE_DIR/kvcache/frank-primary"
# The engine's create is NOT recursive: with only the state dir mounted, its
# "cannot create /data/kvcache/frank-primary: No such file or directory"
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
# the full path on the host so the tier opens on first boot.
mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
# the cache must ride a host mount or it dies with the --rm container
# that wrote it. The dir is pre-created for the same reason as kvcache
# (the engine's create is not recursive).
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
exit 1
fi
# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
# pages_limit (the current kernel exposes NO pages_max; the cap function
# takes the min over the knobs that exist). Below FRANK_PRIMARY_TTM_MIN_MIB
# the load dies mid-pin with an allocator error instead of this named
# refusal — raise the knob first (runtime sysfs write; kernel cmdline
# fallback + one reboot only if the file is read-only), see the runbook.
# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
ttm_cap_mib() {
# min over the knobs THIS kernel exposes — the current kernel has
# pages_limit only (no pages_max); a knob that is absent is skipped,
# not a zero (a zero would refuse a healthy cap).
local v cur=""
for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
[ -r "$p" ] || continue
v="$(cat "$p" 2>/dev/null)" || continue
[ -n "$v" ] || continue
v=$(( v * 4 / 1024 ))
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
done
echo "${cur:-0}"
}
TTM_CAP="$(ttm_cap_mib)"
if [ "$TTM_CAP" -lt "$FRANK_PRIMARY_TTM_MIN_MIB" ]; then
echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_PRIMARY_TTM_MIN_MIB" >&2
echo "frank-primary: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_PRIMARY_TTM_MIN_MIB in 4 KiB pages" >&2
exit 1
fi
# A crashed/killed unit can leave the container name registered (systemd
# SIGKILLs conmon before cleanup runs); the launcher refuses an existing
# name forever after. Clear any orphan of OUR name first — idempotent.
docker rm -f "$R9V_CONTAINER_NAME" >/dev/null 2>&1 || true
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
# identities (the FreeToken-era frank-primary-qwen, the retired R9V
# container and the ad hoc experiment containers all hold this GPU); all
# idempotent.
for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
podman rm -f "$CTR" >/dev/null 2>&1 || true
done
# launch.sh detaches after preflight; from here on, this script's lifetime
# IS the container's: wait returns the container's exit code, so a crash is
# a nonzero ExecStart and Restart=on-failure fires. (systemd stop goes
# through ExecStop in the unit — docker wait alone cannot stop the child.)
"$LAUNCHER"
exec docker wait "$R9V_CONTAINER_NAME"
LOG=/home/dank/logs/frank_primary.log
mkdir -p "$(dirname "$LOG")"
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
# --network host (the server binds the host's addresses directly, no NAT),
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
# (Docker/podman's default profile refuses io_uring, which the n-gram
# table's reads go through, and the NUMA binding calls the ROCm runtime
# makes for host memory), memlock unbounded (the pinned ledger — legal
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
# The template rides a single-file mount; the .rad dir is ro; /data holds
# the prefix-cache disk tier.
podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
--network host \
--init \
--security-opt seccomp=unconfined \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-v "$FRANK_PRIMARY_STATE_DIR":/data \
-v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
-v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
"$FRANK_PRIMARY_IMAGE" \
--model "/models/$FRANK_PRIMARY_MODEL_FILE" \
--host :: --port "$FRANK_PRIMARY_PORT" \
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
--override-chat-template /templates/template.jinja \
--tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
--max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
--max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
--placement "$FRANK_PRIMARY_PLACEMENT" \
--host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
--gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
--vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
--vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
--prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
--prefix-cache-dir /data/kvcache/frank-primary \
--prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
--ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
--num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
--max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
> "$LOG" 2>&1
# From here on, this script's lifetime IS the container's: wait returns the
# container's exit code, so a crash is a nonzero ExecStart and the unit's
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
# podman wait alone cannot stop the container child.)
exec podman wait "$FRANK_PRIMARY_CONTAINER"
+132
View File
@@ -0,0 +1,132 @@
#!/usr/bin/env bash
# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
# frank-secondary on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
#
# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off:
# this GGUF carries no MTP tensors at all (verified from its tensor-name
# list), so there is no in-file head to draft with. Verbosity 4 ships so the
# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal.
#
# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4
# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers),
# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool
# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at
# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per
# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same
# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279
# at ubatch 2048 — the boot ledger, not arithmetic, is the referee.
# Vision comes from the publisher's own BF16 projector (the model is natively
# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
# the documented text-only opt-out.
#
# Sampling: llama.cpp takes it per request (the fleet harness sends the
# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
#
# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_SECONDARY_ENV_FILE:-$SCRIPT_DIR/frank-secondary.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
fi
# --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
: "${ALIAS:=frank-zealot}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7771}"
# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
# per conversation, inside the model's trained 262,144. The harness
# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
# with a 400, never truncated.
: "${CTX_SIZE:=401408}"
: "${PARALLEL:=2}"
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
# throughput-neutral at a fixed ubatch.
: "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=2048}"
# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that
# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry
# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408
# pool. Explicit either way so the contract is visible in the argv, and
# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's
# FP8 class; no true fp8 KV dtype exists in this build).
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
# there is no in-file head (the law: MTP is a property of the file, not of the
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
: "${SPEC_TYPE:=none}"
: "${SPEC_DRAFT_N_MAX:=3}"
: "${SPEC_DRAFT_FILE:=}"
: "${NGL:=99}"
# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
# the VRAM-fit forensics need. Levels above 4 flood the journal.
: "${VERBOSITY:=3}"
# Vision projector. No colon on purpose: an explicitly empty value means
# "text-only, on purpose" and must NOT be replaced by the default path.
: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
# Fail loudly rather than half-loading: a missing file is a download that
# has not finished, and Restart=always would turn it into a crash loop that
# reads like a runtime fault.
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
[[ -e "$path" ]] || { echo "frank-secondary: missing $path" >&2; exit 1; }
done
EXTRA_ARGS=()
if [[ -n "$MMPROJ_FILE" ]]; then
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-secondary: missing $MMPROJ_FILE" >&2; exit 1; }
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
fi
# Speculative flags move as a set: they reach the argv ONLY when a drafter
# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
# baseline must not carry half a spec pair.
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-secondary: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
fi
fi
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
# open after boot; make sure it exists before the server needs it.
nvidia-modprobe -u -c=0 2>/dev/null || true
# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
# the q8_0 KV contract in frank-secondary.env is a real path — without this
# flag or that build option the engine silently falls back and the
# contract becomes a lie.
ARGS=(
--model "$MODEL_FILE"
--alias "$ALIAS"
--host "$BIND_ADDR"
--port "$LISTEN_PORT"
--ctx-size "$CTX_SIZE"
--parallel "$PARALLEL"
--batch-size "$BATCH_SIZE"
--ubatch-size "$UBATCH_SIZE"
--n-gpu-layers "$NGL"
--flash-attn on
--cache-type-k "$CACHE_TYPE_K"
--cache-type-v "$CACHE_TYPE_V"
--log-verbosity "$VERBOSITY"
--jinja
--reasoning auto
--reasoning-format deepseek
)
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
-140
View File
@@ -1,140 +0,0 @@
#!/usr/bin/env bash
# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
#
# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
# weights plus the BF16 vision projector and q8_0 KV inside the card's
# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
# belongs to the halogen engine that serves frank-large.
#
# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
# passed via --model-draft with --spec-type draft-mtp: the draft context
# loads the assistant model. The same --spec-type value chains the
# draftless ngram-mod matcher, which attaches unconditionally; two
# context-backed drafters cannot share one process, so this chain is the
# only shape that carries both.
#
# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
# maximises the TOTAL pool instead, which would divide a smaller window
# across slots. The 4-slot layout is measured to fit the card (see the
# SIZING note in frank-small.env); more slots would fail the KV allocation.
#
# Vision + audio: --mmproj loads the BF16 projector, which also carries
# an audio encoder — input_audio content parts transcribe verbatim
# (verified 2026-09-17). The fleet's image path lives here, which is why
# halogen no longer sets HALOGEN_VISION_TOWER.
#
# Sampling: greedy by default (the fleet contract). Clients that send their own
# temperature get their own values, and the harness overrides per model from
# models.yml. --jinja is pinned even though this build defaults it on: the
# model's own template supplies reasoning_effort, preserve_thinking and the
# tool-call grammar, and a future build that flips that default must not
# silently take tools away.
#
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
fi
# --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}"
: "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
: "${DRAFT_FILE:=}"
: "${ALIAS:=frank-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7771}"
: "${CTX_SIZE:=}"
: "${PARALLEL:=1}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${CACHE_RAM:=8192}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
# Decode-shape knobs: the MTP drafter's per-round proposal length (build
# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
# busy-wait level (build default 50).
: "${SPEC_DRAFT_N_MAX:=3}"
: "${POLL_LEVEL:=50}"
: "${NUM_THREADS:=8}"
: "${THREAD_BATCH:=8}"
: "${NGL:=999}"
: "${FLASH_ATTN:=on}"
: "${SPEC_TYPE:=draft-mtp}"
: "${TEMPERATURE:=0.0}"
: "${REASONING_FORMAT:=deepseek}"
: "${REASONING_BUDGET:=-1}"
# Fail loudly rather than half-loading a model into VRAM: a missing file here
# is a download that has not finished, and the unit's restart policy would
# otherwise turn that into a crash loop that reads like a runtime fault.
for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
[[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
done
if [[ -n "${DRAFT_FILE:-}" ]]; then
[[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
fi
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
# open after boot; make sure it exists before the server needs it.
nvidia-modprobe -u -c=0 2>/dev/null || true
MMPROJ_ARGS=()
if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
fi
SPEC_ARGS=()
if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
fi
DRAFT_ARGS=()
if [[ -n "${DRAFT_FILE:-}" ]]; then
DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
fi
# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
# whatever the weights, projector, draft context and compute buffers leave
# on the card; the engine prints the chosen value at startup.
if [[ -n "${CTX_SIZE:-}" ]]; then
CTX_ARGS=(--ctx-size "$CTX_SIZE")
else
CTX_ARGS=(--fit on)
fi
exec "$LLAMA_BIN" \
--model "$MODEL_DIR/$MODEL_FILE" \
"${MMPROJ_ARGS[@]}" \
"${DRAFT_ARGS[@]}" \
--alias "$ALIAS" \
--host "$BIND_ADDR" \
--port "$LISTEN_PORT" \
--gpu-layers "$NGL" \
--flash-attn "$FLASH_ATTN" \
"${CTX_ARGS[@]}" \
--parallel "$PARALLEL" \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--threads "$NUM_THREADS" \
--threads-batch "$THREAD_BATCH" \
--poll "$POLL_LEVEL" \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--cache-ram "$CACHE_RAM" \
"${SPEC_ARGS[@]}" \
--jinja \
--metrics \
--temp "$TEMPERATURE" \
--reasoning-format "$REASONING_FORMAT" \
--reasoning-budget "$REASONING_BUDGET"
+556
View File
@@ -0,0 +1,556 @@
#!/usr/bin/env python3
"""stack-qwen38-official-fp8.py — reshape the official Qwen/Qwen3.8-Flash-Next-FP8
checkpoint into the form the shipped radiance qwen4exp plugin maps.
Run INSIDE the staging dir's tree (python3 + numpy on the host; data never
leaves the box). Two transforms, both VALUE-PRESERVING:
1. EXPERTS: the shipped plugin's stacked layout names ONE tensor per family
(`mlp.experts.gate_up_proj` [E, 2F, H], `mlp.experts.down_proj` [E, H, F])
plus a `<family>_scale_inv` plane at the same index (NO `.weight` in the
stacked name — scale_name() appends _scale_inv to the tensor name verbatim),
while the official FP8 repo ships per-expert unfused tensors
(`experts.<e>.{gate,up,down}_proj.weight` + scale). Stacking is pure byte
concatenation along dim 0 — gate rows
first inside each expert's gate_up slab, which is the order the block
wants (rad_block_moe_fp8.h: "a stacked gate_up is ALREADY FUSED in the
order this block wants"). No requantisation: bytes are moved, not decoded.
2. PLE TABLE: the plugin's fp8 branch wants the table's codes beside a
ONE-ELEMENT bf16 scale plane (`blk.%d.ple_ngram.scale`, view {1}); a
per-shard-blocked scale can never fit that view, and `rtn` re-encoding the
RAW scaleless codes would overflow (v / 1.99e-4 exceeds e4m3's 448 max).
So staging decodes the scaleless shards with the checkpoint's OWN bf16
scale into a bf16 table (exact: every e4m3 code value and the scale are
exactly representable in bf16 to within the code's own 2^-3 grid), and the
convert's rtn rule re-encodes at that same scale — the nearest-code
distance is 2^-3 while the bf16 rounding perturbs by 2^-9, so every
original code is restored exactly and the container gets the checkpoint's
own codes plus the one-element scale plane.
The convert then runs with NO recipe except:
blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=<s>
with <s> read from the checkpoint and written to ple_scale.txt.
Idempotent: re-runs skip when the staged index already carries the stacked
names, the merged bf16 shard_0, and split_ngram_parts == 1. --force redoes.
"""
import argparse
import json
import mmap
import multiprocessing as mp
import os
import re
import struct
import sys
import time
import numpy as np
F8, BF16 = "F8_E4M3", "BF16"
def log(msg):
print(msg, flush=True)
# ---------------------------------------------------------------- safetensors
def parse_header(path):
"""Return (header_len, {name: (dtype, shape, data_offsets)})."""
with open(path, "rb") as fh:
(hlen,) = struct.unpack("<Q", fh.read(8))
hdr = json.loads(fh.read(hlen))
return hlen, hdr
def write_safetensors(path, tensors, data_iter):
"""tensors: [(name, dtype, shape, nbytes)]; data_iter yields bytes per tensor."""
hdr = {}
off = 0
for name, dtype, shape, nbytes in tensors:
hdr[name] = {"dtype": dtype, "shape": list(shape),
"data_offsets": [off, off + nbytes]}
off += nbytes
blob = json.dumps(hdr, separators=(",", ":")).encode("utf-8")
pad = (-len(blob)) % 8 # keep the data start 8-byte aligned
blob += b" " * pad
with open(path, "wb") as fh:
fh.write(struct.pack("<Q", len(blob)))
fh.write(blob)
for _, _, _, nbytes in tensors:
fh.write(next(data_iter).tobytes())
fh.flush()
os.fsync(fh.fileno())
return 8 + len(blob) + sum(t[3] for t in tensors)
# ------------------------------------------------------------------- preflight
def build_meta(stage, index):
"""name -> (file, global_byte_offset, nbytes, dtype, shape) for every tensor."""
files = sorted(set(index["weight_map"].values()))
meta = {}
for f in files:
path = os.path.join(stage, f)
hlen, hdr = parse_header(path)
for name, e in hdr.items():
if name == "__metadata__":
continue
b0, b1 = e["data_offsets"]
meta[name] = (f, 8 + hlen + b0, b1 - b0, e["dtype"], e["shape"])
return meta
def e4m3_lut():
lut = np.zeros(256, np.float32)
for b in range(256):
sign = -1.0 if b & 0x80 else 1.0
e = (b >> 3) & 0xF
m = b & 0x7
if e == 15 and m == 7:
v = np.float32(np.nan) # OCP e4m3fn: only 0x7F/0xFF are NaN
elif e == 0:
v = sign * (m / 8.0) * 2.0 ** -6
else:
v = sign * (1.0 + m / 8.0) * 2.0 ** (e - 7)
lut[b] = v
assert lut[0x38] == 1.0 and lut[0x7E] == 448.0
assert np.isnan(lut[0x7F]) and lut[0x00] == 0.0
assert lut[0x08] == 2.0 ** -6 and lut[0x80] == 0.0
return lut
def f32_to_bf16_rne(v):
"""float32 array -> uint16 bf16 bits, round-to-nearest-even."""
q = np.ascontiguousarray(v, dtype=np.float32)
q[~np.isfinite(q)] = 0.0
bits = q.view(np.uint32)
r = bits + 0x7FFF + ((bits >> 16) & 1)
return (r >> 16).astype(np.uint16)
# ------------------------------------------------------------------- workers
def stack_prefix(job):
"""Byte-stack one layer's 512 experts into a 4-tensor safetensors file."""
(stage, pre, idx, n_e, shapes, meta) = job
(f2, h) = shapes["ffn2_h"] # 2*ffn, n_embd
(f,) = shapes["ffn"] # ffn
gs, ds = shapes["gs"], shapes["ds"] # scale shapes [5,20] and [20,5]
# byte widths: fp8 is 1 B/element, bf16 2 B; the fused gate_up scale is
# the gate and up scale planes concatenated (gate rows first)
g_bytes = (f2 // 2) * h
gs_bytes = gs[0] * gs[1] * 2 # ONE projection's scale plane
gus_bytes = 2 * gs_bytes # fused scale per expert
dn_bytes = h * f
ds_bytes = ds[0] * ds[1] * 2
gu = np.empty(n_e * f2 * h, np.uint8)
gus = np.empty(n_e * gus_bytes, np.uint8)
dn = np.empty(n_e * dn_bytes, np.uint8)
dns = np.empty(n_e * ds_bytes, np.uint8)
mmaps = {}
def view(name):
f_, off, ln, dt, shp = meta[name]
if f_ not in mmaps:
fh = open(os.path.join(stage, f_), "rb")
mmaps[f_] = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
return memoryview(mmaps[f_])[off:off + ln], (dt, shp)
def need(name, dt, shp):
mv, (got_dt, got_shp) = view(name)
assert got_dt == dt and list(got_shp) == list(shp), f"{name}: {got_dt}{got_shp}"
return mv
for e in range(n_e):
base = f"{pre}.mlp.experts.{e}"
slab = gu[e * f2 * h:(e + 1) * f2 * h]
slab[:g_bytes] = need(f"{base}.gate_proj.weight", F8, [f2 // 2, h])
slab[g_bytes:] = need(f"{base}.up_proj.weight", F8, [f2 // 2, h])
sslab = gus[e * gus_bytes:(e + 1) * gus_bytes]
sslab[:gs_bytes] = need(f"{base}.gate_proj.weight_scale_inv", BF16, gs)
sslab[gs_bytes:] = need(f"{base}.up_proj.weight_scale_inv", BF16, gs)
dn[e * dn_bytes:(e + 1) * dn_bytes] = need(f"{base}.down_proj.weight", F8, [h, f])
dns[e * ds_bytes:(e + 1) * ds_bytes] = need(f"{base}.down_proj.weight_scale_inv", BF16, ds)
for mm in mmaps.values():
mm.close()
# byte-identity check on two sampled experts (first and last), all four
# tensors — the stack IS the checkpoint bytes, so they must compare equal
for e in (0, n_e - 1):
base = f"{pre}.mlp.experts.{e}"
s = e * f2 * h
assert bytes(gu[s:s + g_bytes]) == bytes(stack_src(f"{base}.gate_proj.weight", meta, stage))
assert bytes(gu[s + g_bytes:s + f2 * h]) == bytes(stack_src(f"{base}.up_proj.weight", meta, stage))
t = e * gus_bytes
assert bytes(gus[t:t + gs_bytes]) == bytes(stack_src(f"{base}.gate_proj.weight_scale_inv", meta, stage))
assert bytes(gus[t + gs_bytes:t + gus_bytes]) == bytes(stack_src(f"{base}.up_proj.weight_scale_inv", meta, stage))
d = e * dn_bytes
assert bytes(dn[d:d + dn_bytes]) == bytes(stack_src(f"{base}.down_proj.weight", meta, stage))
u = e * ds_bytes
assert bytes(dns[u:u + ds_bytes]) == bytes(stack_src(f"{base}.down_proj.weight_scale_inv", meta, stage))
out = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
tensors = [
(f"{pre}.mlp.experts.gate_up_proj", F8, [n_e, f2, h], n_e * f2 * h),
(f"{pre}.mlp.experts.gate_up_proj_scale_inv", BF16,
[n_e, gs[0] * 2, gs[1]], n_e * gus_bytes),
(f"{pre}.mlp.experts.down_proj", F8, [n_e, h, f], n_e * h * f),
(f"{pre}.mlp.experts.down_proj_scale_inv", BF16,
[n_e, ds[0], ds[1]], n_e * ds_bytes),
]
bufs = iter([gu, gus, dn, dns])
written = write_safetensors(out, tensors, bufs)
return (pre, written)
def stack_src(name, meta, stage):
"""Re-open a source tensor for the sampled-expert check."""
f_, off, ln, dt, shp = meta[name]
fh = open(os.path.join(stage, f_), "rb")
mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
fh.close() # mmap keeps the mapping alive
return mm[off:off + ln]
def fill_ple_shard(job):
"""Decode one fp8 shard with the table scale, write bf16 rows into the mmap."""
(src_path, src_off, src_rows, dst_path, dst_off, scale_f32,
cols, chunk_rows) = job
lut = e4m3_lut()
with open(src_path, "rb") as fh:
sm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
with open(dst_path, "r+b") as dfh:
dm = mmap.mmap(dfh.fileno(), 0, access=mmap.ACCESS_WRITE)
for r0 in range(0, src_rows, chunk_rows):
r1 = min(r0 + chunk_rows, src_rows)
codes = np.frombuffer(sm, dtype=np.uint8,
count=(r1 - r0) * cols,
offset=src_off + r0 * cols)
q = lut.take(codes)
del codes # release the mmap export before sm.close()
q[~np.isfinite(q)] = 0.0
q *= np.float32(scale_f32)
packed = f32_to_bf16_rne(q)
dm[dst_off + r0 * cols * 2: dst_off + r1 * cols * 2] = packed.tobytes()
dm.flush()
os.fsync(dfh.fileno())
dm.close()
sm.close()
# ---------------------------------------------------------------------- main
def main():
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
ap.add_argument("--src", default=None,
help="raw checkpoint dir (documentation only; unused)")
ap.add_argument("--stage", required=True, help="staging dir to reshape in place")
ap.add_argument("--force", action="store_true", help="redo even if already staged")
ap.add_argument("--workers", type=int, default=min(8, os.cpu_count() or 1))
args = ap.parse_args()
stage = args.stage
t0 = time.time()
index_path = os.path.join(stage, "model.safetensors.index.json")
cfg_path = os.path.join(stage, "config.json")
if not (os.path.isfile(index_path) and os.path.isfile(cfg_path)):
log(f"ERROR: {stage} is not a staged checkpoint (missing index/config)")
return 2
with open(index_path) as fh:
index = json.load(fh)
with open(cfg_path) as fh:
cfg = json.load(fh)
wm = index["weight_map"]
meta = build_meta(stage, index)
# ---- collect prefixes -------------------------------------------------
pre_re = re.compile(r"^(?P<pre>.+)\.mlp\.experts\.(?P<e>\d+)\.gate_proj\.weight$")
pres, n_e = {}, 0
for k in wm:
m = pre_re.match(k)
if m:
pres.setdefault(m.group("pre"), set()).add(int(m.group("e")))
n_e = max(n_e, int(m.group("e")) + 1)
prefixes = sorted(pres)
if not prefixes:
log("ERROR: no per-expert tensors found in the staged index")
return 2
mtp = [p for p in prefixes if p.startswith("mtp.")]
trunk = [p for p in prefixes if p.startswith("model.language_model.layers.")]
log(f"expert prefixes: {len(prefixes)} (trunk {len(trunk)}, mtp {len(mtp)}), "
f"{n_e} experts each")
# shapes from expert 0 of the first prefix
pre0 = prefixes[0]
shapes = {}
gdt, gsh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight"][3:5]
ddt, dsh = meta[f"{pre0}.mlp.experts.0.down_proj.weight"][3:5]
gsd, gssh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight_scale_inv"][3:5]
dsd, dssh = meta[f"{pre0}.mlp.experts.0.down_proj.weight_scale_inv"][3:5]
scale_law = (gdt == F8 and ddt == F8 and gsd == BF16 and dsd == BF16
and list(gssh) == [(gsh[0] + 127) // 128, (gsh[1] + 127) // 128]
and list(dssh) == [(dsh[0] + 127) // 128, (dsh[1] + 127) // 128])
if not scale_law:
log(f"ERROR: unexpected expert shapes: gate {gdt}{gsh} scale {gsd}{gssh} "
f"down {ddt}{dsh} scale {dsd}{dssh} (scale law 128x128)")
return 2
shapes = {"ffn2_h": (gsh[0] * 2, gsh[1]), "ffn": (dsh[1],),
"gs": tuple(gssh), "ds": tuple(dssh)}
f2, h = shapes["ffn2_h"]
f = shapes["ffn"][0]
log(f"shapes: gate_up [{n_e},{f2},{h}] F8, down [{n_e},{h},{f}] F8, "
f"scales {gssh}x2 / {dssh} bf16")
# ---- PLE table --------------------------------------------------------
ple_re = re.compile(r"^(?P<pre>.+)\.ple\.ple_embedding\.ngram_embedding\.shard_(?P<i>\d+)\.weight$")
ple_pre, shard_ids = None, {}
for k in wm:
m = ple_re.match(k)
if m:
ple_pre = m.group("pre")
shard_ids[int(m.group("i"))] = k
if ple_pre is None:
log("ERROR: no PLE table shards found")
return 2
n_shards = len(shard_ids)
if sorted(shard_ids) != list(range(n_shards)):
log(f"ERROR: PLE shard ids not contiguous 0..{n_shards - 1}")
return 2
sc_name = f"{ple_pre}.ple.ple_embedding.ngram_embedding.weight_scale"
sdt, ssh, sln = meta[sc_name][3], meta[sc_name][4], meta[sc_name][2]
if sdt != BF16 or list(ssh) != [1]:
log(f"ERROR: PLE scale is {sdt}{ssh}, expected BF16 [1]")
return 2
s_bits = struct.unpack("<H", bytes(view_src_tuple(meta, sc_name, stage)))[0]
scale_f32 = struct.unpack("<f", struct.pack("<I", s_bits << 16))[0]
log(f"PLE: {ple_pre}, {n_shards} shards, scale bf16 = {scale_f32!r}")
rows = [meta[shard_ids[i]][4][0] for i in range(n_shards)]
total_rows = sum(rows)
hd = meta[shard_ids[0]][4][1]
if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
log("ERROR: PLE shards disagree on HD")
return 2
if hd % 128 != 0 and hd != 160:
log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
# ---- idempotence gate -------------------------------------------------
stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
merged_name = f"{ple_pre}.ple.ple_embedding.ngram_embedding.shard_0.weight"
already = (all(n in wm for n in stacked_names)
and merged_name in wm
and find_split(cfg) is not None and read_split(cfg) == 1)
if already and not args.force:
log("already staged (stacked names, merged shard_0 and split_ngram_parts=1 "
"all present) — nothing to do")
return 0
# ---- disk gate --------------------------------------------------------
stacked_bytes = len(prefixes) * n_e * (f2 * h + h * f)
ple_bytes = total_rows * hd * 2
need = stacked_bytes + ple_bytes
st = os.statvfs(stage)
avail = st.f_bavail * st.f_frsize
if avail < need * 1.05:
log(f"ERROR: {avail / 2**30:.1f} GiB free < {need / 2**30:.1f} GiB needed (+5%)")
return 2
log(f"disk: {avail / 2**30:.1f} GiB free, need {need / 2**30:.1f} GiB "
f"(experts {stacked_bytes / 2**30:.1f}, ple {ple_bytes / 2**30:.1f})")
# ---- stage experts ----------------------------------------------------
jobs = []
for idx, pre in enumerate(prefixes):
jmeta = {n: meta[n] for n in meta if n.startswith(f"{pre}.mlp.experts.")}
jobs.append((stage, pre, idx, n_e, shapes, jmeta))
ctx = mp.get_context("fork")
written_total = 0
with ctx.Pool(args.workers) as pool:
for pre, nbytes in pool.imap_unordered(stack_prefix, jobs):
written_total += nbytes
log(f" stacked {pre} ({nbytes / 2**30:.2f} GiB)")
log(f"experts stacked: {written_total / 2**30:.2f} GiB in {len(prefixes)} files")
# ---- stage PLE --------------------------------------------------------
# The file name sorts BEFORE the model-000NN shards ON PURPOSE: the reader
# opens every index-named file in sorted order and emplaces EVERY tensor
# it physically holds, first-wins (checkpoint.cpp add_shard). The raw
# shards carry tensors of the SAME name as the merged one
# (...ple.ple_embedding.ngram_embedding.shard_0.weight, [2500012,160]
# fp8), so ple.merged.safetensors lost the race and the plan read one raw
# shard as the codes. "00-" wins the sort and the name.
out_name = "00-ple.merged.safetensors"
out_path = os.path.join(stage, out_name)
data_bytes = total_rows * hd * 2
hdr = {merged_name: {"dtype": BF16, "shape": [total_rows, hd],
"data_offsets": [0, data_bytes]}}
blob = json.dumps(hdr, separators=(",", ":")).encode("utf-8")
pad = (-len(blob)) % 8
blob += b" " * pad
hlen = len(blob)
with open(out_path, "wb") as fh:
fh.write(struct.pack("<Q", hlen))
fh.write(blob)
fh.truncate(8 + hlen + data_bytes)
jobs = []
row0 = 0
for i in range(n_shards):
f_, off, ln, _, shp = meta[shard_ids[i]]
jobs.append((os.path.join(stage, f_), off, shp[0], out_path,
8 + hlen + row0 * hd * 2, scale_f32, hd, 131072))
row0 += shp[0]
assert row0 == total_rows
with ctx.Pool(args.workers) as pool:
pool.map(fill_ple_shard, jobs)
log(f"PLE merged: {data_bytes / 2**30:.2f} GiB bf16 "
f"({total_rows} rows x {hd} cols)")
# ---- post-verify ------------------------------------------------------
verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
scale_f32, out_path, hlen)
# ---- rewrite index + config -------------------------------------------
new_wm = {k: v for k, v in wm.items() if k not in set(shard_ids.values())}
for idx, pre in enumerate(prefixes):
fname = f"stacked.{idx:03d}.safetensors"
for t in (f"{pre}.mlp.experts.gate_up_proj",
f"{pre}.mlp.experts.gate_up_proj_scale_inv",
f"{pre}.mlp.experts.down_proj",
f"{pre}.mlp.experts.down_proj_scale_inv"):
new_wm[t] = fname
new_wm[merged_name] = out_name
total = sum(meta_size(meta, k) for k in new_wm
if k in meta) # originals
# new tensors' sizes
total += written_total + data_bytes
index["weight_map"] = new_wm
if "metadata" in index and isinstance(index["metadata"], dict):
index["metadata"]["total_size"] = total
# ATOMIC REPLACE, never open(path, "w"): the stage's index is a symlink
# to the RAW checkpoint (staging symlinks everything but config.json) —
# open("w") would rewrite the raw file through the link. os.replace
# swaps the link itself for a real file; the raw tree stays pristine.
for path, payload in ((index_path, json.dumps(index, indent=2)),
(cfg_path, json.dumps(cfg, indent=2))):
tmp = path + ".tmp"
with open(tmp, "w") as fh:
fh.write(payload)
os.replace(tmp, path)
n = set_split(cfg, 1)
log(f"config: split_ngram_parts=1 written at {n} location(s)")
with open(os.path.join(stage, "ple_scale.txt"), "w") as fh:
fh.write(repr(scale_f32) + "\n")
dt = time.time() - t0
log(f"staged in {dt / 60:.1f} min: {written_total / 2**30:.1f} GiB experts + "
f"{data_bytes / 2**30:.1f} GiB PLE; scale {scale_f32!r} -> ple_scale.txt")
return 0
def meta_size(meta, name):
_, _, ln, _, _ = meta[name]
return ln
def view_src_tuple(meta, name, stage):
f_, off, ln, _, _ = meta[name]
with open(os.path.join(stage, f_), "rb") as fh:
fh.seek(off)
return fh.read(ln)
def find_split(cfg):
"""Locate split_ngram_parts keys (recursive); return list of paths."""
hits = []
def walk(d, path):
if isinstance(d, dict):
for k, v in d.items():
if k == "split_ngram_parts":
hits.append(path + [k])
walk(v, path + [k])
walk(cfg, [])
return hits
def read_split(cfg):
hits = find_split(cfg)
if not hits:
return None
d = cfg
for k in hits[0][:-1]:
d = d[k]
return d[hits[0][-1]]
def set_split(cfg, value):
hits = find_split(cfg)
for path in hits:
d = cfg
for k in path[:-1]:
d = d[k]
d[path[-1]] = value
if not hits: # plugin's meta_int2 falls back to the top-level key
cfg["split_ngram_parts"] = value
hits.append(["split_ngram_parts"])
return hits
def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
scale_f32, out_path, hlen):
"""Cheap strong checks: headers parse, sizes match, sampled bytes identical."""
f2, h = shapes["ffn2_h"]
f = shapes["ffn"][0]
gs, ds = shapes["gs"], shapes["ds"]
# headers of every new file
for idx, pre in enumerate(prefixes):
path = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
hlen2, hdr = parse_header(path)
want = {
f"{pre}.mlp.experts.gate_up_proj": (F8, [n_e, f2, h]),
f"{pre}.mlp.experts.gate_up_proj_scale_inv":
(BF16, [n_e, gs[0] * 2, gs[1]]),
f"{pre}.mlp.experts.down_proj": (F8, [n_e, h, f]),
f"{pre}.mlp.experts.down_proj_scale_inv": (BF16, [n_e, ds[0], ds[1]]),
}
for name, (dt, shp) in want.items():
e = hdr[name]
assert e["dtype"] == dt and list(e["shape"]) == shp, name
assert e["data_offsets"][1] - e["data_offsets"][0] == \
int(np.prod(shp)) * (1 if dt == F8 else 2), name
_, phdr = parse_header(out_path)
e = phdr[next(iter(phdr))]
assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
# sampled PLE rows: recompute and compare bytes exactly
lut = e4m3_lut()
out_off = 8 + hlen
for i in (0, len(shard_ids) // 2, len(shard_ids) - 1):
name = shard_ids[i]
f_, off, ln, _, shp = meta[name]
with open(os.path.join(stage, f_), "rb") as fh:
mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
for r in (0, shp[0] - 1):
codes = np.frombuffer(mm, dtype=np.uint8, count=hd,
offset=off + r * hd)
q = lut.take(codes) * np.float32(scale_f32)
del codes # release the mmap export before mm.close()
want = f32_to_bf16_rne(q).tobytes()
got_row = sum(rows[:i]) + r
with open(out_path, "rb") as ofh:
ofh.seek(out_off + got_row * hd * 2)
got = ofh.read(hd * 2)
assert got == want, f"PLE row {got_row} mismatch"
mm.close()
log("verify: new-file headers, sizes and sampled PLE rows all exact; "
"expert slabs were byte-checked in-stack")
if __name__ == "__main__":
sys.exit(main())
+3 -3
View File
@@ -204,12 +204,12 @@ for path in "${CHANGED[@]}"; do
scripts/qwen-sharp-v22.4.0.jinja)
add_test tests/frank_large_unit.bats
;;
# frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
# frank-secondary is llama.cpp on the 3090 as a ROOT unit: a native CUDA
# binary, so no generated quadlet is involved. Its env, unit and serve
# script all live in scripts/, which is why this case has to precede
# the generic scripts/*.service and scripts/* fallbacks below.
scripts/frank-small.*|scripts/serve-frank-small.sh)
add_test tests/frank_small_unit.bats
scripts/frank-secondary.*|scripts/serve-frank-secondary.sh)
add_test tests/frank_secondary_unit.bats
add_test tests/llm_units.bats
add_test tests/deployment_scoping.bats
add_test tests/scripts_syntax.bats
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/env python3
"""Streaming absmax audit of the Swift1.5 PLE n-gram table (BF16).
The PLE fp8 encoding uses ONE fixed scale (1.9931793212890625e-04, the
official checkpoint's plateau value). fp8_e4m3 saturates at 448, so the
fixed scale is legal iff absmax <= 448 * SCALE. This walks every
ngram_embedding.shard_*.weight tensor in the checkpoint (mmap, no copy),
computes the absmax from the BF16 bit pattern (clear the sign bit, max
the remaining uint16 — BF16 magnitudes are monotone in their bits), and
reports the headroom. Exit 0 only when headroom >= 1.5x.
Run from any cwd EXCEPT /tmp on dankserver (/tmp/inspect.py shadows the
stdlib). Usage: verify-swift15-ple.py <ckpt-dir>
"""
import json
import mmap
import os
import struct
import sys
SCALE = 1.9931793212890625e-04 # bf16 0x3951, the official plateau scale
FP8_E4M3_MAX = 448.0
import numpy as np
gmax = 0.0
gfile = ""
over_cap = 0
total = 0
ckpt = sys.argv[1]
index = json.load(open(os.path.join(ckpt, "model.safetensors.index.json")))
shards: dict[str, list[str]] = {}
for k, f in index["weight_map"].items():
if "ngram_embedding.shard_" in k and k.endswith(".weight"):
shards.setdefault(f, []).append(k)
files = len(shards)
tensors = sum(len(v) for v in shards.values())
print(f"n-gram shards: {files} files, {tensors} tensors")
import math
cap = FP8_E4M3_MAX * SCALE
# largest bf16 <= cap: truncate the float32 bits (cap > 0); bf16
# magnitudes are monotone in their bits, so |w| > cap <=> bits > cap_bits
cap_bits = int((np.float32(cap).view(np.uint32) >> 16) & 0xFFFF)
for fname, keys in sorted(shards.items()):
path = os.path.join(ckpt, fname)
with open(path, "rb") as fh:
hlen = struct.unpack("<Q", fh.read(8))[0]
header = json.loads(fh.read(hlen))
base = 8 + hlen
mm = mmap.mmap(fh.fileno(), 0, prot=mmap.PROT_READ)
for k in keys:
meta = header[k]
off, end = meta["data_offsets"]
a = np.frombuffer(mm, dtype=np.uint16, count=(end - off) // 2,
offset=base + off)
v = a & 0x7FFF
m = int(v.max())
over_cap += int((v > np.uint16(cap_bits)).sum())
total += v.size
del a
if m > gmax:
gmax = m
gfile = k
mm.close()
# rebuild the BF16 value: the 16 bits are the HIGH half of the float32
absmax = np.frombuffer(b"\x00\x00" + np.uint16(gmax).tobytes(),
dtype=np.float32)[0]
print(f"absmax = {absmax!r} (tensor: {gfile})")
print(f"2^log2 = 2^{math.log2(absmax):.3f}")
print(f"fp8 cap = 448 * {SCALE:.8g} = {cap:.8g}")
print(f"headroom = {cap / absmax:.3f}x")
ok = cap / absmax >= 1.5
print("VERDICT:", "PASS (>= 1.5x headroom over the fixed scale)" if ok
else "FAIL — fixed scale does not cover the Swift PLE")
print(f"values > cap = {over_cap} of {total} ({over_cap / total:.3e})")
nxt_bits = int((np.float32(absmax / FP8_E4M3_MAX).view(np.uint32)
>> 16) & 0xFFFF) + 1
nxt_scale = np.frombuffer(b"\x00\x00" + np.uint16(nxt_bits).tobytes(),
dtype=np.float32)[0]
print(f"next-bf16 scale covering absmax: {nxt_scale:.8g} "
f"(0x{nxt_bits:04x}, headroom {FP8_E4M3_MAX * nxt_scale / absmax:.4f}x)")
sys.exit(0 if ok else 1)
@@ -0,0 +1,111 @@
---
type: reference
title: Agent Docs and Edits
description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
tags: [agents, docs, markdown, hashline, edit-recovery]
timestamp: 2026-10-05T00:00:00-07:00
---
# Agent Docs and Edits
## Doc-structure audit (after parallel agent edits)
Use after one or more subagents have edited markdown docs (OKF/Obsidian
vault with `---` frontmatter), before committing. Link checkers and
completeness seams validate *references*, not *structure* — agents reliably
produce structurally broken markdown that still links cleanly.
Damage catalogue:
1. Banner/prose prepended **above** the opening `---` → the frontmatter is
no longer frontmatter (title/tags/description become body text).
2. A rewrite **appended** instead of merged → duplicate headings and a stray
second `---` mid-file.
3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
`+- `, `- - `.
4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
5. Line-range replacement that dropped a sentence tail → dangling fragment,
often mid-parenthesis.
Checks (fast, run all):
```bash
head -3 <file> # must be `---` if the page has frontmatter
grep -c '^---$' <file> # exactly 2 (opener + closer)
grep -n '^## ' <file> | sort -t: -k2 | uniq -f1 -d # duplicate sections
grep -rnE '^\+- |^- - |<\|' <files> # artifacts
python3 scripts/check-vault-links.py <vault-dir>
./tests/run.sh # docs completeness seam etc.
```
Repair order (matters):
1. If the file was destroyed rather than edited: stash the agent version to
`.AGENTTRASH/<YYYYMMDD-topic>/`, then `git checkout HEAD -- <file>` and
re-apply only the intended changes by hand.
2. If it merely duplicated: **salvage unique content first** — cut the
one-of-a-kind section, paste it into the copy you keep, THEN delete the
duplicate ranges. Never delete first.
3. Re-add missing sentence tails (insert before the orphaned fragment),
keeping list continuation lines indented.
4. Re-run all checks above.
Gotchas: agents can write AFTER they appear finished (job settles, file
changes) — re-verify structure after the last background job reports, not
before. Editing an agent's already-damaged file with fresh line numbers is
safe, but its snapshot tag changes on every edit — re-read before each
subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so
nothing moved there reaches the repo — that is intended (temporary hold, not
archive).
## Hashline edit recovery
Failure mode: multi-op edits resolve every `PUT N.=M:` / `CUT N.=M:` index
against the **original** file numbering, and the post-apply diff view
**elides lines** (it shows an anchor window, not the whole file). Two traps
follow:
1. A second edit call built from a diff-view snapshot uses line numbers from
a *partial* view. Lines outside the window are invisible but still
counted.
2. If the first apply was itself wrong, the file shifted. The diff view
tells you the damage, but not the true offsets.
Result: the repair pastes fresh text **beside** stale text instead of over
it.
Procedure:
1. **Never carry line numbers across calls.** Re-read the target range for a
fresh tag before every subsequent edit. A read is cheap; a duplicated
paragraph costs a commit and a revert.
2. **One op per call when the previous apply was imperfect.** Fixed offsets
are only trustworthy when every preceding apply landed exactly as
specified.
3. **Re-read after every apply**, and verify the whole affected region — the
diff view is a sample, not proof. Check the *boundaries*: the line before
the edit and the line after it. Damage shows up as a repeated sentence
straddling the seam.
4. **Repair by CUT, not by re-PUT.** Once you know the exact stale lines
from a fresh read, delete them. Do not try to PUT a corrected superset
over a range whose extent you are guessing at.
5. Verify with a content assertion, not a diff review: `grep -c` for the
duplicated sentence, plus a syntax/compile check on any touched script.
Same rule for any imperative `edit` tool with an old/new match: the matched
text must be the **entire** text being replaced. Matching a bare fragment
inside a paragraph splices the replacement into the middle of the sentence
and silently corrupts the file. Re-read the file afterwards to confirm.
### Anchored in-page links
Renaming a Markdown heading breaks `[[#Old Heading]]` anchors. When a
section is retitled, grep for the old heading text and fix every anchor in
the same pass — doc link-checkers frequently do not validate `#fragment`
targets, so a green suite will not catch a dangling one.
## Related
- [[OMP Harness]] — the session context these repairs protect
- [[Command Center Dotfiles]] — the repo these edits commit to
- [[Testing Policy]] — the seams that catch broken vault structure
@@ -0,0 +1,87 @@
---
type: reference
title: Command Center Dotfiles
description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
tags: [dotfiles, omp, deploy, rsync, blankpad]
timestamp: 2026-10-05T00:00:00-07:00
---
# Command Center Dotfiles
Repo `~/.omp` is itself the deployment: packages are home-relative trees
under `dotfiles/<pkg>/`; `scripts/deploy.sh` symlinks live files into the
repo checkout. `git pull` + `deploy.sh` is the whole propagation story.
## Transport decision (agreed with Frank, executed 2026-09-09)
- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to
`<user>@<host>:/home/<user>/.omp/`, NOT clone-from-Gitea — `main` can be
ahead of `origin/main`; push is a user-only trigger, so clone would ship
stale state.
- Deploy runs on-target as the target user: `bash
/home/<user>/.omp/scripts/deploy.sh` (resolves the repo via `git -C
$(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be
present at the target path — full-replica transport guarantees this).
## Pre-overwrite audit (MANDATORY before first sync on a non-blank host)
Diff every file that `deploy.sh --check` reports as DRIFT against the repo
copy BEFORE letting quarantine-and-replace run. Host-specific divergence
belongs in an override package, not in quarantine:
1. Pull divergent live files from target
(`ssh ... 'cat <path>' > local-staged-copy`).
2. Secret-audit staged copies (grep token/secret/password/key shapes)
before anything enters git.
3. Compose `dotfiles/zz-<hostname>/` on the source machine from those live
copies; commit to repo first.
4. Ship the package dir via rsync, then run deploy sync — override deploys
LAST and win over base packages.
## deploy.sh mechanics (verified on blankpad)
- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it
first, always.
- Default mode = sync: adopt identicals silently, link new,
quarantine-and-replace divergents into `.quarantine/<host>/<ts>/`
(gitignored; never rm).
- **Override-target bug (fixed cc2c652):** `zz-<host>` packages nest
base-package dirs internally (`zz-blankpad/niri/.config/...`); target
derivation must strip the nested package segment, else overrides deploy to
phantom `$HOME/niri/.config/...` while real targets keep pointing at
canonical copies (visible symptom: check shows 0 actions but
`readlink ~/.zshrc` points at base `zsh/.zshrc`).
- systemd units auto-enable only when ExecStart binary resolves on that
host; dconf packages load via `dconf load /`, never symlinks.
- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once
post-transfer even before first sync.
## ssh/sshpass transport gotchas (all bit live)
- Resolve the target via mDNS FQDN (`avahi-resolve -4 -n <host>.local`); a
bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or
firewall default-drop, not auth failure — needs a local/console fix on
target.
- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; an env passed
via the tool's env parameter did not propagate to the sshpass child
process.
- The remote login shell may be **fish**: never send bash syntax
(`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash -c
"..."` or ship a script. See [[SSH and Remote Access]].
- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A
here-doc attached to the ssh command is consumed by `cat` — the intended
script body becomes the file content, silently. Verify shipped bytes with
an md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat`
drained it (false DIVERGED).
## Verification of a completed deploy
Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped
= idempotent steady state. Spot-check `readlink` on override files points
into `dotfiles/zz-<host>/`, not base packages.
## Related
- [[SSH and Remote Access]] — the transport mechanics this deploy rides
- [[Noctalia and Niri]] — the desktop packages this repo ships
- [[Deploy State Triage]] — the serverAppConfig deploy flow (different repo)
+274
View File
@@ -0,0 +1,274 @@
---
type: reference
title: OMP Harness
description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
tags: [omp, harness, compaction, models-yml, context, samplers]
timestamp: 2026-10-05T00:00:00-07:00
---
# OMP Harness
The harness reaches the fleet through `~/.omp/agent/models.yml` (model
registry; single source of truth — there is no copy on the servers) and
`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
this file is repo-managed; edits are harness-side only.
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
:7771, the 3090 compaction hub) — formerly `frank-primary`, `frank-large`,
`frank-secondary`. The SERVICES keep the old names (units, scripts, env
files, runbook pages); pages written in the old era name the service,
the harness names the model. The wire now matches: each endpoint's
`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
picker/grants, the homepage docs and the raw API all use one id per model
(see [[OpenWebUI]] for the grant re-key and the [[frank-primary-radiance-swift15-w4dn8]]
note on radiance echoing the requested name).
## Compaction chain arithmetic (verify before ANY window change)
When a fleet endpoint's context grows past its harness-declared window, OMP
compacts by sending the conversation to a designated target model. Before
changing any `contextWindow`/`maxTokens`, verify the chain arithmetically —
a target whose window is smaller than the source's trigger point silently
breaks compaction.
Where the contract lives:
- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL),
`methodOrder`, `thresholdPercent` (70), idle settings
- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
`compactionModel` references
- Implementation: omp plugin `dist/core/compaction/compaction.js`
The arithmetic (the coupled contract every endpoint's maxTokens comment
encodes):
1. **Source trigger** = `thresholdPercent × source contextWindow`
(0.70 × 200,704 = 140,492 for frank-primary on the current shape)
2. **Output budget** = capped at the target model's `maxTokens`
(32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
(frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary).
Procedure (recompute from the LIVE files every time — windows rotate with
every swap; the numbers above are the 2026-10-05 state):
1. Read the `config.yml` compaction block + `models.yml` entries for source
and target. Every endpoint's `contextWindow` comment carries its own
arithmetic — read them.
2. Read the target checkpoint's `max_position_embeddings` (native window) —
the served window may be a concurrency contract, not the native limit
(`--parallel N` DIVIDES the pool; the per-slot number is the contract).
3. If native ≥ trigger + budget, reslice the pinned pool:
`MAX_RUNNING × CONTEXT_LENGTH ≤ MAX_TOTAL_TOKENS` (the pin exists because
the runtime's autofit leaves too little for deferred loads + prefill
workspace; check `nvidia-smi` headroom — expect ~0.8 GB workspace margin).
4. Changes: host env (CONTEXT_LENGTH-style env tunables) via git →
`./tests/run.sh` → deploy; harness `models.yml` directly (no repo copy).
5. Verify: `/health` 200, boot log's pool line, then a long-prompt probe at
≥ trigger height (the compaction-fit guarantee), then concurrent requests.
Known reconciliation trap: comments in models.yml have described compaction
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
dankclaw — the comment/config disagreed. The comments now name the real
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
truth; fix comments in the same change.** `maxTokens` never changes for a
reslice — it caps output; `contextWindow` caps input. Boot-shape OOM
laddering is a different problem ([[Llama Cpp Serving Sizing]]).
## Per-model sampler config (`compat.extraBody`)
`models.yml` has **no per-model sampler keys**. Sampling is global
(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is
`compat.extraBody`:
**Why it works:** `applyOpenAIExtraBody` runs **last** in the
openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
`Object.assign(params, extraBody)`. So extraBody overrides global settings
and compat-policy output.
```yaml
compat:
reasoningEffortMap:
max: xhigh
extraBody:
temperature: 1.0
top_p: 0.95
top_k: 20
min_p: 0.0
presence_penalty: 0.0
repetition_penalty: 1.0
preserve_thinking: true
```
Use **wire names**, not OMP camelCase. Two traps:
1. **`preserve_thinking` is not user-settable as a compat key.**
`qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
*auto-detected* `thinkingFormat` + local baseUrl, **before**
`applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
models.yml does **not** enable it. Custom-named models (`frank-large`)
match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
also sets it — `Object.assign` is shallow and replaces the whole object,
silently dropping the `reasoning_effort` kwarg the compat policy set.
Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
FreeToken entry currently carries a `chat_template_kwargs` extraBody with
a comment asking to verify-and-prune — the exact trap to check at that
endpoint's next swap.)
**Gotcha: silent greedy decode.** halogen has **no server-side sampler
defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
endpoint decodes greedy while looking configured. This is why frank-large's
entry carries the recipe on the wire and a comment saying "do not drop it".
**Reasoning-effort folding.** Qwen3.8's template accepts only
`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
else ("Unexpected reasoning effort high"). Every endpoint's
`reasoningEffortMap` folds all harness levels onto that set; a role whose
level is unmapped fails on its FIRST request, and the failure names Jinja,
not the model.
### Verify the exact wire body (echo-capture)
Don't trust the YAML — capture what OMP actually emits:
1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and
returns a canned `chat.completion` (include `reasoning_content` so
thinking paths exercise).
2. Copy the **real** models.yml into a throwaway profile with only the
baseUrl swapped:
```bash
sed 's|http://<host>:7770/v1|http://127.0.0.1:18099/v1|' \
~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml
```
3. Run and inspect:
```bash
omp --profile wiretest --model "<provider>/<model>" -p "hi" \
--no-tools --no-session --no-extensions --no-skills --no-rules
jq 'del(.messages, .tools)' last-body.json
```
Confirm every intended field is present with the right value.
**Verify a field is *honoured*, not just accepted** — unknown fields often
return 200 anyway. Prove behavior with a **prompt-token delta**: replay an
assistant turn carrying a long `reasoning_content`, then compare
`usage.prompt_tokens` with the flag true vs false. A large delta means the
field changed template rendering; zero delta means it was ignored.
Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
never `rm`.
## Static-preamble audit ("the context feels bloated")
What loads statically every session (measure each, don't guess):
1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k
tokens — historically the largest block).
2. Skill index — only names+descriptions load; bodies are on-demand. Sum
description bytes across all SKILL.md frontmatter:
```bash
python3 - <<'EOF'
import glob, re, os
for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
n=tot=0
for sk in glob.glob(root+"/*/SKILL.md"):
fm=re.search(r'^---.*?---', open(sk, encoding="utf-8", errors="replace").read(), re.S)
d=re.search(r'description:\s*(.*(?:\n(?:\s{2,}.*)*)?)', fm.group(0) if fm else "")
n+=1; tot+=len(d.group(1)) if d else 0
print(root, n, tot)
EOF
```
Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user
skills.
3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name
'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest
MEMORY.md.
4. Harness/system prompt + tool schemas + CLI persona: fixed cost, ~20 KB;
not prunable.
Known rot patterns:
- Near-duplicate auto-captures of the same lesson (three "authentik ORM"
entries, the freetoken-build pair, the three job-runner skills) — dedupe
via `manage_skill delete`/`update`.
- Corrupted captures containing stray CJK characters (`一团ly`, `等方法`)
mid-sentence — a known defect of the auto-capture path; delete the corrupt
copy, keep the clean one.
- Stale host/service names in consolidated memory (pre-rename
`frank-default-large`) — the repo's AGENTS.md topology table is ground
truth.
Fix strategy: AGENTS.md prose (measured runbook narratives) → move to
`serverAppConfigs-docs/LLM/` runbooks, keep laws + tables in AGENTS.md (the
repo's own docs-tree rule mandates this). Managed skills live in
`~/.omp/agent/managed-skills` (safe to edit); user-authored skills in
`~/.omp/agent/skills` are NEVER touched by tooling. Memory store edits:
stage removed captures to trash first (`~/.AGENTTRASH/YYYYMMDD-topic`).
Baseline to beat (2026-10-05): ~35–40k tokens of preamble — AGENTS.md ~18k,
skill index ~10k, lessons ~3.5k, harness ~6k. Proposed reductions: −12k
(AGENTS.md slimming), −3k (skill dedupe).
## Managed-skill library hygiene
Use when auditing, pruning, or consolidating `~/.omp/agent/managed-skills`
(the auto-captured skill store; the full index is injected into EVERY
session's context, so dead or duplicate entries are a permanent tax).
The static-context budget (measured 2026-10-05):
| Block | Size | Notes |
|---|---|---|
| `AGENTS.md` (repo) | 72,179 B ≈ 18k tok | biggest single item |
| Skill index (~173 skills) | ~40 KB ≈ 10k tok | names+descriptions only |
| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures |
| Harness system prompt + tool schemas | ~23 KB | fixed |
Audit procedure:
1. Build an inventory: per skill, name, mtime, body word count, description,
first 12 headings (one pass, out to a file; read in pages).
2. Group into: retired-runtime (host/service gone; its runbook already lives
in `serverAppConfigs-docs/LLM/`), verbatim-duplicate pairs/triples (same
lesson captured twice — often the SAME DAY), same-lesson families
(N skills = one procedure), keep.
3. Read full bodies of every merge candidate before proposing a merge;
description alone proves overlap only for exact dupes.
4. Present A/B/C/D verdicts for user approval. NEVER execute before the
discussion concludes — the user gates execution explicitly.
5. On approval: stage everything to trash first
(`~/.AGENTTRASH/YYYYMMDD-<topic>/`), never rm. After merging: recreate
the surviving skill via `manage_skill` update with body = superset,
delete absorbed ones.
Known rot classes (this library, 2026-10-05):
- Same-day duplicate captures (freetoken-build-dankclaw vs
freetoken-dankclaw-build; the three job-runner skills).
- CUDA-skill vs its self-declared HIP twin (pcie-link-capability-probe vs
pcie-link-probe-hip-rdna4) → merge.
- Era ghosts: skills naming retired runtimes (freetoken, lucebox, r9v/
radiance, old halogen-flash-server image, ik_llama, GLM-5.3 experiment).
- Skills contradicting user standing orders (live-bench-dashboard vs the
2026-09-29 delete-dashboards order) — flag for user, don't assume.
- Memory store: corrupted captures with CJK noise from broken ssh commands;
stale host/service renames.
Sizing verdicts that worked: bench family → 2 skills (harness vs
discipline), not one mega-skill; keep profiler/kernel-time attribution
standalone; fold go/no-go arithmetic skills into their probe skill
(heterogeneous-accelerator-split-triage → PCIe probe).
## Related
- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes
+23
View File
@@ -0,0 +1,23 @@
---
type: index
description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
tags: [agent, index]
timestamp: 2026-10-05T00:00:00-07:00
---
# Agent
Pages about the agent's own tooling and context — how omp is configured, how
docs get edited safely, and how the agent's home is deployed.
- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
`compat.extraBody`, preamble and skill-library audits
- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
hashline edit recovery
- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
transport, override packages
## Related
- [[Testing Policy]] — the seams these pages' checks ride
- [LLM](../LLM/index.md) — the fleet these agents operate
@@ -0,0 +1,151 @@
---
type: reference
title: Deploy State Triage
description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
tags: [deployment, deploy-state, systemd, podman, triage]
timestamp: 2026-10-05T00:00:00-07:00
---
# Deploy State Triage
The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
on each target (rootless, user `dank`, format exactly two lines
`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
+ `mv`). It is written ONLY by a `deploy.sh` dispatch.
## 1. Stale marker vs live container
**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
a restart after the checkout ff-pulled to a newer revision — starts the
container from the CURRENT `scripts/<svc>.env` but does NOT advance the
marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
"behavior pending".
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
harness's own model backend, so a blind deploy is a self-inflicted outage for
possibly zero behavioral change.
**Procedure (read-only first):**
1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
operational delta is usually one `scripts/<svc>.env`.
2. Probe the target (ssh as `dank`, passwordless sudo present):
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
- `git -C ~/serverAppConfigs rev-parse HEAD`
- `systemctl is-active/is-enabled <svc>`;
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
- the on-host env file already carries the new keys (it landed via the
ff-pull).
3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
`dank` (rootless store; bare `docker` is NOT the shim → "No such object";
`sudo podman` inspects an empty store):
- `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
the pull ⇒ it read the new env
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
- confirm the launcher maps each tunable to its flag
(`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
--enable-prefix-caching`.
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
live — **do NOT restart**; reconcile the marker (below). Only if the live
container lacks the flag/env (or predates the pull) is a real
`deploy.sh <target> --service <svc>` warranted.
**Reconcile (no restart)** — mirror the deployer's atomic idiom:
```bash
S=$HOME/.local/state/serverAppConfigs/deploy-state
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
```
Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
`tests/e2e/.env.local`; expect the documented `11-admin` red if the
FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
no-op).
**Anti-patterns:** trusting a commit message ("X deployed") or the marker
alone — both are bookkeeping; the container's argv/env is ground truth.
Reading container state with `sudo podman` / bare `docker`. Restarting a
healthy heavy endpoint to satisfy a stale marker.
## 2. Pending lockout
Symptom: `deploy.sh <target>` fails or silently stops with
`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
appears.
1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
`echo $?`.
2. Inspect the state store (no sudo):
`ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
A `status=pending` entry for a service absent from `service_is_allowed`
in `scripts/deploy-remote.sh` = retired-service lockout.
3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
not-allowed pending entries to
`~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
continues. Verify the WARN line names the service and the state file is
gone.
4. On a host without the fix: move the stale entry to
`~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
`deploy-state/` no longer holds the retired name.
Related contracts: the superseded seam
(`deployment_scoping.bats` "remote runner recognizes a pending service at
the same revision") now asserts prune-and-continue (exit 0), not exit 2;
`deployment_flow.bats` extracts `add_pending_services` via sed and
HOME-overrides the trash path. **When retiring a service going forward:**
delete or reconcile its deploy-state entry as part of the retirement commit —
the prune is a safety net, not the intended cleanup path.
## 3. Interrupted or raced deploys
The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
(runs on the target over ssh). **Killing the dev-side job does NOT kill the
host-side one** — two `deploy-remote.sh` processes can end up managing the
same unit, one deploying a revision you already superseded. Check:
`pgrep -af "deploy-remote"` on the target, read the revision argument on
each, kill stale ones by explicit PID.
A killed deploy leaves behind:
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
explicit stop — start it yourself);
- the state marker **pending** (pre-deploy or in-flight revision);
- the e2e gate un-run (the deployer runs it last).
Reconcile the marker ONLY after the service itself is verified healthy — a
marker that runs ahead of reality is worse than one that lags (the planner
uses it to decide what is pending). Then close the loop:
1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
slots, argv flags, `NRestarts`), not just health 200.
2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
does NOT have it; copy it in first).
3. If the killed revision was restoring a known-good shape, confirm the
numbers match the last measured-good boot of that shape — equality is the
proof, not health alone.
Boot failures must be read from the journal, not `podman logs`: serve scripts
`podman rm -f` the container on every start (previous attempt lost), and
`podman logs` on a container still *Stopping* returns the OLD configuration's
numbers — which looks exactly like your change having no effect. Key on
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
procedure.
## Related
- [[Deployment Flow]] — the deploy pipeline itself
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
@@ -63,7 +63,7 @@ service on the other host.
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
| `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
| `frank-small` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
@@ -20,11 +20,12 @@ user sees it; the `bats` seams assert config shape, not behavior.
| Path | Role |
|------|------|
| `playwright.config.ts` | four projects (below), real-browser UA, trace on failure |
| `playwright.config.ts` | five projects (below), real-browser UA, trace on failure |
| `specs/00-auth.setup.ts` | setup project: two real authentik logins, saves storage states |
| `specs/10-homepage.spec.ts` | user panel set + open links derived from `services.json` (registry-derived — no hardcoded service lists) |
| `specs/11-admin.spec.ts` | `admin` project: admin grid + `/admin/` RBAC (403 for standard user, members-only 200) |
| `specs/12-crawl.spec.ts` | every user open link reaches its real app through the guard chain |
| `specs/19-frankbook.spec.ts` | `frankbook` project: walks the six admin links that terminate on FrankBook-Pro (192.168.0.112) — conditional, see below |
| `specs/13-sso.spec.ts` | direct guarded-vhost visits resolve silently with ONLY the homepage session (single-login enforcement) |
| `specs/14-guard.spec.ts` | `guest` project, no storage state: unauthenticated traffic lands on the IdP login |
| `lib/login.ts` | `akLogin` helper (two-stage flow, shadow-DOM locators) |
@@ -38,7 +39,16 @@ user sees it; the `bats` seams assert config shape, not behavior.
`.auth/qa-e2e{,-admin}.json` storage states.
- **standard** — 10/12/13 with the user state.
- **admin** — 11 with the admin state.
- **guest** — 14 with an empty state.
- **guest** — 14/15 with an empty state.
- **frankbook** — 19 with the admin state. **Excluded from every default run**: the
*arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
sleeping Mac mini must never fail an unrelated deploy. The project joins only
when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
`shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
that touches the arr links must therefore be paired with a `19-frankbook` run
against an awake host before it is trusted live.
Credentials live in `tests/e2e/.env.local` (git-ignored; placeholders in
`.env.local.example`). The scaffold bats seam fails the commit if creds or
@@ -47,8 +57,9 @@ auth states are ever tracked.
## Run
```bash
cd tests/e2e && npx playwright test # all projects
npx playwright test --project=standard 12-crawl.spec.ts # one slice
tests/e2e/run.sh # default set — frankbook project excluded
E2E_INCLUDE_FRANKBOOK=1 tests/e2e/run.sh # default set + the frankbook walk
cd tests/e2e && npx playwright test # ALL projects, frankbook included
```
Requires LAN reachability of `www.franksdank.cloud` (Pi-hole DNS or public).
@@ -0,0 +1,209 @@
---
type: reference
title: Quadlet Lifecycle
description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
timestamp: 2026-10-05T00:00:00-07:00
---
# Quadlet Lifecycle
Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
working container MUST have a self-healing, boot-persistent Quadlet unit —
`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
A container that runs only from compose dies silently at reboot. This page
is the operating manual for that lifecycle.
## 1. Ownership audit — find the containers that will not survive a reboot
```bash
for c in $(podman ps --format '{{.Names}}'); do
printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
done
```
- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
show-user <user> -p Linger` = yes.
- **label absent, compose service exists** → the classic gap: dies at reboot.
Check the publish address while there — compose forms often bind the
wildcard (an unauthenticated UI/API on every interface).
- **label present AND a compose service with the same name** → two owners; a
stray `podman compose up` can create a duplicate that fights the unit for
the name and the port. Remove the compose service, leave an explanatory
comment (repo convention).
## 2. Cutover: compose service → Quadlet unit
Checklist (precedents to copy:
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
`dankclaw/observability/quadlet/`):
1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
line for line: `Image=` (`localhost/` prefix for locally built images +
`Pull=never`), `ContainerName=` (Caddy and scrape configs address
containers by name), `Network=`, `User=0:0` when the image's own user
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
`Exec=` for compose `command:`, `PublishPort=`, health keys,
`[Service] Restart=always` + `RestartSec=10`, `[Install]
WantedBy=default.target`. **Never `systemctl --user enable` the generated
unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
is healthy; explicit enable fails with "transient or generated").
2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
it (Caddy reaches containers by name over the shared network; the wildcard
publish is a guard-chain bypass).
3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
`systemctl --user restart <name>.service` → `wait_for_health`; retire any
legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
`~/.config/systemd/user/<svc>.service` that shells out to
`podman compose up` collides BY NAME with the generated unit);
`podman rm -f <container>` so the unit's container can take the name.
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
ordered. Add a `test-changed` case above the generic
`*/quadlet/*.container` fallback.
5. Retire the compose entry in the SAME change — two owners is how a stale
duplicate comes back.
6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
wildcard publish, container name matches Caddy/scrape references, health
command, deployer installs+restarts+waits, never `rm`, never `enable`.
Migrate any older assertion that read the compose file for devices/flags.
7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
active/generated, container `healthy`, endpoint answers,
`list-dependencies default.target` shows the unit, off-host probes on
previously published ports now refuse, and the container no longer carries
`io.podman.compose.project`.
**Traps that cost real time:**
- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
no shell → the check fails forever. `--health-cmd` won't take exec form
either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
with `--format docker` — the default OCI format has no HEALTHCHECK field
and drops it with only a warning.
- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
the unit dies with `stat /root/.config: permission denied`. Use a
per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
handler with a single-variable filter, validated so a missing secret fails
loudly.
- `Network=host` exporters scraped by containerized Prometheus must bind the
host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
for the homepage probe — two `--web.listen-address` flags, never the
wildcard.
- alloy 1.x refuses the `ui` config block; the unit must pass
`--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
no supplementary groups).
- When the last service leaves a compose file, the compose-validity seam
fails (requires ≥1 service): keep the file as comments and relax the seam
deliberately, or delete the file and fix the planner's compose-diff path —
decide with the owner, the deploy tooling reads it.
## 3. Drift / crash-loop diagnosis (5 minutes, in order)
Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
looks right; or a service quietly down for days because a restart policy kept
something running. Cause: the running container was created from an OLDER
definition — a compose/unit edit does not touch an existing container, and
the restart policy faithfully resurrects it forever.
1. **Who started it:** `podman inspect <name> --format 'unit={{index
.Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
created={{.Created}}'`; compare `created` with when the definition changed
(`git log -p --follow <config>`). Created-before-the-edit is the signature.
2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
`{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
for containers started with `--device`/`--group-add` — a false negative
that sends you hunting the wrong root cause.
3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
(a capability error like `No HIP GPUs are available` beside a missing
device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
instance reached 9,146 restart attempts on stale config.
Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
= the §2 cutover. **Verify the capability, not the status:** `podman exec
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
- **Hard-fail keys** — the whole unit silently fails to generate
(`systemctl --user cat <unit>` shows "could not be found"): any
dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
in group 'Container'`); `Command=` is unsupported, use `Exec=`.
- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
spaces into the healthcheck argv; a quoted probe
`/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
command-not-found → container reads **unhealthy while the DB is up**.
HealthCmd must contain only space-free tokens
(`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
unhealthy but logs look fine, compare the baked
`{{.Config.Healthcheck.Test}}` array against the intended command before
believing the status.
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
`daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
"unsupported key"`, `systemctl --user cat probe.service` (shows the
generated ExecStart verbatim), then remove + reload.
## 5. Rootless Quadlet log retrieval
Step 0 — establish ownership: `systemctl --user status <name>.service`
(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
`--user` knows the unit, EVERY command runs as that user with NO sudo.
| Need | Command |
|---|---|
| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |
Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
recreation** — the journal is the only durable record. Traps:
`sudo podman logs` → "no such container" (root's store does not contain
rootless containers; same for `sudo podman ps`); a system-level
`journalctl -u <name>` returns nothing for a user unit; during first-start
weight downloads health reports unhealthy BY DESIGN (check the payload file
is growing). Verbose vendor switches are a config change — through git and
deploy.sh, never a live env poke.
## 6. Compose bring-up with ground truth (rootless hosts)
1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
check every namespace (`sudo podman ps -a`) and native units
(`systemctl list-units`) before declaring a service absent. Compare
compose mount sources against the real filesystem — compose files name
phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
`/mnt/entropy`). Fix compose through git first, deploy second.
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
or creation fails `statfs …: no such file or directory`.
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
`{{.State.Health.Status}}`, then the rest. Fresh containers answer
000/refused for the first seconds — poll.
4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
request for nonexistent/private repos ("manifest unknown"/token len 0 =
repo or tag does not exist, not auth). Query the registry API
(`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
`/v2/<ns>/<repo>/tags/list`).
5. **Read the interface from inside the image** before guessing flags/envs:
`podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
configured it serves cwd read/write for everyone (an unauthenticated
world-writable share).
6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
mangled pattern is not absence — re-verify the target exists.
7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
(host 1000 == container 0): drop the user line, run as container-root.
## Related
- [[Deployment Flow]] — the deployer that installs these units
- [[Stale Signal Audit]] — the audit that catches drift's victims
- [[Container Network Namespaces]] — reachability between containers
- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
- [[Infrastructure/Observability]] — same class, twice
@@ -0,0 +1,144 @@
---
type: reference
title: Stale Signal Audit
description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
tags: [testing, observability, audit, signals, gates]
timestamp: 2026-10-05T00:00:00-07:00
---
# Stale Signal Audit
Run after any change and before trusting a green gate. The productive question
is **which signal here cannot possibly be correct?** Every pass on this repo
(4 of 4) found real defects — including one in the rule file every future
agent reads first.
## The surfaces, and what each vantage point can reach
| Surface | Producer's vantage point | Can reach |
|---|---|---|
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
Anything else — another host's loopback, a container-internal port never
published — is unreachable no matter how healthy the service is. The failure
mode is always the same shape: a signal that **cannot** succeed by
construction, which teaches the operator to ignore the surface it lives on.
## 1. Permanently-red probes (reachability)
For every monitor/probe/alert target ask: *can this checker physically reach
that endpoint?* Seen classes: a card probing another host's container-internal
port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
`/metrics` that is disabled (Gitea 404).
- A scrape cannot express "is this HTTP service answering?" — the right probe
is the **blackbox exporter** in probe mode (which also gives guarded vhosts
a real uptime signal, since Prometheus cannot solve an Anubis challenge).
- **Fix in the direction of honesty, never of silence:** unreachable target →
mark the homepage card `wip: true` with the reason in `description` (the
checker skips `wip`; `wip` is the existing convention for "real service,
not probeable"), or repoint the job at something reachable. NEVER open a
database port on the LAN to make a badge green, and never delete a scrape
job — an absent scrape and a healthy one are indistinguishable.
- Alert rules must not depend on always-red signals — scope them to services
that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
exactly that reason).
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
`connection refused` to `unsupported Content-Type "text/html"` means the
service came up and only the scrape is wrong. Read the error, not just
up/down.
## 2. Vacuous assertions (highest severity — the gate is already dead)
A seam that greps a file passes *by absence* once the file is retired:
`grep` exits 2 on a missing path and `!` inverts it.
```bash
grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
```
Fix shape — one place asserts the invariant, each local site keeps a live
check:
```bats
@test "no <host> compose file competes with the quadlet units" {
local cf="$REPO_ROOT/<host>/compose.yaml"
[ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
}
# at the per-service site: still fails if the entry returns
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
```
**Prove teeth:** recreate the offending file
(`printf 'services:\n <svc>:\n image: x\n' > <path>`), run the seams,
confirm RED, then move the file to trash. A re-armed check you have never
seen fail is still unverified.
## 3. Gates that never run
Playwright specs in `tests/e2e/` are matched by **number prefix** in
`playwright.config.ts`; an unregistered spec silently never runs and the
suite count stays the same. Count tests before/after;
`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
## 4. Validate the checker before acting on it
A checker's output is a hypothesis about another tool's semantics. Modelling
Obsidian with naive text matching produced 45 "broken links" of which 3 were
real — the rest were case-sensitivity (a `qBittorrent` link to
`qbittorrent.md`), path-qualified resolution (a path-qualified
`folder/Note` link resolves by path SUFFIX, not
stem), and `%20` URL-encoding in markdown links. Fix the checker first,
re-derive the report, only then edit documents. The result lives in
`scripts/check-vault-links.py` with those three rules in its docstring,
gated by the `no unresolved links in the docs vault` seam. Same class: a
`find … | head -1` selector that reads the wrong file; a truncated listing
that makes you "fix" a file that exists.
## 5. Prose invalidated by your own change
After retiring a file or adding a feature, grep for EVERY reference, not just
code:
```bash
grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
```
This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
after it was restored with GPU access; a unit said "there are no alerting
rules" hours after five were added; a docs page documented three scrape jobs
as working when they could never report up. **Clean cutover includes prose** —
docs, unit docstrings, and the repo's rule file (the most expensive place to
be stale).
## Order and verification
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
gates, docs/rule files, dashboard panels).
2. For each, ask what the signal CAN express, not what it says.
3. Validate any checker before editing the things it flags.
4. Fix at the root; never weaken an assertion to make it pass.
5. Gate the class so it cannot return (a seam), then prove the seam fails on
the defect.
Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
Prometheus panel expression against the API (0 series on a panel that should
show data is a defect; 0 `ALERTS` when nothing fires is not); read the
deployed `/srv/homepage/status.json` and triage every `state != up` entry
against reality; query `/api/v1/targets?state=active` and print
`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
Report per finding: the surface, why the signal could not have been correct,
the fix, and the red/green evidence — flag any regression you introduced
yourself in the same breath (one was found this way: a tightened bind killed
a reverse-proxied vhost while every gate stayed green, because the spec
covering that hostname did not exist).
## Related
- [[Testing Policy]] — the seam inventory this audit walks
- [[Observability]] — the scrape/probe surfaces it audits
- [[Quadlet Lifecycle]] — the ownership labels the probes read
@@ -0,0 +1,121 @@
---
type: howto
description: "Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which models/LoRAs to add next."
tags:
- dankclaw
- comfyui
- qwen-image
timestamp: 2026-10-05
---
# ComfyUI Qwen-Image Workflows
Two ready-to-load text-to-image workflows for the Qwen-Image 2.1 payload
already on Dankclaw, both proven end-to-end on 2026-10-05 by POSTing to the
live API on port 8188 and verifying the rendered PNG. The repo copies are the
source of truth; the host copies under `/home/dank/comfyui-data/workflows/`
are for convenience.
## What is on disk
| Path (under `/home/dank/comfyui-data/models/`) | Size | Role |
|---|---|---|
| `diffusion_models/qwen_image_2.1_int8_convrot.safetensors` | 6.8 GB | **The one the workflows use** — int8-convrot quant, ComfyUI-core native |
| `diffusion_models/qwen_image_2.1_bf16.safetensors` | 14 GB | Full-precision fallback |
| `diffusion_models/qwen-image-2.1-Q8_0.gguf` (+ `-arch`) | 7.1 GB | GGUF alternative (needs ComfyUI-GGUF custom node) |
| `text_encoders/qwen3vl_8b_w4a8.safetensors` | 5.9 GB | **The one the workflows use** — 4-bit-weights/8-bit-act Qwen3-VL 8B encoder |
| `text_encoders/qwen3vl_8b_int8_convrot.safetensors` | 8.8 GB | Mid quant |
| `text_encoders/qwen3vl_8b_bf16.safetensors` | 17 GB | Full precision |
| `vae/qwen_image_2.1_vae_bf16.safetensors` | 645 MB | 2.1 VAE (used) |
| `vae/qwen_image_vae.safetensors` | 243 MB | Smaller VAE |
No LoRAs are on disk yet — see [Recommended additions](#recommended-additions).
## The workflows
Both are API-format JSON (ComfyUI 0.37 loads them via the UI's **Load** dialog
too): `dankclaw/comfyui/workflows/` in the repo,
`/home/dank/comfyui-data/workflows/` on the host. Node chain in both:
```
UNETLoader (int8_convrot) ──┐
CLIPLoader (qwen_image, w4a8) ─┤ CLIPTextEncode pos/neg
VAELoader (2.1 bf16) ────────┤
EmptySD3LatentImage ─────────┴─ KSampler ─ VAEDecode ─ SaveImage
```
**Common sampler settings:** `cfg 1.0` (Qwen-Image 2.1 expects CFG-free
sampling), `euler` + `simple`, seed fixed in the file.
| Workflow | File | Shape | Steps | Made for |
|---|---|---|---|---|
| Art | `qwen-image-2.1-t2i-art.json` | 1024×1024 | 20 | General illustration/prose prompts |
| Legible text | `qwen-image-2.1-t2i-legible-text.json` | 832×1216 | 25 | Posters, captions, any in-image text |
### Legible-text prompt recipe
Qwen-Image 2.1 is one of the best open-weight text renderers, but only if the
prompt is written for it:
1. **Quote the exact strings**: `large bold white text reading "SOLIDARITY OF
THE SOVIET COMPUTER SYSTEMS"` — quoted text is rendered literally.
2. **Describe layout, not vibes**: say *centered / below it / large vs small*,
the typeface family (sans-serif), and the color.
3. **Say what NOT to draw**: end with `no other words, no watermark`.
4. Keep the negative side on `misspelled, warped letters, extra text` (already
in the workflow).
5. More steps help text: the text workflow uses 25 vs 20.
Verified render: `SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS` + `DANKCLAW
DANKSERVER EST 2026` came back crisp and correctly spelled at 832×1216
(`qwen21-text_00001_.png`, 953 KB). One caveat from the same render: a
shorter secondary line is where misspellings creep in — proofread the first
render before shipping.
## Driving it from the API
ComfyUI 0.37 **requires the graph wrapped in a `prompt` key** — a bare
`{node-id: node}` body gets `{"error": {"type": "no_prompt"}}` (verified
2026-10-05):
```bash
curl -s -X POST http://127.0.0.1:8188/prompt -H 'Content-Type: application/json' \
-d "$(python3 -c 'import json;print(json.dumps({"prompt": json.load(open("dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json"))}))')"
# -> {"prompt_id": "...", ...}
curl -s http://127.0.0.1:8188/history/<prompt_id> # status.status_str == "success"
```
Outputs land in `/home/dank/comfyui-data/output/`. The queue is FIFO — the
two workflows above ran back-to-back unattended.
**Note:** the container's UI-saved workflows live in the ephemeral
`/opt/ComfyUI/user` layer and do NOT survive an image rebuild — hence the
host-side `workflows/` dir and the repo copy.
## Recommended additions
Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
frank-secondary's). All repos verified 2026-10-05. None are on disk yet.
| # | What | Repo | Grab | Size | Why |
|---|---|---|---|---|---|
| 1 | **Lightning 8-step LoRA** (T2I speed) | `lightx2v/Qwen-Image-Lightning` | `qwen_image_lightning_8steps_*.safetensors` (LoRA) | ~1–2 GB | 20 steps → 8 with a `LoraLoader` node; 2–3× faster renders on the iGPU, negligible quality loss |
| 2 | **Qwen-Image-Edit-2511 GGUF** (image editing) | `unsloth/Qwen-Image-Edit-2511-GGUF` | Q4_K_M (~13.2 GB) or Q4_0 (~11.9 GB) | ~12–13 GB | Unlocks img2img "edit this photo" workflows — a capability class the fleet has zero of; Q4 keeps it iGPU-friendly |
| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI |
| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` CLIP type is already in this build's loader enum) |
Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
#3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count +
sha256 checks) or a plain `hf download` with the token in the environment —
never on the command line.
Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
them at GTT capacity, and they double the disk bill).
## See Also
- [[ComfyUI]] — the service itself (image, unit, GPU access)
- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path
- [[Dankclaw Stack]] — the host
@@ -110,5 +110,6 @@ The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `
- [[Dankclaw Stack]]
- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
- [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions
- [[Lemonade and sd.cpp]] — the ggml-family alternative
- Knowledge base: [[ComfyUI]] (tools)
+172
View File
@@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto
The npx package cache lives in the container layer, so a recreation (not just a
restart) pays one first-call npm download; subsequent calls are cached.
## Config-home drift ("cannot connect" to models that answer curl)
`podman exec <container> printenv HERMES_HOME` for each hermes container. If
the webui reads the image default (`/opt/data`) while the gateway reads
`/data`, the webui runs the image-default template config (`provider: auto` →
openrouter.ai with no key = connect failure).
The recreation trap is the same one the volumes table shows: image `VOLUME`
points get a fresh anonymous volume on every container recreation
(`podman run --replace --rm`) — edits inside such a mount are lost on
restart/redeploy. Compare volume ids before/after a restart to confirm.
Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
root), deploy service-scoped. Verify with the app's own code, not guesses:
```python
from api import config as C; C.resolve_model_provider("<model-id>")
# -> (model_id, provider, base_url); bare ids map to the owning provider by
# scanning the providers: dict models allow-list; @provider:model forces one.
hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
# -> ProviderDef with real base_url; accepts api/url/base_url keys.
```
Named providers (`providers:` dict) are a catalog + connection source;
`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
neither holds the provider name, the request has no endpoint and fails to
connect. Probe the endpoint with the **exact served model id** — the retired
FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
checks miss this class entirely.
## Live config edits
Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
uid 109999; all reads/edits need `sudo`). The repo file
`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
live — always diff the live file first; never treat the example as truth**.
1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
(in-dir `.AGENTTRASH/` also exists for removals).
2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
the live file are tool-generated — preserve them.
3. **Sweep stale references** across the whole data dir:
`sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
--include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
historical — leave them.
4. **Derived caches:** `webui/models_cache.json` is keyed by a config
fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
is stale. Let it regenerate lazily or trash it — never hand-edit.
5. **Restart the model-consuming units** (they parse config at startup):
`systemctl --user restart hermes-webui.service hermes.service`.
`hermes-dashboard.service` is unaffected by routing changes.
6. **Verify behaviorally**: one tiny completion per endpoint with the exact
served id — POST `/v1/chat/completions` and read the body, never just the
status code (endpoint names rotate; see the LLM knowledge base).
7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
live shape, redacting secrets (`password_hash`, phone account stay
live-only), then `./tests/run.sh` before commit.
`provider: custom` + `base_url` entries route by base_url and send the model
id verbatim; named providers resolve via their `api:` URL. **Renaming a
served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
config in the same change** or Hermes requests 404 on nonexistent models —
this exact gap caused the 2026-09-05 outage.
## Auxiliary routing against local backends
When a Hermes feature that calls an auxiliary task (e.g. title generation)
fails against a local endpoint, the usual cause is a **request shape the
local runtime does not implement**, not a routing mistake.
Diagnose with the real call path — run the actual function inside the
container with the app's own constants (the probe cannot drift from
production):
```bash
base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
```
Copy the file AFTER any restart (`--replace --rm` discards the container
layer). The CLI at `/opt/hermes/hermes` uses system python and dies
`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
```python
import sys; sys.path.insert(0, "/opt/hermes")
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
from agent.auxiliary_client import call_llm
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
```
What actually breaks, and the fix:
- **`response_format` (json_object / json_schema) is a hard 400 on every
local backend here** — neither FreeToken nor halogen implements constrained
decoding. Config cannot override it: `call_llm` merges the task's
`extra_body` first and then `update()`s the caller's, so the hardcoded one
wins. Only temperature and max_tokens have reactive retry paths.
- **Fix: move that one task to the Responses transport**, which never
forwards `response_format`:
```yaml
auxiliary:
title_generation:
provider: dankserver-small # names track the live config; diff first
model: frank-secondary
api_mode: codex_responses # real per-task key
extra_body:
reasoning:
effort: none # mandatory, see below
```
- **Disable thinking for any cap-limited task.** The adapter omits reasoning
only when config says `reasoning.enabled: false`. The
`reasoning_effort: none` *shorthand* produces exactly that and leaves the
server default (thinking ON) — the 64-token title budget gets spent on the
think: empty content, `finish_reason: length`. The explicit
`extra_body.reasoning.effort: none` instead sends
`{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
honors.
- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
proxy needed for the alternate transport.
Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
it must pass from config alone (that is the production path); keep the live
file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
config change is docs+example only, no service deploy needed.
Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
`apply_instant_title`, so disabling a broken title task also removes the
derived title — it does not degrade gracefully the way a failure does. A
local call can also fail at the model level: a 200 with an error-text body,
or a chatty non-title answer. Always read the raw content, not the status.
## Webui password rotation ("Hermex")
Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
on match, 401 on miss. No username field. `/health` answers 200
unauthenticated and `/` answers 302 — **never verify auth via those**; probe
login with a deliberately wrong password (expect 401) and the real one
(expect 200). Details on the webui surface: [[Hermes-WebUI]].
1. Run the whole probe as a heredoc'd bash script
(`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
inline `${…}` even inside single-quoted ssh args; never interpolate curl
`-w %{http_code}` inline.
2. Update BOTH env files with
`sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
- `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
- `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
quadlet's `[Container] EnvironmentFile=` is what the container reads at
create; rotating only the root file leaves the running container serving
the old secret.
3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
env; no deploy needed if the unit file itself didn't change).
4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
client at `http://192.168.0.143:8787` (webui publishes on the LAN
interface since commit `4443897`; off-network ingress is SSH tunnel only —
user `tunnel`, port **22**, key-only auth, forward remote end
localhost:8787).
Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
see rootless containers; run `podman exec hermes-webui printenv
HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
600). If rotating because a secret leaked into transcripts/logs, rotate in
place via a script that fetches the live value from the running container
(`printenv`) instead of transcribing it.
## See Also
- [[Hermes-Dashboard]] — the LAN browser management UI
+172 -13
View File
@@ -24,29 +24,48 @@ https://github.com/open-webui/open-webui
| Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
| Backends | Picker exposes two models — `frank-zealot` (host.containers.internal:7771) and `frank-dragoon` (host.containers.internal:7770, added 2026-10-09). The `frank-archon` (Dankserver) connection remains configured in the DB, disabled — see [The LLM backends](#the-llm-backends) |
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
## The LLM backends
OpenWebUI serves the three fleet LLM endpoints. The model picker is built from
the `model` table in `webui.db` (one row per model), NOT from live connection
discovery — so the picker shows exactly the models seeded there.
The picker is built from the `model` table in `webui.db` (one row per model)
plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
connection discovery — so the picker shows exactly what is seeded there.
As of 2026-10-09 the picker exposes **two models: `frank-zealot` and
`frank-dragoon`**.
| Picker model | Connection URL | Runtime | Host |
|--------------|----------------|---------|------|
| `frank-large` | `http://host.containers.internal:7770/v1` | unsloth llama.cpp (Strix Halo iGPU, 200k ctx) | Dankclaw |
| `frank-medium` | `http://192.168.0.235:7770/v1` | vllm-radiance TP=2 (FP8, DFlash2, 262k ctx) | Dankserver |
| `frank-small` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw |
| Connection (`openai.api_configs` idx) | URL | Enabled | `model_ids` | Runtime | Host |
|---|---|---|---|---|---|
| 0 | `http://host.containers.internal:7770/v1` | yes (2026-10-09) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw |
The `model` table carries one decoration row per picker model:
`frank-zealot` (context length 200,704 = the per-slot window; description:
Qwen3.5-9B Q6_K + BF16 vision mmproj) and `frank-dragoon` (context length
262,144 = the per-slot `HALOGEN_CTX`; description: Swift1.5 v2 abliterated,
MTP, vision ON).
A third gate sits *after* the table, and it is the one that bites family
accounts (2026-10-07): a model row is only visible to a non-admin user when
its **access permission** grants them. A seeded row defaults to admin-only,
so the picker renders "No results found" for every regular user even though
the connection is enabled, `model_ids` is correct, and the endpoint answers
`/v1/models` — and `GET /api/v1/models` returns `[]` for that user while an
admin token returns the model. Fix: set the row's `access_control` to public
(the `access` column on the `model` row) via the admin UI (Admin Panel →
Models → visibility) or the DB. The verification battery below must therefore
run **as a non-admin user**, not only as an admin.
Two gotchas make the wiring non-obvious:
- **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns
`{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the
OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
the llama.cpp endpoints (frank-large, frank-small). The fix is to set
the llama.cpp endpoints (frank-large, frank-secondary). The fix is to set
`model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
builds the model list directly and never fetches `/v1/models`. frank-medium
(vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
@@ -62,7 +81,72 @@ The Ollama connection is disabled (`ollama.enable=false`).
> All three endpoints are bare-metal systemd services managed by this
> repository. Model swaps, context sizing, and spec tuning happen there. See
> the [LLM knowledge base](../LLM/index.md).
> the [LLM knowledge base](../LLM/index.md). Disabling a connection here only
> hides it from this picker — the endpoint itself keeps serving every other
> client (the harness, Hermes, API consumers).
## Container introspection (DB and backend python)
- Working dir for imports: `/app/backend`; package name `open_webui`
(underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
BEFORE importing any `open_webui` module (import-time hard requirement);
the secret lives at `/app/backend/.webui_secret_key`.
- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
module (`podman exec open-webui python3 -c "import sqlite3;
conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config
keys use parameterized LIKE queries (avoids shell quoting).
- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` —
`Models.get_all_models()` is a coroutine (await it); model objects are
pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
`status`/`enabled` attribute — use `is_active` from the DB.
- **API auth for verification:** `create_token` lives in
`open_webui.utils.auth` (NOT `open_webui.utils.utils`);
`get_verified_user` requires an `id` claim (not just `sub`);
`SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
container:
```
podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \
python -c "from open_webui.utils.auth import create_token; \
print(create_token({\"id\":\"<UID>\",\"sub\":\"<UID>\",\"email\":\"…\",\"role\":\"admin\"}))"'
```
(admin UID from the `user` table).
## Verification battery
1. `GET /api/models` (the picker endpoint; response key `data`) → lists
EXACTLY the intended models, **as an admin AND as a `role='user'` token**.
Do NOT gate on `GET /api/v1/models/list` (the Admin Panel → Models page):
on the 0.6.x image it returns `items: []` for connection-backed models
(workspace-list only; measured 2026-10-09 on both picker ids).
2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
with exactly one word: pong") → each returns content `pong`. Reasoning
models need a generous budget: frank-secondary (Qwen3.5 template) returns
empty content + `finish_reason: "length"` at `max_tokens 200` (the
thinking block eats it) and answers `pong` at 1500.
3. A 500 "Server Connection Error" → `podman logs open-webui` for
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
`host.containers.internal`), not a model problem.
**Durability:** the model table + connections are DB-only (not git-managed);
a fresh data volume loses them — including the public access grant above.
For reproducibility, add an idempotent first-start seed script or document
the setup in the service doc.
## API access (auth-gated)
Every `/api/*` endpoint requires an authenticated session (SSO login JWT)
or a per-user API key (`auth.enable_api_keys=true`; keys are minted per
account under Profile → API Keys). Anonymous web visitors get the `pending`
role (`ui.default_user_role=pending`), which cannot chat. Verified
2026-10-07 from loopback `:3005` and over the public vhost:
`GET /api/v1/models/list`, `GET /api/v1/models` and
`POST /api/chat/completions` all return **401** with no key (also with a
bogus bearer token); only `/api/config`, `/api/version` and `/health`
answer publicly, and none of them expose the model list. An external
program needs an account + API key — there is no keyless access.
## Web search
@@ -106,6 +190,18 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
into the existing local account with the same email — the pre-SSO admin
(frank8465@protonmail.com) keeps its data. `ENABLE_OAUTH_SIGNUP=true` lets
other `openwebui_users` create accounts on first login.
- **First login mints role `pending`.** `ui.default_user_role=pending` (config
DB, verified 2026-10-09), so a freshly-provisioned `openwebui_users` account
completes SSO and then cannot chat (picker empty) until its role is flipped:
`UPDATE "user" SET role='user' WHERE email='<user>@franksdank.cloud';` —
run with host `python3` over ssh against
`/home/dank/Containers/open-webui/data/webui.db` (no `sqlite3` CLI on
Dankclaw; table is `user`, singular; no `active` column). No per-user model
grant is needed: both picker models (`frank-zealot`, `frank-dragoon`) carry
the global `access_grant` row `(model, <id>, user, *, read)`, so any
`role='user'` account sees them. Flip on FIRST-LOGIN SIGHTING — the row does
not exist until the account signs in (2026-10-09: five provisioned accounts
had no row).
- **IdP reachability:** the container is on its own bridge (10.89.3.0/24) and
cannot reach `authentik-server:9000` by name. It reaches the IdP over the
public URL `https://auth.franksdank.cloud` — Caddy publishes `0.0.0.0:443`
@@ -147,6 +243,23 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
`podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
## Image generation (ComfyUI)
Image generation rides the local [[ComfyUI]] integration, not the chat
picker — ComfyUI's models never appear in the chat model picker (that is
chat LLMs only), so ComfyUI needs its own image-generation connection
configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
serves `:8188`.
Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
route to `POST /api/v1/images/generations`; the `comfyui` engine is still
upstream) with `Authorization: Bearer <token>` and body
`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
carries base64 image data or a URL.
Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
## Retired instances
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
@@ -158,18 +271,64 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
- The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
- The default admin account is created on first login; secure it immediately.
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
- The on-host LLM services (frank-large `:7770`, frank-small `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`.
- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`.
## History
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-small` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-secondary` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
- **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`.
- **2026-09-24 — flipped to SSO-only.** The OIDC round-trip verified in a real browser (login view renders no password form, only "Continue with authentik", and the flow lands on a signed-in session), so the local username/password path was retired: `ENABLE_LOGIN_FORM=false` + `ENABLE_PASSWORD_AUTH=false` in the quadlet and `ui.enable_login_form` set to `false` in the config DB (the DB wins over env, so both were changed together). `qa-e2e-admin` was demoted out of the `authentik Admins` group (it keeps `openwebui_users` / `homepage-admins` for app access).
- **2026-10-07 — picker narrowed to `frank-secondary` only.** Per Frank: the only model accessible in OpenWebUI is `frank-secondary`. Rebuilt the `model` table to a single `frank-secondary` row (the old rows `frank-large`/`frank-primary`/`frank-small` are deleted), set `openai.api_configs` indices 0 (frank-large) and 1 (frank-primary) to `enable: false`, and retargeted index 2's `model_ids` from the stale `frank-small` label to `frank-secondary` — llama.cpp serves by alias, so the old label addressed a model name the server does not know. Pre-change snapshot: `/home/dank/Containers/open-webui/data/webui.db.bak-20261007` (+ `-wal`/`-shm`). Verified: `/api/v1/models/list` returns exactly `["frank-secondary"]`, chat completion answers `pong` (needs `max_tokens≥1500` — the Qwen3.5 thinking block consumes a 200-token budget), and anonymous API probes are all 401 (see [API access](#api-access-auth-gated)).
- **2026-10-09 — re-keyed to the Protoss ids (wire rename).** The endpoints
began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three
id-keyed objects had to move in the same window as the alias flip:
`model.id`+`name` (`frank-secondary` → `frank-zealot`, `params`
`{"context":{"length":200704}}` and the `meta.description` kept),
`access_grant` (`resource_type='model'`, `resource_id`, principal
`user`/`*`, permission `read`), and `openai.api_configs` per-backend
`model_ids` (index 2 = `:7771` is the enabled family backend; indices 0/1
= `:7770` and Dankserver `:7770` stay `enable: false`). Snapshot first:
`/home/dank/Containers/open-webui/data/webui.db.bak-20261009`. Apply from
inside the container (`podman exec -i open-webui python3 -`, sqlite3; the
image has no sqlite3 binary) then `systemctl --user restart open-webui.service`
(it is a rootless quadlet) to drop the cached model list. Verify in a real
browser as the family QA account: the welcome screen selects
`frank-zealot` and answers. **A stale id fails silently** — the picker
simply lists nothing, because OpenWebUI intersects the backend's
`/v1/models` with `model_ids`.
Law: **the model id is a foreign key in three tables.** Any future alias
change on an endpoint OpenWebUI fronts must ship with this re-key, in the
same deploy.
- **2026-10-09 — composer is `contenteditable`, not `<textarea>`.** A
Playwright probe that waits for `textarea` after SSO times out on the
welcome screen: OpenWebUI 0.6 renders the composer as a contenteditable
div. The stable anchor is the model button (`getByRole("button", {
name: /selected model/i })`) and the welcome text "How can I help you
today?"; that snapshot is also the cheapest proof the picker is populated.
- **2026-10-09 — `frank-dragoon` added to the picker.** Per Frank: expose the
halogen endpoint (service `frank-large`, `host.containers.internal:7770`)
beside `frank-zealot`. Three id-keyed writes (the trio the 10-07 rebuild
proved): `openai.api_configs` idx 0 `enable: true` (its `model_ids` already
carried `frank-dragoon` from the wire rename), a `model` row (`id`/`name`
`frank-dragoon`, `params.context.length` 262144 = the per-slot
`HALOGEN_CTX`, description of the v2 abliterated payload), and the global
`access_grant` `(model, frank-dragoon, user, *, read)`. Snapshot
`webui.db.bak-20261009b`, apply with host `python3` over ssh, then
`systemctl --user restart open-webui.service`. Verified: `/api/models`
lists `frank-dragoon` as admin AND as the QA `role='user'` account, and
`POST /api/chat/completions` through the app answers `pong` at
`max_tokens 1500`. Halogen's `/v1/models` is OpenAI-shaped, so discovery
would have worked too — `model_ids` stays authoritative.
## See Also
- [[Dankclaw Stack]]
@@ -117,6 +117,18 @@ frank-large `health=200` throughout.
the model is loaded. Trust the engine's `host memory left for everything
else` line, not `free`.
- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
snapshots in the swapfile path — check `df -T` before planning, or
`mkswap` will reject the file.
- **Renaming an *active* swapfile works (inode-based) but leaves a stale
path in `/proc/swaps`** — swap it off first, then rename, then `swapon`
(which is exactly what the step-3 consolidation does). Migration is
observable while it runs: in the 2026-09-12 run `swapoff` of 4 GiB took
65 s and used-RAM rose only ~3 GiB (pages touched during migration), the
rest relocating on disk. If used-RAM climbs toward the whole swapped
amount, let it finish rather than aborting.
## Trash
The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at
@@ -0,0 +1,303 @@
---
type: reference
title: Noctalia and Niri
description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
timestamp: 2026-10-05T00:00:00-07:00
---
# Noctalia and Niri
Noctalia (`/usr/bin/noctalia`) is the desktop shell on niri, running on both
the workstation and blankpad. Since the 2026-09-10 unification both run the
**bare `noctalia` v5 binary** (niri `spawn-at-startup "noctalia"`); the old
Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is
retired on both hosts.
## Config layers
- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g.
`config.toml`). Main config: `~/.config/noctalia/config.toml`; the niri
config includes `noctalia.kdl` for display settings.
- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and
**overrides declarative values for the same keys — never hand-edit it**;
UI/IPC writes land there.
- Main reference: `noctalia-reference` facts — IPC `noctalia msg <command>`
replaced the `qs -c noctalia-shell ipc call` wrapper; app templates live at
`/usr/share/noctalia/assets/templates/` (gtk, ghostty, alacritty, hyprland,
sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango,
umbriel, foot, emacs, kitty); palette generation `noctalia theme [image]`.
## Remote IPC over ssh (the D-Bus trap)
`noctalia msg <command>` resolves the running instance via the D-Bus session
bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain
ssh exec prints `error: noctalia is not running` even when the shell IS
running — do not conclude "shell dead" from this error. Working remote
procedure (runs in the systemd user-manager context where the bus address
exists):
```
systemd-run --user --pipe --wait noctalia msg <command> [args]
```
Verified: `msg status` returns live JSON state; unknown panel ids return an
error listing all valid panel ids (clipboard, control-center, launcher,
polkit, session, setup-wizard, test, tray-drawer, wallpaper).
Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
`media next|previous|playPause` (graceful no-op, rc 0, with no active
player), `brightness-up [target] [step]`, `brightness-down`,
`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
`screenshot-fullscreen`.
**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
resolves to XDG Pictures via user-dirs.dirs; an explicit directory key
overrides. Fleet default pattern: point `directory` at the repo-tracked
wallpapers dir so git-tracked wallpapers serve the shell without copying
files. After writing config.toml over ssh the file watcher hot-reloads
automatically; `config-reload` via systemd-run forces it immediately. Note:
the *current* wallpaper stays the built-in default until the next rotation
or an explicit `wallpaper-next`; `wallpaper-get` then reporting a path
inside the configured directory is the confirmation the config took effect.
**Lock chain (unified):** swayidle-lock.service ExecStart on both hosts:
`/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock'
before-sleep '…' after-resume '…'`. `msg` resolves via the session bus, so
no WAYLAND_DISPLAY pinning is needed in unit context.
## Custom-palette recolor (without restarting the session)
Why custom, never the community cache:
`~/.local/state/noctalia/community-palettes/*.json` is a **sync cache** —
`.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches
from `https://api.noctalia.dev/palettes`, so edits there get reverted.
Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
(`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never
synced. Missing/invalid file → silent fallback to the builtin palette.
Palette JSON shape = the community file's shape: `dark`/`light` maps of
`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
*generator* format, not this one. Copy an existing community JSON and mutate
with `jq`.
Procedure:
1. Read the live scheme first — never assume: `noctalia msg color-scheme-get`
(e.g. `community Gruber Darker`); `noctalia msg theme-mode-get`.
2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml
/tmp/noctalia-settings-backup-$(date +%s).toml`.
3. Build the palette from the current one (jq keeps structure):
```sh
mkdir -p ~/.config/noctalia/palettes
jq '.dark.mPrimary = "#f43841"
| .dark.terminal.normal.yellow = "#f43841"
| .dark.terminal.bright.yellow = "#ff4f58"
| .dark.terminal.foreground = "#f43841"' \
"$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \
> ~/.config/noctalia/palettes/"Gruber Red.json"
```
Prefer a hex already present in the palette (its own `mError` red) so the
theme stays internally consistent. Set BOTH `normal` and `bright`
variants, and both `dark` and `light` if the user toggles modes.
4. Switch (persists `theme.source = custom` + `theme.custom_palette`):
`noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
`color-scheme-get`.
5. Re-render every configured template: `noctalia msg templates-apply`.
Regenerates `~/.config/ghostty/themes/noctalia`,
`~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
starship, etc. Symlinked-into-dotfiles paths show up as modified files
worth committing.
6. Reload surfaces: compositor — `niri msg action load-config-file`
(`reload-config` is NOT a valid niri subcommand); ghostty — see the trap
below.
Verification: `ghostty +show-config | grep -E '^(foreground|palette)'`
(proves the theme file resolves — note it prints only the delta from
defaults; cross-check with `ghostty +show-config --default` before
concluding a key is unset); `noctalia config validate` → `✓ Config is
valid`; visual via `noctalia msg screenshot-fullscreen` (saves to
`~/Pictures/screenshot_<ts>.png` and ALSO overwrites the clipboard with the
PNG — warn the user); `grep -c <old-hex> ~/.config/noctalia/palettes/*.json`
→ 0 for no leftovers.
Track it in the dotfiles repo (`~/.omp`, the command-center —
`scripts/deploy.sh` is a stow-equivalent over `dotfiles/<pkg>/` trees):
mirror the shape for the new palette so `git pull` + `deploy.sh` is the
whole propagation story — move the JSON into
`~/.omp/dotfiles/noctalia/.config/noctalia/palettes/` and symlink it back
into `~/.config/noctalia/palettes/`; re-set the custom palette (re-reads
through the symlink). Do the link by hand for one file instead of running
`deploy.sh` — sync mode quarantines and replaces every divergent live file.
Traps:
- **NEVER `kill -USR1` ghostty to reload.** It does not reload config on
this fleet — it kills/respawns the terminal (took out Frank's window
2026-09-13). Correct: the default keybind `ctrl+shift+,` = `reload_config`
(per `ghostty +show-config --default`). Existing windows keep old colors
until reloaded; new windows inherit immediately.
- Palette hex values are NOT a proxy for what the user sees: in
`Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change
the yellow" means the accent too, not just terminal index 3.
- Saturating terminal text makes ANSI 1/9 indistinguishable from default
text (grep highlights flatten) — flag it and offer a lighter red.
- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats
white-on-color here, ~3.9:1 — keep `mOnPrimary` dark).
- Revert = `noctalia msg color-scheme-set community "<original name>"`; the
community cache is untouched by the whole flow.
- Generated theme files symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0`
`noctalia.css`, ghostty theme) show up as tracked modifications after
`templates-apply`; commit them together with the new palette JSON so every
machine gets the same recolor.
## Offline template render (when templates-apply never ran)
Failure signature: niri `failed to read included config from
~/.config/niri/noctalia.kdl: No such file or directory` — the desktop runs
but with DEFAULT config (niri 26.04 falls back, does not exit): no custom
keybinds/rules, no noctalia autostart, default output placement. ghostty:
`theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`.
alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing.
Root cause pattern: upstream dotfiles commits theme switches (niri include,
ghostty `theme = noctalia`, alacritty import) that depend on noctalia
TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from
the local palette) and are intentionally NOT in the repo; on any host where
`templates-apply` never ran, they don't exist.
Fix (offline, works while a session runs) — built-in templates live at
`/usr/share/noctalia/assets/templates/` (declared in `builtin.toml`); render
each needed one with the live palette:
```bash
# niri include (plain host-local file; do NOT symlink into repo — regenerated per host)
noctalia theme --theme-json ~/.config/noctalia/colors.json \
--render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl"
mkdir -p ~/.config/ghostty/themes && \
noctalia theme --theme-json ~/.config/noctalia/colors.json \
--render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia"
mkdir -p ~/.config/alacritty/themes && \
noctalia theme --theme-json ~/.config/noctalia/colors.json \
--render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml"
```
All template tokens (primary, surface, error, primary_container, …) resolve
from the flat 16-key colors.json; container tokens are derived, never blank.
Verify: `niri validate` → rc 0; `ghostty +validate-config` → rc 0; next
login journal shows `loaded config from config.kdl` with NO parse WARN,
custom output placement (`niri msg outputs` vs display.kdl), noctalia
autostarted (`pgrep noctalia`).
Traps: **`noctalia msg templates-apply` on 5.0.1 returns "ok" but applies
NOTHING** (this build's opt-in config schema does not act on it) — do not
trust its exit code, use the explicit `--render` calls above. Niri config
errors at boot are NOT fatal — the compositor keeps running on defaults,
which reads as "some settings silently stopped working", not a crash.
Regenerate after any palette change; the rendered files are static
snapshots.
## Niri output rotation and pinning
Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
**Ground truth first:** `niri msg outputs` (connector names, current mode,
scale, transform, logical pos/size) and `niri --version`. Output **names are
connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
the EDID strings printed before them are decoration. On blankpad the live
config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
repo path (that is what gets committed); check with `ls -ld`.
**Transform direction is counter-clockwise** (`niri msg output <NAME>
transform --help`): `90` = content rotated 90° CCW ⇒ the panel's **top
bezel sits on the LEFT**; `270` = top bezel RIGHT; `180` = upside down;
`flipped-90`/`flipped-270` = mirrored variants. `90` vs `270` is a
physical-stand fact no tool can read (portable monitors expose no
orientation in EDID/MCCS) — pick one, apply, and hand the user the one-liner
to flip; don't stall on the question.
Two traps when editing the config:
1. **An `output "X" { … }` block resets every property you don't restate to
its default.** Pinning position on `eDP-1` without repeating `scale 1.25`
silently drops the laptop panel to scale 1.0. Restate `scale` (and
`mode`) in every block you touch.
2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning
only the externals shoved `eDP-1` from `0,0` to `3936,0`. Pin every
output's position to keep the left-to-right order. Positions are logical
pixels at each output's own scale; x-offsets chain (eDP-1 logical width
1536 → DP-2 at 1536 → +1200 portrait width → HDMI-A-1 at 2736):
```kdl
output "DP-2" { scale 1 transform "90" position x=1536 y=0 }
output "HDMI-A-1" { scale 1 transform "90" position x=2736 y=0 }
output "eDP-1" { scale 1.25 position x=0 y=0 } // MUST restate scale
```
Apply and verify:
```bash
niri validate -c ~/.config/niri/config.kdl # exit 0
niri msg action load-config-file # live reload; NOT reload-config
sleep 1; niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
```
Success looks like `Logical size: 1200x1920` (swapped) plus
`Transform: 90° counter-clockwise` on each rotated panel, positions matching
the intended chain. One-off rotation without touching config (temporary):
`niri msg output DP-2 transform 90`. Commit via the dotfiles repo
(`~/.omp`, `main`; push to the local Gitea permitted): `git pull --ff-only`
→ add the display.kdl → commit → push. Pitfalls: `reload-config` is not a
niri subcommand; never SIGUSR1 a terminal to "reload" (niri config reloads
do not touch ghostty themes); no man page for niri on this host — `niri
<sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
## GPU RGB over I2C (EVGA iCX3)
Turn RGB on/off (or read it back) on an EVGA FTW3 / iCX3-controller GPU
(the 3090 on dankclaw) using the GPU's own nvidia i2c bus. No display
engine, no X, no daemon — the mode is EEPROM-saved and survives reboot.
- **Find the controller:** the nvidia i2c driver exposes the card's lighting
ports as `/dev/i2c-N`; on the 3090 FTW3 that is ports 1,5,6,7,8 →
i2c-1..i2c-5, and only i2c-1 carries a live slave (NVIDIA port 1). Probe
each bus. The iCX3 MCU answers at address **0x2D**. `/dev/i2c-*` are
`root:i2c 660` — run with `sudo` (dank has passwordless sudo).
- **Protocol (raw I2C_RDWR, NOT the SMBus ioctls)** — write the register
byte first, then the payload:
- enable `0xB2` write `04 C6 EB EA 15` (required once before mode writes)
- mode `0xC0` write `09 m0 m1 m2 m3 .. .. .. .. SYNC`; m0..m3 are the four
zone modes, `MODE_OFF = 0x00`; OFF = `09 00 00 00 00 FF FF FF FF 00`
- save `0x90` write `04 9E EB 00 90` (commit to EEPROM — the
reboot-persistence step)
- fw `0xB1` block read (firmware version)
- Read the mode back: write reg `0xC0`, then read 16 bytes; bytes 1..4 are
the zone modes (0x00 = OFF), bytes 6..9 the per-zone LED counts.
- **Verify persistence:** after a reboot, read the mode — all four zone
modes `0x00` means the EEPROM save held (confirmed 2026-09-19 post-reboot).
- **Tool:** a ~90-line I2C_RDWR reader/writer lived at `/tmp/evga_raw.c`
(`evga_raw <bus> mode|off|save|enable|fw`) — `/tmp` is wiped by reboot, so
recreate it if needed; OpenRGB's `EVGAGPUv3Controller.cpp` carries the
same protocol plus the full colour API.
- Gotchas: raw I2C_RDWR only (the iCX3 answers the raw form, not the SMBus
ioctls); the `enable` write must precede the first mode write in a
session; without the `save` write the mode is RAM-only and resets on power
cycle.
## Related
- [[Command Center Dotfiles]] — the stow-equivalent deploy this page links into
- [[Hermes]] — the agent that drove the CDP daemon alongside this desktop
- [[GGUF Compatibility]] — unrelated but shares the "read the file, not the name" lesson
@@ -5,3 +5,115 @@ description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Reada
tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
timestamp: 2026-09-04T00:00:00Z
---
# *arr Stack
The *arr automation stack runs on **FrankBook-Pro** (the Mac mini) as quadlets
inside its podman machine; all egress rides the gluetun WireGuard netns. The
per-service pages ([[Prowlarr]], [[Sonarr]], [[Radarr]], [[Lidarr]],
[[Readarr]], [[qBittorrent]]) carry deployment facts; this page holds the
cross-cutting API and auth-posture procedures.
## API version split (memorize before any *arr HTTP call)
- Sonarr/Radarr: `/api/v3/…`; Lidarr/Readarr/Prowlarr: `/api/v1/…`.
- Prowlarr is port **9696** — 8090 is Jackett.
- **Empty body = endpoint absent**, not a failure: probe endpoints by status
code first; v1 has no `/api/v1/profile` endpoint at all.
- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object
(GET it, patch `settings`, PUT the whole object back). An array body fails
with `Path: $ | BytePositionInLine: 1`.
- API keys live in each container's config:
`podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
They are local-only secrets — they stop being secret the moment a vhost
serves the API unauthenticated.
## Schema-first probing (validators are free documentation)
Send the FULL schema template object with fields filled; v1 validators reject
partials, and each error names the next required field — peel validators one
live call at a time instead of guessing payload shapes:
- `'Fields' must not be empty` → send a full schema copy, not `fields:[]`.
- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
- `DefaultMetadataProfileId must be > 0` → fetch real ids from
`GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
`GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
profiles.
- `targetType null` → configContract not registered in that build (this
Prowlarr build has no Prowlarr-plugin indexer; integration is
Torznab-family only).
- `testingPackageId` in a PUT body downgrades validation failures to
warnings.
- `syncLevel` is enum `ApplicationSyncLevel` — omit it or use the int; a
string is rejected.
**Verification rules:** `POST /downloadclient` returns the COLLECTION, not
the created object — verify via GET. Test endpoints return verdicts in the
BODY at HTTP 200 (`{}` = pass, a list = fail) — judge from bodies, not
status codes. Command POSTs (`MoviesSearch`, `ApplicationSync`) run
server-side and respond only when the work finishes: curl `-m` timeouts
(HTTP 000) during a search are NOT failures — poll queue/history afterward
instead of re-firing; warm small commands (`RefreshMonitoredDownloads`)
answer in ~20 ms, so a hang means live work in progress, not a dead API.
**DB as arbiter:** when endpoint shapes are opaque, stop the container and
edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly,
then start. The indexers table columns are
Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the
enable flag column is `Enable` (not `EnableRss`). Prowlarr's real application
settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
`/api/v1/applications` GET responses carry `fields: []`).
## Auth posture probe (ground truth is behavior, not config flags)
`AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in
`config.xml` is ambiguous on paper; on the wire it means a login screen with
**zero users** — first-account-wins.
1. **Confirm which host runs the service before probing** (`podman ps` on
the target, not the box you ssh'd into first — the arrs live on
FrankBook-Pro).
2. Probe an API endpoint that bypasses the SPA shell, locally and through
any vhost: `curl -s -o /dev/null -w "%{http_code}"
http://<host>:<port>/api/v3/system/status`. **SPA roots (`/`) always
serve a 200 shell — never conclude "open" or "gated" from `/`.**
401 = gated; 200 + JSON = API fully open (anyone on the net can drive it
with curl).
3. For config truth, dump the whole file
(`podman exec <svc> cat /config/config.xml`), not a piped
`grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the
pipeline hides a miss and you conclude "empty field" when the grep itself
failed.
4. Decision framing: when posture = gated-with-zero-users, present it as a
decision menu (None behind the front-door wall vs External/authentikarr),
not a silent change — first-account-wins means whoever logs in first owns
the instance.
## The `None` + `Enabled` auth-config trap (v1 .NET apps)
`<AuthenticationMethod>None</AuthenticationMethod>` +
`<AuthenticationRequired>Enabled</AuthenticationRequired>` is
self-contradictory: the UI Save is permanently disabled, and any save throws
`DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every
page after restart. The valid no-auth representation is **absence of the
`<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix =
delete the key from `/config/config.xml`
(`sed -i "/<AuthenticationRequired>/d"`), restart the container. Tell-tale:
one v1 app boots clean while siblings 500-sprawl → diff their config.xml
auth blocks; the healthy one shows the correct shape. (Prowlarr runs
`Forms` + `Enabled` healthily — the crash combo is `None`+`Enabled` only.)
General rules that saved the session these came from: container
`State=running` ≠ app serving — read `podman logs --tail 12 <c>` before
theorizing; when a UI gate blocks saving, go under it (these apps keep
plaintext XML config on disk — read/edit via `podman exec`).
## See Also
- [[Prowlarr]] — indexer feeds and the v2 sync model
- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] — the managers
- [[qBittorrent]] — download client, 4.6.7 pin, E2E verification
- [[FrankBook-Pro]] — the host and its podman machine
- [[Homepage Status Probes]] — how the e2e gate treats this sleeping host
@@ -111,6 +111,90 @@ Restart=always
probe one ssh-context out returns the residential IP — a false leak alarm.
- Ensure umask is set to `022` so downloaded files are readable by arr apps.
## E2E verification after any pipeline change
Run after ANY change to the torrent pipeline — WireGuard credential rotation,
qB version change, gluetun reconfig, category re-seeding. Proves the full
grab path (tunnel egress → qB download → category save path → clean
removal). All commands from a dev box via `ssh frankbook-pro`; inside the
Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
`podman machine ssh`.
**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
(NOT core — `core` is the podman-machine VM user, reached only via
`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
low MaxAuthTries: when password-bootstrapping use `-o
PreferredAuthentications=password -o PubkeyAuthentication=no
-o IdentitiesOnly=yes` or the key offers exhaust the attempts before the
password is tried.
**1. Egress gate — MUST run inside gluetun:**
```bash
podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
```
Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config); judge by org,
never country alone. NEVER run this one ssh-context out (bare `wget` on the
Mac host): it silently returns the residential Comcast IP and reads as a
hard-gate leak that does not exist. A residential reading ⇒ re-run inside
the container before diagnosing.
**2. Login — generic cookie parse.** The QPW fetch is flaky (~1 in N returns
empty); assert length before use — an empty-password run once produced a
Forbidden loop that cost hours:
```bash
QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
[ ${#QPW} -eq 20 ] || exit 1
SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
http://127.0.0.1:8080/api/v2/auth/login \
| grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
curl -b "$SID" … # pass the WHOLE match
```
WebUI from the LAN: `http://192.168.0.112:8080` (published on the gluetun
unit). Username `admin`; password = the machine-local `QBIT_WEBUI_PASSWORD`
(20 chars, generated once at provisioning by the deploy handler — never in
the repo). Parse-bug 403s never reach qB as failed logins — no ban accrues;
server-side truth is `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log`
(`WebAPI login success/failure` + source IP). Repeated REAL failures accrue
toward qB's 5-failures/3600 s IP ban.
**3. Add a legal test torrent.** Blender open-movie torrents (~300 MB, e.g.
Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file
(`-F "file=@…"`) and `category=Finished Torrents`. Never leave a real-looking
grab behind.
**4. Poll and verify.** Poll `GET /torrents/info` (~20 s): state
`downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading`
at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents`
(qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT
inherit category paths — one more reason for the 4.6.7 pin). Confirm on
disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is
normal for the tunnel.
**5. Cleanup — delete only your artifact:**
```bash
curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
--data-urlencode "hashes=<literal-hash>" \
--data-urlencode "deleteFiles=true" \
http://127.0.0.1:8080/api/v2/torrents/delete # expect 200
```
Target the known hash literally — the user's real grabs share the directory
and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir
gone, all other entries intact.
**Version-coupling constraints:** qB is pinned to 4.6.7 (Readarr's EOL
client rejects qB ≥ 5's `204` login at runtime — see the pin rationale
above). Do not unpin while Readarr exists. After ANY qB version flip,
regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k
iterations) — a 5.x-written hash line makes 4.6.7 reject every login.
Rewrite while qB is stopped.
## See Also
- [[ARR-Stack]] — full *arr stack documentation
@@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
distinguish challenge from deny at this release; grep the body for
`Access Denied` / `Making sure you` instead.
## Guard-chain layer diagnosis (a user reports a guarded page "broken")
Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
"broken / links gone / not loading" and the fault must be localized to
server-side or client-side/session. Chain topology:
`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
→ Caddy :8017 (@home/@data handlers) → app/static`.
Layer-by-layer checks (run from the host via ssh):
1. **Caddy unit + guard containers**: `systemctl --user is-active
caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
'anubis|authentik|caddy'` — expect `Up … (healthy)`.
2. **Anubis challenge page validity**: `curl -sk
https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
— a healthy challenge contains `"anubis_challenge"` JSON with
`difficulty` and the main JS.
3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
-iE 'error|exception|traceback|500|provider'` — routine dramatiq task
logs are normal.
4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
connection refused, which is EXPECTED, not a fault):
`podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
`/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
this container has NO `-H` flag — use `--header "Host: …"`.)
5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
Tool/behavior notes: the anubis container is distroless — `podman exec anubis
sh` fails (`sh: executable not found`); inspect via its logs or the served
challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
podman IP) for ALL clients — rootless-podman published-port SNAT;
clients are indistinguishable by IP and clearance is cookie-based. Do NOT
mistake that for the bug. Homepage render: `app.js` does
`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
failure the catch sets `#summary` to `status unavailable: <err>` and the
grid stays EMPTY — which a user reads as "links gone / not loading." A
30-service status.json with a 9-service services.json renders fine
(stateByName lookup).
**Decision:** if every layer is healthy (challenge valid, outpost clean,
handlers 200, files render), the fault is client-side/session. STOP
thrashing the live system — repeated failed browser challenges share the
masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
symptom (challenge spinning? login loop? empty grid? "status unavailable"
text?) or the authentik login to drive the page in a real browser.
## Verify live
```
@@ -342,6 +342,178 @@ The silent re-auth path was proven end-to-end: after deleting
`qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
state and passed with no login prompt (2 tests, 2.9 s).
## Guard-chain flip recipe (adding a guarded vhost)
Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
client → TLS edge → Anubis :8923 → outpost decision → internal
`http://:8017` listener → Host-matched handler → real upstream.
**Order (never reorder):**
1. **Provider first.** Create the outpost ProxyProvider before touching
routes, or authed traffic hits the outpost error page instead of the app.
Fleet convention for `external_host` is the exact browser-facing origin
(the inventory table carries both bare origins and trailing slashes;
routing matches on hostname, verified 2026-09-02).
`internal_host = http://caddy:8017`, all three flows set, and an
Application attached — the two provisioning rules above.
2. **Route flip in ONE commit/deploy**: the vhost becomes
`reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
inside the `http://:8017` listener add `@<svc> host <vhost>` +
`handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
edge, `:8017` routing table, and provider list change together.
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
**Verification (all three):**
- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
challenge page; grep the body for `challenge|not a bot|anubis` (~8
markers on a live challenge). A 502 here instead of the challenge = chain
mis-wired.
- Post-auth routing proof: grep the LIVE config inside the container —
`podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
after the deploy prints its reload line; confirm the reload actually ran,
don't trust exit 0.
- Outpost reachability: `podman exec authentik-server getent hosts caddy`
resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
000 (connection refused) — the post-auth listener lives in the caddy
container's netns; probe inside it or grep the config. Non-browser clients
cannot solve the JS challenge — before flipping, audit service-to-service
traffic: ingestion/push paths must target internal addresses (`loki:3100`),
never a public hostname of a guarded service. Exempt-by-construction
categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
endpoints, services with their own complete auth — document the exemption
in the vhost comment.
## Driving the ORM (`ak shell`)
All authentik object work happens through the server container's ORM shell —
no repo change, no deploy (these providers are NOT repo-managed; capture new
provider+application pairs as a blueprint or setup script to avoid drift).
```bash
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
from authentik.core.models import User, Group
...
EOF
```
- The heredoc IS the program: `ak shell` reads all of stdin and execs it
(heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
one).
- Bootstrap prints JSON log lines, then
`### authentik shell (2026.8.3)` and `229 objects imported automatically…`
— pipe through `sed -n '/229 objects/,$p'` to drop the noise.
- 229 models are pre-imported. Import paths (each wrong guess costs a failed
exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
`PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
`LDAPProvider` → `authentik.providers.ldap.models`;
`authentik.core.models.application` does NOT exist as an import path; there
is no `authentik.policies.group` module in this build.
- For live DB truth prefer the postgres container
(`podman exec authentik-postgres psql -U authentik -d authentik -At -c
"select * from …"`) — **query with `select *` and stop guessing column
names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
FieldError — `type` moved to subclass tables; the outpost `enabled` column
may not exist under that name; provider pk is `oauth2provider_ptr_id`).
## User provisioning playbook (user-level account)
There is no `User.create_user` classmethod, and `is_superuser` is NOT a
constructor kwarg (it is a property derived from group membership) — passing
it raises `TypeError: User() got unexpected keyword arguments`:
```python
from authentik.core.models import User, Group
u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
u.set_password("...") # password arrives on stdin only, never argv
u.save()
for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
u.groups.add(Group.objects.get(name=g))
```
`path` defaults to `users` (regular). The access map (verified live
2026-10-04): there are NO per-app policy bindings on the proxy providers
(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
access is decided entirely by GROUP MEMBERSHIP consumed downstream —
- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
regex gate) and the homepage admin panel; regular users get only
`category: user` panels from `services.json`.
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
"Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
"Open WebUI roles" property mapping → `roles` claim. Verify live with
`u.app_entitlements(app)`.
- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
(filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
every human account).
- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
debo, freya — kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
and alex created 2026-10-08, artem (pk 16) created 2026-10-07): `username` +
`<user>@franksdank.cloud` + display name +
`is_active=True` + the password set via `set_password`, membership in
`Non-Admin Users` + `jellyfin_users` + `openwebui_users`. `alex` was
recreated from scratch on 2026-10-08 into this shape (the old row had an
empty email and no `openwebui_users`; audit first — the row carried no MFA
device, token, entitlement, or proxy session, and events survive a user
delete because `Event.user` is a denormalized JSON field, not a FK).
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
`openwebui_admins`, `authentik Admins`, `authentik Read-only`.
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
Plex, Seerr, Marreta) need no per-user group: any authenticated user
passes the outpost.
Verify login behaviorally — `authenticate()` is the exact PasswordStage code
path:
```python
from django.contrib.auth import authenticate
assert authenticate(request=None, username="ridley", password="...") # -> User
assert authenticate(request=None, username="ridley", password="nope") is None
```
Look flows up by slug, not name:
`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
can be null (guard with `getattr`); `LDAPProvider` has
`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
(use `type(p).__name__`); `User` has no `.status`.
Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
every outpost session fleet-wide; purge the stale `proxysession` rows
instead (section above).
## Probing the outpost from inside the netns
Rootless podman gives each container its own netns; **host `ss -ltn` cannot
see listeners inside other containers' namespaces** — a missing listener on
the host is an observation artifact, not a fault. Distroless images have no
shell (`podman exec authentik-server sh` fails), so probe by joining the
namespace:
```
podman run --rm --network container:<container> --entrypoint /bin/busybox \
docker.io/library/busybox:latest \
wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
```
- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
triggers globbing (`no matches found`).
- Boot transience: immediately after a restart the same probe can return 503
(configure task pending); poll until the answer changes before concluding.
- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
version — the field is `.HostConfig.NetworkMode`.
## Failure modes
| Symptom | Cause | Fix |
@@ -0,0 +1,158 @@
---
type: reference
title: Homepage Status Probes
description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
tags: [homepage, status, probes, e2e, homepage-status]
timestamp: 2026-10-05T00:00:00-07:00
---
# Homepage Status Probes
The homepage is a static site —
`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
`scripts/deploy-remote.sh` refuses other targets). A checker
(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
each card's target and writes `/srv/homepage/status.json` (mode **0644** so
Caddy can serve it — losing the mode 403s the page).
## The checker's vantage and its three false-down shapes
The checker probes TCP/HTTP **from the host netns**. It only has three states
— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
never render honestly and must be dropped or re-pointed, not "fixed".
1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
card says down while PMS answers `200` over HTTPS. Fix:
`"method":"tcp"` (connect is the honest liveness signal; the public URL
stays HTTP because Caddy re-keys).
2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
compose→quadlet move moved the dashboard to `9119` and the webui to
`8787`; nothing listened on 8090, so a healthy stack read `down`.
3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
`systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
netns-scoped, so from the host these are unreachable *by construction* —
no probe value makes them honest. Drop those cards.
A host-vantage probe can only reach a listener **published to the host**
(rootlessport/pasta); unpublished ports are invisible from the host netns.
See [[Container Host Reachability]] for the netns rules.
## Triage procedure
1. Read the card's probe target in `shared/homepage/services.json`; note
method (`tcp` vs `http`).
2. Does that port have a **host-visible listener**?
`podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
`podman inspect <c> --format '{{json .NetworkSettings.Networks}}
{{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
published nothing → structurally unprobeable from the host).
3. Probe from the **correct vantage**: from a bridge-net container the host
is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
they expand **inside** `podman exec bash -c`.
4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
killer) vs false down (running + unreachable target).
5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
only exposure is an internal netns; never invent a probe target.
6. Verify by re-running the checker path, not by eyeballing: after deploy,
re-read `/srv/homepage/status.json` and diff the `down` list
before/after.
Removing a card's `probe` key without a code change crashes the checker
(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
fix. The `description` field is consumed only by `app.js` `descHTML()`, so
adding a `details` object (role/spec/inputs/use) is schema-safe:
`tests/homepage_config.bats` checks only name/category/section/probe.
## Local render check (before every deploy)
The page fetches `services.json` + `status.json` at load; `file://` is
CORS-blocked, so serve over HTTP:
```bash
cd shared/homepage
python3 -c "import json; s=json.load(open('services.json')); \
json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
python3 -m http.server 8899 --bind 127.0.0.1
```
Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
(panels render after the async fetch; checking immediately returns 0 panels),
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
time) — harmless.
The scratch `status.json` is untracked and NOT git-ignored — move it to
`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
reset so `?bare=1` strips it to black.
## Deploy
```
./scripts/deploy.sh dankclaw --service homepage --skip-e2e
```
Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
datetime), restarts the `homepage-status` timer. Verify live:
`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
no remote wake) times out that test — a pre-existing environmental failure,
not a regression. All non-Mac-mini e2e tests pass.
## e2e raw-IP *arr failures — probe from the browser's box, not a peer
When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
so a healthy probe from dankclaw proves nothing about the **browser's** path:
```bash
# 1. From the box that RUNS the browser:
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
# 2. From a LAN peer:
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
--max-time 6 http://192.168.0.112:$p/; done'
# 3. Liveness (ICMP, not TCP):
ping -c 2 -W 3 192.168.0.112
```
| peer probe | dev-box probe | ping | meaning |
|---|---|---|---|
| 200 | 200 | ok | transient — re-run the spec |
| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
Third row: confirm the Mac is innocent through the dankclaw jump
(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
between the dev box and the Mac.
Don't hold a deploy hostage to a known-environmental red gate: verify the
deployed service **directly** (its own ledger/health/real completion), state
the gate was skipped or partially red and why (`--skip-e2e` prints the
warning, so the skip is recorded), and report the residual gap plainly
("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
probe returns 200; deploy verified by other means"). Do not weaken or skip
the spec — it guards a real SSO-plus-RBAC contract.
## Related
- [[Container Host Reachability]] — the netns rules behind the vantage
- [[Observability Provisioning]] — the monitoring stack reading the same fleet
- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
- [[Testing Policy]] — the seam/e2e contract this page's checks live in
@@ -23,7 +23,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
page (`shared/homepage/docs.html`) — access walkthrough, the running-model
guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
remain a separate thread. The three LLM cards (`frank-large`, `frank-medium`,
`frank-small`) moved from admin-only to the **user** page with live model
`frank-secondary`) moved from admin-only to the **user** page with live model
descriptions and a `docs` link to `/docs#models`.
**Sections + WIP split (2026-09-21):** both pages now group their panels
@@ -36,6 +36,211 @@ deliberately unprobeable) via the `wip` value (`"open"` / `"closed"`).
Tautulli (Plex statistics, `192.168.0.235:8181`) joined the admin page's
Media Library Apps section.
**Amber CRT theme + API docs (2026-10-07):** the palette moved from green
phosphor to amber CRT-AMBIOS (`--fg #ffb000` on `--bg #0a0600`), and the font
stack now leads with **VT323** loaded from the Google Fonts CDN (the deploy
also ships `shared/homepage/img/` beside the page, so `/img/*` serves the docs
screenshots — the post-auth listener serves files only through the `@data`
path allowlist, so new top-level assets must be added there (the `@home`
catch-all answers anything else with `index.html`, i.e. a PNG arrives as
`200 text/html` and renders broken; pinned in `homepage_caddy.bats` — this
exact bug bit the screenshots on 2026-10-08). The docs page gained a full
(`/docs#api-access`): per-user OpenWebUI API keys (User menu → Settings →
Account → API keys), the base URL `https://ai.franksdank.cloud/api/v1`, and
copy-paste configs for **Opencode** (`~/.config/opencode/opencode.json`) and
**Pi** (`~/.pi/agent/models.json`) pointing at `frank-secondary`. Shipped
together with the OpenWebUI side: `features.api_keys=true` for regular users
and the restored `frank-secondary` model row + public `read` access grant
without which the picker showed "No results found" for every family account
(see [[OpenWebUI]]).
**Capability equivalence notes (2026-10-08):** each fleet card (and the
`/docs#models` section) carries a `details.equiv` line — frank-large
"equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
intelligence)", frank-primary "equivalent to Claude Opus 4.8 / OpenAI-Terra
(same era)", frank-secondary "equivalent to last year's frontier level
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
pinned in `homepage_config.bats`.
**Protoss rename + Model Information component (2026-10-08):** the model
cards became the StarCraft trio — `frank-large` → **frank-dragoon**
(iGPU), `frank-primary` → **frank-archon** (Dankserver), `frank-secondary`
→ **frank-zealot** (3090). The endpoint/API ids did NOT change in this wave; only the
family-facing names. The three cards fused into one **Model Information**
component (`app.js` `renderModelInfo()`, `.model-info`/`.mi-*` styles):
Protoss unit art (`/img/unit-<unit>.jpg`, hex-clipped), per-unit accent
colours (Dragoon gold, Archon cyan, Zealot blue), smaller type (15px vs
19px body), and clickable unit-voice lines that play the original
StarCraft `.ogg` quotations from `/sfx/` (fetched from
starcraft.fandom.com via its MediaWiki API; the quote→file mapping lives
in the `model.sfx`/`model.voice.sfx` fields). New `services.json`
per-model fields: `unit`, `tagline`, `sfx[]`, `voice`, `supervision`
(dragoon/archon autonomous-safe, zealot tight-supervision), and `warn`
(uncensored legal note, rendered as a hover tooltip, not wall text).
A compaction-model recommendation for frank-zealot rides in its `use`
text. The Caddy `@data` allowlist grew `/sfx/*`; `deploy_homepage` copies
`sfx/`. A `wip: "down"` badge state ("WIP: DOWN", red) was added; Seerr,
Calibre-Web, RomM, and Jellyfin are WIP: DOWN with their Open links
removed. Card notes added: Copyparty holds every other media app's
content (including the WIP ones); Gitea and Plex need logins separate
from the dankcloud SSO; SearXNG is obfuscated, not anonymous (upstream
API keys tie to Frank). The user docs `/docs#models` section carries the
same renames, api-id mapping, supervision and compaction notes.
**Icons, card polish, admin identity gate (2026-10-08, later wave):** every
app card now carries hex-clipped art beside its name (`services.json` `art`
field, files under `img/apps/`): 19 brand marks tinted amber from the
Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323
initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard),
node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy,
Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model
cards render in the order zealot → dragoon → archon (registry order is the
contract; `homepage_config.bats` pins it). Dragoon and zealot share one
common uncensored legal warning (adult content, harmful/hateful/illegal
descriptions, "sensitive offensive cybersecurity topics", severe legal
risk, "don't ruin this server for everybody"), rendered as the hover
label; archon's ask-Frank rule is a bordered red `.mi-perm` callout.
SearXNG's obfuscation-not-anonymity note moved out of its description into
a card-level `warn` (`.svc-warn` hover label) and the same hover treatment
in the docs. Gitea's note now documents self-service signup — no need to
ask Frank. Footer: docs / admin / account settings are bordered glowing
`.foot-btn` buttons; **the admin button is identity-gated** — Caddy's
`/whoami` echoes the outpost-injected `X-Authentik-Username`/`X-Authentik-Groups`
headers (the outpost strips client-supplied copies, so the echo is
spoof-proof), and `app.js` shows the button only to `homepage-admins`
members. The `/admin*` 403 stays the real gate; the e2e specs pin both
sides (standard user: hidden + 403; admin: visible). Docs were swept for
naming consistency: Protoss names primary everywhere, bracketed api ids
at the model headings, and a note that a future switchover renames the
ids themselves (OpenWebUI picker + raw API + models.yml all move together
— not done yet).
**Icons v2, clickable tiles, header nav (2026-10-08, third wave):** five
icons replaced on request — Copyparty gets an improvised amber folder,
Marreta a crumbling wall (bricks + red crack), Calibre-Web a book,
OpenWebUI its official logo (`openwebui.com/logo.png`), Seerr the official
`seerr.dev/os_logo_filled.svg`. Tiles with an open link are now clickable
whole (`.panel.clickable`, hover lift + glow; clicks on inner links pass
through). Any panel whose live state is DOWN gets a red gradient wash +
slow pulsing glow (`.panel.down`, and `.mi-card:has(.badge.down)` for
model cards). docs/admin/account-settings moved from the footer to the
header (`nav.top-links`). The zealot compaction advice moved out of the
Use paragraph into a bordered glowing `model.compaction` callout
(`.mi-comp`), pinned by the component seam test.
**Protoss theme, brand-color icons, portraits (2026-10-08, fourth wave):**
the full-page angular treatment: chamfered 14 px-cut panels and model
cards, hex-lattice + amber-glow backdrop (`body::after`), scanline
overlay (`body::before`), diamond-glyph section titles with gradient
underlines, the gradient MODEL INFORMATION gateway, a blinking cursor
beside the title, amber scrollbars, cyan text selection, cyan hover glow
on clickable tiles, and a red pulsing glow on down tiles/cards (the
`data-state="down"` attribute `applyStatus` now sets). Icons recut to
official brand colors (Simple Icons CDN): SearXNG, Gitea, Plex, Jellyfin,
Sonarr, Radarr, Lidarr, Readarr, qBittorrent, Prowlarr, Seerr, OpenWebUI,
Calibre-Web (amber glyph on amber tile). Handcrafted monograms carry
per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki
indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red,
Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow,
Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service
fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`),
`warn` on OpenWebUI (searches ride the Seerr→SearXNG path),
`model.compaction` (the amber "set this as compaction model" callout
frank-zealot carries). frank-dragoon's card portrait is the SC2 unit
portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole
effect layer is stripped under `data-bare="1"` (the e2e bare page).
Unit: `tests/homepage_config.bats` (referenced art files exist, model
blocks well-formed, deployer ships the `img/` tree).
**Docs reorder + model source links (2026-10-08, fifth wave):** the docs
page section order is now getting-in → troubleshooting → media → chat →
models → api-access (non-AI first, AI last). The *arr stack (Prowlarr,
Sonarr, Radarr, Lidarr, Readarr, qBittorrent) is out of the family docs
entirely — admin tooling; Seerr is presented as the front door for
requests. Each fleet model gained a Hugging Face source link: card field
`model.hf` rendered as the cyan `model on huggingface ↗` line under the
role, and the docs `Model:` bullets link the checkpoint names (zealot →
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive, dragoon →
Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, archon →
Qwen/Qwen3.8-Flash-Next-FP8).
**Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth
wave):** the amber CRT palette is retired. The page is now deep space
navy (`#050d16`) with warp-blue/psychic-cyan accents (`--proto-blue
#3fa9ff`, `--proto-cyan #7fe7ff`) — StarCraft Protoss console colours.
Cornering went maximal: every panel, model card, badge, button, callout
and the MODEL INFORMATION gateway are octagon-cut (all four corners
chamfered, 18–24 px), and panels/model cards gained a second inset
octagonal frame line (`::before`) — a double-bevel gateway look. Section
titles are chamfered plates with the diamond glyph inside the plate.
Fonts: body is **Share Tech Mono** (legible retro terminal), headings
are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all
three pages load the pair from Google Fonts. Four tiles that had
reverted to initials got pictograms: Calibre-Web an open book (orange),
RomM a purple GameBoy, Copyparty a blue folder with upload arrow,
Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`).
frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching
the new palette. Bare mode strips the frames, octagon cuts and fonts.
**Simplified console (2026-10-08, seventh wave):** "too busy — clean up
and simplify/solidify." The sixth-wave effect stack was stripped to
essentials. Removed: the scanline overlay (`body::before`), the hex
lattice + twin radial backdrop (one quiet pylon glow above the horizon
remains), the panel/model-card inner frame lines, all ambient text
glows (body, title, model names, callouts), the blinking title cursor,
the gradient section-title plates (headings are plain Rajdhani with a
hollow diamond glyph + steel underline), the gradient `border-image`
header rule (solid 2px warp-blue), the pulsing down-state animation
(down tiles and cards now show a STATIC red border + tint), and the
button/hover glows. Kept: octagon-cut panels (softer 14 px), unit-color
model cards, the pictograms, and the fonts. Motion is reserved for
state — the only animation left is the speaking-voice pulse. The result
is flat navy panels on crisp 1px borders: same Protoss direction, half
the pixels.
**Model ids official (2026-10-08):** the card ids `frank-zealot`,
`frank-dragoon`, `frank-archon` are now the harness's official model ids
in `models.yml`/`config.yml` (see [[OMP Harness]]); the
systemd services keep the `frank-primary`/`frank-large`/
`frank-secondary` names. Homepage, docs page and harness now agree on
one name per model.
**Api ids renamed on the wire (2026-10-09):** the promise in
`docs.html` is kept — the served model ids are now `frank-zealot`,
`frank-dragoon`, `frank-archon`, so the docs page no longer carries
`api id:` annotations, and the curl/opencode/Pi examples send
`frank-zealot`. The systemd services keep the old names. `docs.html`
names the fleet trio once each; `homepage_config.bats` pins the card
ids, which did not change.
**Console polish (2026-10-08, eighth wave):** five fixes — octagon
clip-paths replaced with `border-radius` (the clipped corners read as
gaps in the 1px border); section-title diamonds moved from absolute
positioning to a fixed-width inline glyph (aligned, wrap-safe); closed
WIP items grey out (dim type, desaturated art, muted border via
`:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track
(`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms
beat; the two SC2 source oggs are trashed); privacy notes became
**`<details>` disclosure chips** ("⚠ privacy note ▸") pinned to the
bottom of each box (`margin-top:auto` in a flex-column panel/card) —
collapsed by default, tap-to-expand, the touch-safe replacement for the
unreachable hover tooltips. The panel click-to-open handler ignores
`.svc-warn` taps so the chip never navigates. Same chips on the docs
page (privacy + uncensored-model warnings). The API-access section was
reblessed against the live Open WebUI 0.6 UI: steps now match the real
Show/copy/rotate flow and `img/openwebui-api-keys.png` is a fresh
screenshot (old shot trashed). The troubleshooting DNS bullet dropped
its Pi-hole specifics — family members off the LAN cannot use
house-DNS advice; the bullet now just says the names resolve on the
home network.
**Archon card runs the fleet's own model (2026-10-08):** the
`frank-archon` homepage card and docs Model bullet now say the payload
is our in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next and link
our HF publication (`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)
in place of the FP8-era base-model link; capacity line says three
submissions (`--max-num-seqs 3`).
## Design decision — 2026-08-22
The live up/down check runs server-side. A checker on Dankclaw probes each backend and writes a same-origin `status.json`. The page fetches only that file and renders the badges. The browser never calls a backend directly. The checker runs as a systemd timer every 3 minutes.
@@ -111,7 +316,7 @@ flowchart TD
## Service classification
**User services** (root page), in section order:
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-small — status + docs link, no Open link).
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-primary, frank-secondary — status + docs link, no Open link).
- **Web Apps** — SearXNG, Gitea, Copyparty, Marreta.
- **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr.
@@ -131,7 +336,10 @@ shared/homepage/
├── style.css # full theme; a toggle strips to bare HTML
├── app.js # loads status.json once, renders the panels
├── services.json # service config (user + admin)
└── fonts/ # Terminus (woff2) for @font-face
├── img/ # served at /img: docs screenshots, unit portraits
│ ├── apps/ # (unit-*.jpg) and app card icons (apps/*.svg)
├── sfx/ # StarCraft unit voice .ogg files served at /sfx
└── fonts/ # optional self-hosted fonts (deployed with the page; VT323 comes from the Google Fonts CDN, this dir is absent)
```
`status.json` is generated at runtime in `/srv/homepage/`. It is not committed.
@@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the
| HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
| Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
## Per-client fix: resolver drop-in + browser DoH bypass
Symptom: a LAN machine gets `connection timeout` (curl rc=28) on
`https://*.franksdank.cloud` while the server is healthy. Root cause: the
public DNS hands out the WAN IP `71.227.227.172`, which has **no 443
port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN
answer → SYN to a filtered port → timeout. There are TWO resolver layers to
check, in order:
**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in
sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route
servers:
```bash
dig +short www.franksdank.cloud @192.168.0.143 # expect 192.168.0.143
dig +short www.franksdank.cloud @1.1.1.1 # expect 71.227.227.172 (the trap)
getent hosts www.franksdank.cloud # what the OS actually resolves
curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \
https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n' # 200 = server fine
sudo mkdir -p /etc/systemd/resolved.conf.d
sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
[Resolve]
DNS=192.168.0.143
Domains=~franksdank.cloud ~franksdank.casa
EOF
sudo systemctl restart systemd-resolved && resolvectl flush-caches
```
The `~` (route-only) prefix is load-bearing: it routes only those domains to
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
Layer 1 is verified but the BROWSER still times out, the browser is doing
its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR
provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it:
`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead
end). Diagnosis details:
- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT
`~/.mozilla/firefox/` — find it via the running process if the default
path is empty.
- `grep network.trr. <profile>/prefs.js` — **no entries = shipped default,
and since Firefox 113 the shipped default is DoH ON.**
Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy
& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** →
`franksdank.cloud` (and `franksdank.casa`). The exception list = domains
that do NOT use DoH; they fall back to the system resolver, which has the
route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome:
Settings → Privacy and security → Security → "Use secure DNS" → off, or
"Use the same settings as your system". Do NOT hand-edit `prefs.js` while
the browser is running — it holds prefs in memory and rewrites the file on
exit, clobbering the edit.
This is a per-client fix: every LAN client that must reach the homelab by
name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not
bypass it with DoH. The server side needs no change. Verified on blankpad
2026-09-21: the drop-in was present since 09-12, curl returned 200, and
Firefox 155 with default (ON) DoH was the remaining hole.
## Porkbun zone ops (franksdank.cloud)
Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so
the API key/secret sourced from `/home/dank/serverAppConfigs/.env`
(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe
scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' <
/tmp/script.sh`.
Endpoints (all POST, JSON body):
- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud`
— auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong
here). Response shape is **top-level `.records[]`**, NOT
`.content.records[]` as some docs claim — parse with
`jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
`type`, `content`, `ttl`, `prio`).
Quirks that bite (verified live 2026-08-31):
1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud`
yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare
label (`ai`) and the API appends the zone itself. Verify after every
rename with a retrieve before declaring done.
2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of
rapid mutations makes some calls return `<!DOCTYPE html>...` where jq
then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do
not assume the call succeeded — sleep ~8 s and retry the failed ids;
verify by re-listing the zone at the end.
3. Deletes/renames are per-record-id: always retrieve first, map name→id,
then mutate by id.
Rename procedure (vhost rename = DNS + Caddy together):
1. `retrieve` → collect ids for the renamed names.
2. `edit` each id with the new **relative** label, preserving
type/content/ttl.
3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the
Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test
fails the suite if any Caddyfile vhost lacks an addn-hosts A record on
192.168.0.143.
5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed
paths). Capture the suite's true exit with
`out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit
code behind `tail`'s own status and has shipped untested commits here.
6. Verify live: `dig +short <name> @192.168.0.143` then
`curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302;
first hit triggers dns-01 issuance, which works without inbound port 80).
## See Also
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
@@ -0,0 +1,95 @@
---
type: reference
title: Observability Provisioning
description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
tags: [observability, prometheus, grafana, provisioning, deployment]
timestamp: 2026-10-05T00:00:00-07:00
---
# Observability Provisioning
The monitoring stack is **provisioned as code**: dashboards, datasources,
scrape jobs and alert rules live in the repo and ride the normal deploy flow.
No hand-edits inside containers — they are wiped on recreate and never
committed.
## The config triad
| Repo path | What it holds |
|---|---|
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
Targets and datasources reference **container names** (`loki:3100`,
`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
are host-netns services, not containers. Prometheus 3 refuses HTML responses
as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
exporter (see [[Stale Signal Audit]]).
## The deploy path
`scripts/deploy-remote.sh` carries two copier functions:
- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
`shared/grafana/**` to the `prometheus`/`grafana` services, so
`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
exist, the copier functions exist, and the provisioned paths match the units'
mounts.
## Entry points
**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
vantage first: container name if the exporter is a container on the same
netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
if it is an HTTP probe, not a metrics endpoint.
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
Rules reload with the prometheus service.
**Add a datasource** — drop a YAML file into
`shared/grafana/provisioning/datasources/`. Point it at the container name of
the backend (`http://prometheus:9090`), not loopback.
**Add a dashboard** — drop the JSON into
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
unified storage, not (only) the SQLite table — grep the API, not the DB, when
verifying (see the verify section of [[Fleet Audit and Posture]]).
## Verify through the APIs, not the filesystem
```bash
# prometheus: rules and targets actually loaded
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
# grafana: datasource + dashboard provisioned
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
```
Health answers from a stale process and provisioned resources can hide in
unified storage — a file existing on disk is not proof the running server
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
table (refused = outage, 404/HTML = config defect).
## Current delivered state (2026-09-17)
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
via `host.containers.internal`) → signal-cli. Some targets read down for
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
the blackbox exporter. See [[Observability]].
## Related
- [[Observability]] — stack topology and the alert delivery chain
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
- [[Quadlet Lifecycle]] — the units that host these services
- [[Deploy State Triage]] — how the planner selects these services
@@ -38,7 +38,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
| `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
| `frank-large`, `frank-small` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
**Three jobs read down, and they are not outages** (audited 2026-09-16):

Some files were not shown because too many files have changed in this diff Show More