Compare commits

...
Author SHA1 Message Date
Frank 1c56dea67a fix(qwen38): record at MNBT 1,024 — the step size the cache can actually pay for
The rerun after 96c5a06 refused every corpus document again ("document 196 is
8507 tokens and the KV pool holds 7436"), and the boot table shows why the
host-pool change could not have helped: vram_budget.cpp put the container in its
`no lender` branch — elastic 25.56 GiB, experts 25.29 GiB (their plan-fixed
share), kv cache 271.25 MiB — identical at 12,288 and at 49,152 MiB of host
pool. The only supplier left there is the activation arena, and the engine says
so at vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena scales
with it)". The log prices both rungs directly (a 2,048-token step costs 1.08
GiB, a 1,024-token step 504.39 MiB), so the default moves to 1,024: the cache
grows to 866 MiB / 24,300 tokens and scores every one of the 197 documents
(longest 11,970). Calibration keeps its wide 65,536; it was never tied to MNBT.

This also retracts the mechanism 96c5a06 recorded in the header ("the pool buys
KV as well as residency"). Inheriting the env's 49,152 / 96 stays right as
single-source-of-truth, but on this payload the pool is not the cache's supplier.
2026-10-27 22:19:51 -07:00
Frank 96c5a0625c fix(qwen38): the rig's card split is inherited, not invented
run-qwen38-bf16.sh carried its own --host-pool-mib 12288 / --gpu-headroom-mib 768,
copied from the w4-era swift15 rig. The engine derives the whole per-card split from
those two numbers, and the KV cache's funding is `elastic - expert_floor` where the
expert floor is a function of the pinned host pool — so the pool buys cache as well
as residency. The first kldref boot got a 270 MiB / 7,436-position pool and the
recorder refused all 197 corpus documents ("document 0 is 10554 tokens and the KV
pool holds 7436"), exiting 1.

The rig now sources FRANK_ARCHON_HOST_POOL_MIB / FRANK_ARCHON_GPU_HEADROOM_MIB from
scripts/frank-archon.env, the file that owns them (49152 / 96), with `:?` guards, and
the header records the law instead of the claim that the pool never touches logits.
tests/qwen38_reference.bats pins the shape verbatim, so its SHAPE contract moves with
it; the suite is green (558 tests).
2026-10-27 21:54:19 -07:00
Frank a7329d190f feat(qwen38): the BF16 reference instrument - build it, drain once, record logits
Two tools the KLD gate depends on.

build-qwen38-bf16-rad.sh converts the checkpoint to a BF16 .rad with NO recipe
line, so every tensor stays bf16 - the reference must not carry the quantization
it is there to measure. It gates before writing 335 GiB: the checkpoint present
and fully verified by the fetcher's own verifier, the peak-disk bill (350,000
MiB: the checkpoint's 343,323 plus margin, because the .rad is built beside it),
an existing output retired to ~/.AGENTTRASH by --force instead of deleted, and
--plan-only before the real convert. --max-ctx 32768 is a declare-time check, not
a weight change: dropped, the resolver is handed the checkpoint's 262,144 window,
which at fp8 cache is 258.8 GiB on a 192 GiB machine.

run-qwen38-bf16.sh serves that container in three modes - kldref (record the
reference logits), score (record an arm's logits against that reference), calib
(open the Gram tap for the imatrix). It drains frank-archon and restores it on
every exit path, and the two logits modes ride ONE step size
(FRANK_QWEN38_MNBT, default 2048) and ONE window (FRANK_QWEN38_MAX_MODEL_LEN,
default 32768) so a reference and an arm cannot be measured at different shapes:
that is the whole validity of the KL number. Four slots over 2,048 matches
frank-archon.env's operating point (MAX_SEQS=4, MAX_BATCHED_TOKENS=2048); the
smaller host pool (12,288 vs 49,152) buys residency, which moves speed, not
logits. calib keeps 65,536 - decode-bound serving, never the gate.

kldref refuses before the drain when the corpus's longest record cannot fit the
declared window (limit = window x 3 chars, the pessimistic tokenizer bound; the
measured corpus is 38,263 chars, 2.6x inside 32,768). It runs on --dry-run too,
so the check is visible before a 20-minute drain is spent. 12 seam tests, each
red-proofed by neutralising the gate it pins.
2026-10-27 19:40:48 -07:00
Frank 5fe04e569f test(qwen38): campaign seam for fetch, protected plane, arms and gate
32 black-box tests over the four tools the campaign runs, driven through their
own argv with fixture corpora - no GPU, no container, no rad-convert. Each test
is an output an operator reads: the verifier's MISSING line and exit, the byte
bill the derivation lands, the refusal text, the plan and the convert the driver
prints per arm, the interval verdict the gate returns.

Includes the planner seam the swift15 campaign lacked: the campaign's paths plan
NO deployable service, so merging this toolchain can never drain frank-archon.
The driver fixture carries four grams so --min-grams has a thin-pool control,
and the arm-driver invocation passes env assignments positionally after the
driver's own flags, so the seam exercises the real call shape.
2026-10-27 19:36:58 -07:00
Frank bb027265db feat(qwen38): the FP8-anchored KLD gate - the served payload is the floor, not BF16
The swift15 gate compared an arm against the BF16 reference and passed on
KL <= 0.03. That bar is now the wrong instrument: the fleet already ships the
publisher's FP8-native iq4r-moe payload, so an arm that beats BF16 but loses to
what is running is a downgrade. The gate takes three recordings - fp8 anchor,
iq4r anchor, candidate - and requires the candidate's KL strictly inside the
interval the two anchors define, at every slice, plus top-1 agreement no worse
than the served payload.

The anchors must be ordered (fp8 better than iq4r) or the run stops: an
unordered pair means the instrument or the recording is wrong, not that the arm
lost, and one number out of the report is named rather than guessed at.
--recover prints the tightened numbers the same way it prints the failing ones,
so the operator sees the bar they would have had to clear.
2026-10-27 19:36:53 -07:00
Frank 4e551fafd7 feat(qwen38): the w5nl64 recipe template and the three-arm convert driver
One template (every routed expert at five bits, E4M3 scales in 64-groups, the
top level rotated through the 128-wide Hadamard chosen by GPTQ against CALIB)
plus a driver that generates the per-arm expert row, so the other 60 lines can
never drift between arms: control (cd=3), cd=6, and act_order=1 with NO cd= -
libquant refuses the pair (lq_registry.cpp:105), so the AoA arm is its own
sweep, not a fourth column.

The header carries the campaign's order (fetch -> build reference -> one drained
window for the calibration tap and the KLD reference -> trash the bf16 container
-> imatrix -> arms -> fetch the FP8 anchor last -> gate -> trio edit), because
that sequence is point-of-use for whoever runs it next, and the disk sequencing
is forced: 754 GiB free does not hold checkpoint + reference + three arms.

Gates that bite before 5h13m of convert time is spent: the BF16 reference still
occupying the volume, a recording too thin to be a calibration (converting
uncalibrated is silently worse and only whispers on stderr), an imatrix outside
the mounted campaign dir, an arm that already exists, a protected derivation
that cannot land its target, and a destination that cannot hold the output. Each
arm runs --plan-only first so the engine's own fit check bills at plan time; the
served container's argv shape is what the arms declare.
2026-10-27 19:36:53 -07:00
Frank 3d8d18a45c feat(qwen38): fold the calibration tap's Grams into libquant's imatrix GGUF
rad-convert's GPTQ path reads importance from an imatrix GGUF; the reference
container's --kld-record pass emits per-expert and per-hidden Grams as raw
recordings. This folds them into one v3 file: a single expert-major f32 entry
per MoE layer plus the hidden-width entries, every data offset a multiple of
general.alignment as the loader requires.

Refusals, each one a silent-worse outcome the swift15 campaign hit: an expert
with no gram of its own carries the layer's pooled diagonal and the count is
reported (a silent zero is a down-quantized expert nobody priced); expert grams
with no pooled gram, a gram short by one float (names the file), the same global
expert recorded twice with different numbers, an empty recording directory, and
an existing output. --min-rows sets the noise floor below which a recording is
dropped. rows is u64 so a corpus that outgrows u32 survives the header instead
of wrapping.
2026-10-27 19:36:38 -07:00
Frank c13a2568d4 feat(qwen38): size the protected bf16 expert plane by byte bill, not by ratio
The swift15 rule (top 7% by imatrix energy, bar-only) left the plane size to the
distribution. The w5nl64 arm trades a *pinned host pool* against KL, so the
operator needs to say how many GiB of bf16 experts the pool can afford and have
the set land there: --target-count walks the ranked candidates until the byte
bill matches, keeps the bar-only pass for experts that are disproportionate for
their size, and reports the pool's ceiling instead of padding when the count
cannot be reached.

TP=2 cuts the expert plane into two 256-expert rank halves, and an odd count in
either half unbalances the placement, so the parity audit is a refusal (exit 2,
'PARITY LAW BROKEN'), not a silent fix-up: the driver stops rather than
converting a payload whose expert placement no one priced.
2026-10-27 19:36:32 -07:00
Frank e7a835e41f feat(qwen38): fetch and verify the vendor Qwen3.8-Flash-Next BF16 checkpoint
The w5nl64 campaign measures against the vendor checkpoint, not the Swift-1.5
finetune: the served payload (qwen3.8-next-flash-fp8-iq4r-moe) is quantized from
this base, so fidelity must be scored against the same reference. 335.3 GiB
across 10 shards.

Verifier gates on the index's own shard list, each shard's declared byte count,
and the Hub's declared total - a curl without -f writes the 401 body as the
'shard' and reports success. --fetch refuses when the destination cannot hold
the checkpoint, because a 335 GiB download that fills the disk mid-shard costs
the whole campaign window.
2026-10-27 19:36:26 -07:00
dank 0ecfe830f7 Merge pull request 'fix(deploy): relay firewall DROP must be appended, not -I-inserted above its ACCEPTs' (#75) from fix/signal-relay-fw-order into main 2026-10-21 16:59:16 -07:00
Frank ce370a5b28 fix(deploy): relay firewall DROP must be appended, not -I-inserted above its ACCEPTs
iptables -I prepends, so the install block's final -I of the blanket DROP
landed it at position 1, above all three ACCEPTs: every packet to
18093/tcp died there (live 2026-10-21: 159 packets on the DROP, 0 on each
ACCEPT). The host's own health probe timed out beside a LISTENing socket
and the whole dankclaw drain aborted at the alertmanager gate twice.

Install now rebuilds the tail: delete any blanket DROP wherever it sits,
keep the ACCEPTs at the head, append the DROP last - converges on hosts
poisoned by the old order. New bats case runs install_signal_relay for
real against an iptables sandbox seeded with the poisoned order (red on
the old code, green on this one).
2026-10-21 16:52:40 -07:00
Frank bfe4d185bd docs(vault): OpenWebUI picker opened to all three models for every account (2026-10-21)
Per Frank, frank-archon is public alongside dragoon/zealot: three-leg DB
trio (conn enable, model row ctx 200000, access_grant principal='*'),
verified as role=user. Token-mint traps recorded (cwd, PID-1 secret).
2026-10-21 16:48:32 -07:00
dank 94a9f3aa31 Merge pull request 'feat(security): digest-pin every registry image; drop batch-killing HostAddr' (#74) from security/quadlet-digest-pins into main 2026-10-21 16:32:44 -07:00
Frank d2810ab610 feat(security): digest-pin every registry image on all three hosts (fleet law)
Closing the command-center security queue #70/#71/#72/#73/#74: rolling tags
let a rebuild move under a running service with nobody approving it, and
make a CVE audit ungradeable. 36 quadlet units across dankclaw, dankserver
and frankbook-pro now carry tag@sha256; localhost/* builds stay exempt
(their build is the pin). The pihole and copyparty pins pull the fixed
builds; valkey 9.1.2 (already running) and redis 7.4.11 are pinned in
place, vendor fixes TBD.

New gate tests/quadlet_image_pins.bats: no unpinned registry image, no
generator-rejected key anywhere. Three seams that pinned the old exact-tag
law (frankbook readarr, open_webui, seerr) now require tag AND digest -
strictly stronger. Quadlet Lifecycle docs: batch-abort law, entrypoint
flags-only law, pin law.
2026-10-21 16:32:36 -07:00
Frank 7a8cc83385 fix(quadlet): drop HostAddr from prometheus.unit - podman 5.7.0 rejects the key and the quadlet generator then aborts the ENTIRE user batch, silently orphaning every other unit (prometheus.service vanished while the container ran; deploy died 'Unit prometheus.service not found', live 2026-10-21). host.containers.internal resolves natively; the key was never needed. 2026-10-21 16:32:36 -07:00
dank c4623f1629 Merge pull request 'fix(blackbox): flags-only Exec; bind loopback+LAN; exec-form HealthCmd' (#73) from fix/blackbox-entrypoint-bind into main 2026-10-21 16:25:32 -07:00
Frank 065a6911f9 fix(blackbox): Exec carries flags only (image has ENTRYPOINT) and the exporter binds loopback+LAN
Three stacked lessons, each proved live 2026-10-21:
- the image's ENTRYPOINT is /bin/blackbox_exporter, so Exec= repeating the
  binary made it a positional arg: 'unexpected /bin/blackbox_exporter'.
- the podman gateway 169.254.1.2 is NOT bindable from the host netns - it
  exists only inside each rootless netns: 'bind: cannot assign requested
  address'. The scrape path does not need it bound: a bridge container
  dialing the gateway lands on the host LOOPBACK (proved 200 from the
  prometheus container), so the exporter binds 127.0.0.1 + LAN.
- HealthCmd shell form ('/bin/sh -c "..."') is passed as one quoted token
  and quadlet folds the following --health-interval/--health-retries (and
  the image) into the command string; the plain exec form the sibling units
  use is correct.

Prometheus.md bind law corrected (gateway is a forward-to-loopback, not a
bind address); deploy-remote gate stays loopback, which is now honest.
2026-10-21 16:04:53 -07:00
dank 745d66d60d Merge pull request 'fix(blackbox): --web.listen-address (repeatable), not --listen-address' (#72) from fix/blackbox-web-listen-flag into main 2026-10-21 15:58:36 -07:00
Frank 4009f118c1 fix(blackbox): the flag is --web.listen-address, repeatable, not --listen-address
Third boot failure: the exporter exited 100 ms in with 'unknown long
flag --listen-address' (proved against the pinned image, plus --help).
The bind semantics were right; the flag name was invented. --web.listen-address
is repeatable - one per address - so the dual gateway+LAN bind stands.
Proven again pre-deploy with the real argv: the run starts and dies
only on the deliberate /dev/null placeholder config.
2026-10-21 15:55:21 -07:00
dank 7de343c171 Merge pull request 'fix(blackbox): complete the unit - Image, Network=host, honest bind topology' (#71) from fix/blackbox-unit-generation into main 2026-10-21 15:52:30 -07:00
Frank 139a3cd4fa fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it
The deploy died at 'Unit blackbox-exporter.service not found': a
.container without Image= is not generated at all, and the header
argued for Network=host it never declared. Completes the unit (digest
pinned prom/blackbox-exporter, Network=host) and fixes the bind
topology the first cut got wrong:

- Prometheus is bridge-networked and the netavark hairpin gap bars it
  from its own host's LAN IP, so a LAN-only prober bind is unscrapable
  and a loopback-only bind was claimed to ride the gateway on faith.
  The prober now binds 9115 on the podman gateway (scrape path) AND
  the LAN (healthcheck/diagnostics); the listen_address line leaves
  blackbox.yml so config and Exec cannot disagree.
- comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188:
  bridge container names do not resolve in the host netns; the
  published 0.0.0.0:8188 answers on loopback there.
- New seam: every observability unit must declare Image= (red on the
  shipped file by name); copier test pinned module presence instead
  of the removed listen line. Docs updated to the real topology.
2026-10-21 15:41:37 -07:00
dank 5ffbf19fb2 Merge pull request 'fix(deploy): blackbox-exporter was gated out of its own deploy path' (#70) from fix/blackbox-service-gates into main 2026-10-21 15:34:49 -07:00
Frank 1fd98cab3f fix(deploy): blackbox-exporter was gated out of its own deploy path
PR #69 wired the copier, planner entry and health gate but missed the
three dispatch gates: the client valid_service, the remote
service_is_allowed, and deploy_service's case list. The client gate
rejected --service blackbox-exporter outright; even past it the
dispatcher's catch-all hard-fails on dankclaw (no compose since
2026-09-16). New seam pins the class: every dankclaw service the
remote gate allows must carry a deploy_service case - proven red by
planting a gated, undispatched name.
2026-10-21 14:59:47 -07:00
dank d61c19d41f Merge pull request 'Observability: kill every structural error (blackbox, labels, gates)' (#69) from fix/observability-errors-gone into main 2026-10-21 14:53:16 -07:00
Frank e6fcf217b4 docs(comfyui): the GTT aperture is not free RAM beside frank-dragoon
Measured 2026-10-21 with frank-dragoon live: MemAvailable 15.3 GiB,
iGPU busy 1.75 GiB. The Qwen-Image weight sets (13.3-14.2 GB) do not
fit that budget, and the page's 122 GB /system_stats figure is the
aperture, not headroom. Adds the fit table and the
stop-mask-render-verify procedure (the 3090 bench law, applied to the
APU); ComfyUI.md gotcha links it.
2026-10-21 13:35:50 -07:00
Frank cb86ad8fd7 docs(observability): post-blackbox state, copier law, label-promotion why
Prometheus/Observability/Provisioning pages: every job up after the
blackbox swap, the copier list learns copy_blackbox_config and the
alloy validate gate, the config table gains alloy + blackbox rows, and
the logs bullet states the stream-label law instead of the retired
'awaiting deploy' note.
2026-10-21 13:33:03 -07:00
Frank 078b550bd8 test(observability): copier sandboxes, probe-target seam exception, cop gates
- caddyfile seam learns the documented exception: URL-shaped targets
  are legal only inside a prober job (state is per-job; the first cut
  leaked it and passed a planted URL).
- observability seams: blackbox unit + planner entries, both copiers
  executed for real in sandboxes, alloy engine-validation refusal
  (image-gated), dashboard panel invariants.
- AGENTS.md: structural target downs retired.
2026-10-21 13:33:03 -07:00
Frank 765a8cb7de feat(deploy): blackbox copier, alloy engine-validation gate, planner wiring
copy_alloy_config now validates the rendered config with the very
alloy image that will run it, before installing: a misplaced attribute
crash-loops the unit behind a green deploy, and no textual seam can
see it. copy_blackbox_config ships the prober config; the planner maps
the new unit and shared/blackbox/** to the blackbox-exporter service,
ordered before prometheus.
2026-10-21 13:33:03 -07:00
Frank 999c4fe440 fix(alloy): promote CONTAINER_NAME to an indexed Loki stream label
Loki metrics queries group only by indexed stream labels, so the
journal-as-JSON stream made every per-container view structurally
empty (measured 6.5k lines/5min flowing while the board read zero).
A loki.process relabel stage now extracts CONTAINER_NAME into a
'container' label. alloy 1.21's json stage has no on_error attribute
(validated against the pinned image) — format_as_json guarantees the
line shape, so no error branch exists.
2026-10-21 13:33:03 -07:00
Frank 2af2e6ac56 fix(grafana): observability and logs-overview boards read real data
observability.json: stale 'structural downs' panel description fixed,
target census matches the post-blackbox job set. logs-overview.json:
panels grouped by the promoted container label instead of the absent
container_name, so the board stops reading empty beside live traffic.
2026-10-21 13:32:58 -07:00
Frank fbe54ca1ad feat(gitea): enable /metrics for Prometheus scraping
Gitea shipped metrics disabled, so its scrape target was a permanent
structural down beside a healthy service.
2026-10-21 13:32:58 -07:00
Frank f073b91033 feat(observability): blackbox exporter replaces the HTML-probe jobs
Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe
jobs were structurally dead targets reading as outages. HTTP liveness
now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback
bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and
ComfyUI via __param_target relabeling. The unit is a boot-persistent,
self-healing root quadlet.
2026-10-21 13:32:58 -07:00
dank 425994515a Merge pull request 'fix(deploy): grafana copier provisions EVERY dashboard json' (#68) from fix/grafana-copier-glob into main 2026-10-21 09:49:15 -07:00
Frank c41384e03d fix(deploy): grafana copier provisions EVERY dashboard json, not a hand-listed subset
copy_grafana_provisioning named llm-fleet.json alone, so a new board in
shared/grafana/provisioning/dashboards/ silently never reached
/opt/grafana/provisioning. Measured 2026-10-21: the containers board was
merged, seam-tested, and documented as working since 2026-10-20, yet was
NEVER provisioned — its panel fixes were undeployable; the other four
boards existed on disk only because someone hand-copied them once (and
host-system.json there had gone stale against the repo). The copier now
globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run
the copier for real into a sandbox (red against the old hand-listed
body, green against the glob). Vault: the copier law + the containers
incident on Observability Provisioning, the corrected dashboards claim
on Observability.

Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the
dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.
2026-10-21 09:40:57 -07:00
dank 127621bdd7 Merge pull request 'docs(vault)+fix(grafana): audit tranche 1 and dashboard data-flow fixes' (#67) from fix/dragoon-context-200k into main 2026-10-21 09:28:43 -07:00
Frank c96d0228a4 fix(grafana): dashboard census fixes — every panel now receives data
Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).

- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
  non-matching pairs and the right side collapses to {job} while the left
  carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
  live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
  container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
  so the '> 0' filter can never match. Rewritten to share-of-host-total via
  label_replace join of the node exporter's MemTotal by host (the parens
  form is a PromQL parse error: nothing may follow group_left); 62 series
  live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
  (ALERTS carries only firing/pending series); description now says so,
  per the Stale Signal rule that 0-series-on-ALERTS is not a defect.

logs-overview and observability: all targets already returning data.
2026-10-21 02:45:57 -07:00
Frank a61308d72e docs(vault): audit tranche 1 — one-owner dedupe of the observability cluster, dead-shape numbers destroyed
Purpose-test pass over the audit findings (every verdict re-verified
against the file before touching it; three scout findings were hallucinated
— a nonexistent filename, my own changelog line, and sanctioned Eras rows):

- Observability Provisioning: destroyed the 63-line 'Current delivered
  state' block — chain topology/relay/delivery restates [[Observability]]
  and [[Alertmanager]] (those are the owners). Its three sole-custody laws
  MOVED to Stale Signal Audit as 'Three rules that outlive the incident':
  down-beside-healthy -> read lastError; Prometheus alert state in process
  vs Alertmanager's --storage.path persistence; candidate-arm metrics are
  never fleet state.
- RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool;
  never reuse a B/token ratio across geometries; pin pool bytes
  floor-sized), era-labelled the rates, destroyed the measured ladder
  table for the superseded shape-G/bf16 rungs.
- Stale Signal Audit: new rules section + Provisioning Related link.

check-okf 109/0, vault links 0 unresolved, docs seams green.
2026-10-21 02:35:44 -07:00
dank ccb805cf0b Merge pull request 'fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches' (#66) from fix/dragoon-context-200k into main 2026-10-21 02:25:42 -07:00
Frank c6e2887f94 fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches
The lane's child dlopens libxrt_core.so.2 from /opt/xilinx/lib/<multiarch>/
as well; our mounts filled only /opt/xilinx/xrt/lib and the install path.
Since the host's libxrt2 was reconfigured 2026-10-20 the child dies naming
the missing prefix ('No such library
/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2') and the engine
supervisor takes the WHOLE container down on a dead lane — measured
2026-10-21: frank-dragoon boot-looped 88 times with chat dead while the
GPU lines looked healthy.

Each XRT library now mounts THREE times from the same real file (never a
symlink); the multiarch name derives from FRANK_DRAGOON_XRT_DIR's basename
so the bats fixture proves the derivation. Proven end-to-end before commit:
production-shaped scratch boot + the three extra mounts answers the full
boot ledger, npu_api buckets u0..u9, and /health ok at the 262144 ceiling.

A dead lane is not survivable on this engine — arm it only with all three
prefixes (page law updated).
2026-10-21 02:21:53 -07:00
dank 9eb53ef937 Merge pull request 'feat(frank-dragoon): 200,000-token harness window via the native 262,144 ceiling' (#65) from fix/dragoon-context-200k into main 2026-10-21 01:40:40 -07:00
Frank 688928d489 docs(dragoon): the 262144 ceiling, the 200,000 window, and the 95 % trigger law
Shape + harness contract on the endpoint page, AGENTS.md, and the OMP
Harness worked example recomputed to the live pair; the 400-zone law now
names the 85 %-comments vs 95 %-config disagreement as the recurring
cause and points at the seam that enforces the arithmetic.
2026-10-21 01:34:28 -07:00
Frank 56503dbad1 feat(frank-dragoon): raise HALOGEN_CTX to the native 262144 ceiling
The 200,704 ceiling squeezed the harness contextWindow down to 155,648
(the 400-zone fit law needs ~45k of gap under the ceiling), which is why
long dragoon sessions kept hitting the 400 zone: the harness clamps
max_tokens to window - est - 4096, its chars/4 est under-counts ~3 %,
and config.yml compacts at 95 % while the pair was designed for 85 %.

Ceiling -> checkpoint's native 262,144 (no RAM move: the KV pool stays
401,408 and is what the ledger bills; the band ctx <= pool <= slots*ctx
still holds). The harness window becomes 200,000 with a 62,144 gap.

The fit-law seam now reads the declared window from the vault page (the
one owner) and checks the arithmetic at the live 95 % trigger, so a
ceiling/window pair that drifts back together fails tests/run.sh
instead of a live session.
2026-10-21 01:34:28 -07:00
dank fbeb9701e4 Merge pull request 'fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway' (#64) from fix/gpu-serve-and-cadvisor-target into main 2026-10-21 00:39:54 -07:00
Frank 650b6e0281 fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:

- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
  serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
  but never accept()ed, loopback deploy gate passed, every scrape that
  reached the other bind timed out on the SYN backlog. Now one thread per
  extra bind. Regression test spins --serve on two loopback binds and
  requires BOTH to answer /metrics.

- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
  Prometheus container: a bridge container cannot dial its own host's LAN
  address at all (hairpin refuses published ports too); the podman gateway
  answers. Target moved to host.containers.internal:8080; the seam that
  pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
  gateway plus a negative pin against the own-host LAN dial. Observability
  page records the measured reachability law replacing the inferred one.
2026-10-21 00:34:17 -07:00
dank b62a05837f Merge pull request 'fix(deploy): client allowlist missing cadvisor/gpu-exporter/prowlarr' (#63) from fix/deploy-sh-allowlist-drift into main 2026-10-21 00:15:45 -07:00
Frank 0367244d2a fix(deploy): client allowlist was missing cadvisor, gpu-exporter, prowlarr
deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are
two gates for the same door, and they drifted: the remote gate gained
cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate
never did, so `--service cadvisor|gpu-exporter` died with "is not deployed
on <target>" at the client — the first-ever deploy of a new service is
exactly the case that needs the client gate to know it. dankserver:prowlarr
(planner can emit it for dankserver) was remote-only too.

Adds the four missing pairs to the client gate and a deployment_flow
contract test that diffs the two extracted lists per target — the test the
cadvisor PR should have carried. searxng-network stays the one sanctioned
client-only exception (network unit, no remote handler), documented inline.
2026-10-21 00:15:21 -07:00
dank 656d9567d5 Merge pull request 'chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)' (#62) from chore/plex-1435 into main 2026-10-21 00:07:28 -07:00
Frank 0b4bc60a38 chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)
ls329 is the newest build on the linuxserver channel and Plex 1.43.5.11029
is their newest release; the live pin (ls325 / 1.43.4.10903) is two Plex
releases behind. Digest-verified at pin time: the tag's manifest digest
equals the registry's current 'latest' (sha256:8f655bad...e9e2334), the
same law the page records. Quadlet restart runs PMS's usual one-minor
schema migration on first boot; the deployer's /identity HTTPS gate
tolerates the first-boot migration up to 600 s.

Rollback: revert this commit (one line).
2026-10-21 00:06:53 -07:00
dank a2011f4eb6 Merge pull request 'feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet' (#61) from feat/gpu-stats into main 2026-10-20 23:50:11 -07:00
87 changed files with 4699 additions and 502 deletions

No files matched your search

+17 -15
View File
@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
(`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
IP, the one bind a container can reach as `host.containers.internal`;
iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
Prometheus targets read down for structural reasons (HTML-scraped "probes",
Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
without `--metrics`. See
`serverAppConfigs-docs/Infrastructure/Observability.md`.
The former structural target downs (HTML-scraped "probes", Gitea metrics off)
are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
every Prometheus job reports up. `frank-zealot` was down for a different reason
until 2026-10-18: llama.cpp will not serve `/metrics` at all without
`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
## Testing
@@ -419,16 +420,17 @@ disagrees with it is the bug.
digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
`/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
`HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
drafter always armed (the adaptive default disarms it and halves decode);
server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
under-counts, so a harness window equal to the ceiling makes every long
request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
**per-request ceiling** (a longer prompt is a 400, never a truncation);
`HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
pool positions buy concurrency, the per-request window never moves;
`HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
default disarms it and halves decode); server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
estimate under-counts (trigger is 95 %), so a harness window at the ceiling
makes every long request a guaranteed 400.
Reasoning arrives in `reasoning_content`; the
template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
`/completion`, `/ping`, `/props`, `/tokenize` are 404.
@@ -20,7 +20,7 @@ Wants=authentik-server.service
[Container]
ContainerName=authentik-outpost
Image=ghcr.io/goauthentik/server:2026.8.3
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
Exec=worker
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -14,7 +14,7 @@ After=network.target
[Container]
ContainerName=authentik-postgres
Image=docker.io/library/postgres:16-alpine
Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
Environment=POSTGRES_DB=authentik
@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
[Container]
ContainerName=authentik-server
Image=ghcr.io/goauthentik/server:2026.8.3
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
Exec=server
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -16,7 +16,7 @@ Wants=authentik-server.service
[Container]
ContainerName=authentik-worker
Image=ghcr.io/goauthentik/server:2026.8.3
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
Exec=worker
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -15,7 +15,7 @@ After=network.target
[Container]
ContainerName=gitea-postgres
Network=gitea.network
Image=docker.io/library/postgres:16-alpine
Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
EnvironmentFile=/home/dank/Containers/gitea/env
Environment=POSTGRES_USER=gitea
Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}
@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
[Container]
ContainerName=gitea-server
Network=gitea.network
Image=docker.io/gitea/gitea:latest
Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
EnvironmentFile=/home/dank/Containers/gitea/env
Environment=DATABASE_TYPE=postgres
Environment=DATABASE_HOST=gitea-postgres
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
Environment=DATABASE_PORT=5432
Environment=USER_UID=1000
Environment=USER_GID=1000
# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
# the container; the only route in is this host's published 3000, and the
# exposed surface is counters only — same posture as the LLM /metrics.
Environment=GITEA__metrics__ENABLED=true
Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
PublishPort=0.0.0.0:3000:3000
PublishPort=2222:22
Volume=/home/dank/Containers/gitea/data:/data
@@ -23,7 +23,7 @@ After=gluetun.service network.target
[Container]
ContainerName=marreta_browser
Network=gluetun.container
Image=docker.io/lightpanda/browser:nightly
Image=docker.io/lightpanda/browser:nightly@sha256:979c36ed869f1016b8dcf22939a0633c262f2c30e892a03a9f720eba2ff1931e
[Service]
TimeoutStartSec=300
+1 -1
View File
@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
[Container]
ContainerName=marreta
Network=gluetun.container
Image=ghcr.io/manualdousuario/marreta:latest
Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
# Relocate off SearXNG's 8080 inside the shared netns.
Environment=NGINX_HTTP_PORT=8085
Environment=APP_NAME=Marreta
@@ -33,7 +33,7 @@ Wants=network-online.target
[Container]
ContainerName=alertmanager
Image=docker.io/prom/alertmanager:latest
Image=docker.io/prom/alertmanager:latest@sha256:e9733bafb1bdef9b00e25a21f8f99dc26a22224bf16641ad754d1649f4c3357a
Network=dankclaw_default
User=0:0
Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager
@@ -0,0 +1,56 @@
# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
#
# Install path on dankclaw:
# ~/.config/containers/systemd/blackbox-exporter.container
#
# WHY Network=host. The probes it runs are (a) the host's own LAN IP
# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
# bridge container cannot reach; netavark DNATs published ports only, see
# Container Host Reachability in the vault) and (b) ComfyUI's published port
# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
# reach neither class reliably.
#
# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
# bridge container dialing the podman gateway 169.254.1.2 lands on the host
# LOOPBACK — that is the scrape path, and it needs this exporter to be
# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
# address itself is NOT bindable from the host netns — it exists only
# inside each rootless netns; binding it dies "bind: cannot assign
# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
# No public DNS name, no guard chain: internal net only, per the exposure
# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
# flag per bind; plain --listen-address does not exist and the exporter
# exits in 100 ms with "unknown long flag" (live 2026-10-21).
[Unit]
Description=Blackbox exporter (HTTP probe prober)
After=network-online.target
[Container]
Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
ContainerName=blackbox-exporter
Network=host
# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
# exits "unexpected /bin/blackbox_exporter".
Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
# is passed as one quoted token and quadlet then folds the following
# --health-interval/--health-retries (and the image name) into the command
# string — unit died twice that way, live 2026-10-21. wget only; this
# busybox variant has no curl, and host-network wget answers on loopback.
HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
HealthInterval=30s
HealthRetries=3
[Service]
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
@@ -35,7 +35,7 @@ Wants=network-online.target
[Container]
ContainerName=cadvisor
Image=gcr.io/cadvisor/cadvisor:v0.49.1
Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
PodmanArgs=--pid=host --cgroupns=host
PublishPort=127.0.0.1:8080:8080
PublishPort=192.168.0.143:8080:8080
@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
[Container]
ContainerName=grafana
Image=docker.io/grafana/grafana:latest
Image=docker.io/grafana/grafana:latest@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
Network=dankclaw_default
User=0:0
PublishPort=127.0.0.1:3002:3000
@@ -33,7 +33,7 @@ Wants=network-online.target
[Container]
ContainerName=node-exporter
Image=docker.io/prom/node-exporter:latest
Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
Network=host
PodmanArgs=--pid=host
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100
@@ -43,8 +43,16 @@ Wants=network-online.target
[Container]
ContainerName=prometheus
Image=docker.io/prom/prometheus:latest
Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
Network=dankclaw_default
# Same-host scrape targets are dialed at host.containers.internal (the podman
# gateway): a bridge container cannot dial its own host's LAN IP (netavark
# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
# it unsupported and the WHOLE user quadlet batch then fails to generate,
# silently orphaning every other unit (live 2026-10-21: prometheus.service
# vanished while its container kept running as an orphan).
User=0:0
Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
PublishPort=127.0.0.1:9090:9090
@@ -27,7 +27,7 @@ After=network.target
[Container]
ContainerName=open-webui
Network=open-webui.network
Image=ghcr.io/open-webui/open-webui:main
Image=ghcr.io/open-webui/open-webui:main@sha256:4860ba005ae8f19f08d1e12b0868855b7b339ea0dad9927f196820fc0de04575
PublishPort=3005:8080
Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
Environment=ANONYMIZED_TELEMETRY=False
+1 -1
View File
@@ -24,7 +24,7 @@ Wants=network-online.target
[Container]
ContainerName=pihole
Image=docker.io/pihole/pihole:latest
Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
Network=slirp4netns
PublishPort=0.0.0.0:53:53/udp
PublishPort=0.0.0.0:53:53/tcp
+1 -1
View File
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
[Container]
ContainerName=gluetun
Network=searxng.network
Image=docker.io/qmcgaw/gluetun:latest
Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
AddCapability=NET_ADMIN
AddDevice=/dev/net/tun:/dev/net/tun
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
ContainerName=searxng-valkey
Network=searxng.network
NetworkAlias=valkey
Image=docker.io/valkey/valkey:9-alpine
Image=docker.io/valkey/valkey:9-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
Volume=/opt/searxng/valkey:/data
[Service]
+1 -1
View File
@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
[Container]
ContainerName=searxng
Network=gluetun.container
Image=ghcr.io/searxng/searxng:latest
Image=ghcr.io/searxng/searxng:latest@sha256:5ed0323b5f5f8701264ef65e239a2bd83cb57098f2b971fa30bf79c06eeae271
Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
Volume=/opt/searxng/config:/etc/searxng
+1 -1
View File
@@ -29,7 +29,7 @@ Wants=network-online.target
[Container]
ContainerName=seerr
Image=ghcr.io/seerr-team/seerr:v3.4.1
Image=ghcr.io/seerr-team/seerr:v3.4.1@sha256:f4768de5f616248d723e05891f3345a1402123775d03bf0890dbfedc0831bda1
Network=dankclaw_default
PublishPort=127.0.0.1:5055:5055
Volume=/home/dank/Containers/seerr/data:/app/config
@@ -43,7 +43,7 @@ After=network.target
[Container]
ContainerName=calibre-server
Image=lscr.io/linuxserver/calibre:9.15.0
Image=lscr.io/linuxserver/calibre:9.15.0@sha256:59e4b0e036e62f228f8bcfae1ade42f47047719885fe454ddbad38e4597b987b
Network=pasta
EnvironmentFile=/home/dank/Containers/calibre-server/env
Environment=PUID=1000
@@ -16,7 +16,7 @@ After=network.target
[Container]
ContainerName=calibre-web
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402@sha256:802729966faed5e32d671cd4f92ba6196e9e754a6c0f740ce1ee9acd86b10e55
Network=pasta
Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
Environment=LANG=en_US.UTF-8
@@ -21,7 +21,13 @@ After=network.target
[Container]
ContainerName=copyparty
Image=ghcr.io/9001/copyparty-ac:latest
# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
# the pin rides :latest@digest (the tag in the reference is cosmetic; the
# digest is the pull). Pinned 2026-10-21 at the build whose label is
# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
# (1.20.26); the previously running build was the rolling :latest at
# 1.20.24 (issue #73 on command-center).
Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
Network=dankserver_appnet
EnvironmentFile=/home/dank/Containers/copyparty/env
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank
@@ -16,7 +16,7 @@ After=network.target
[Container]
ContainerName=jellyfin
Image=ghcr.io/jellyfin/jellyfin:latest
Image=ghcr.io/jellyfin/jellyfin:latest@sha256:c3efa69a22f2691fc88d6bb035e87a8560e395a6047f062fb381c61c3185e8f5
Network=dankserver_appnet
Environment=DEBIAN_FRONTEND=noninteractive
Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
@@ -18,7 +18,7 @@ Wants=network-online.target
[Container]
ContainerName=cadvisor
Image=gcr.io/cadvisor/cadvisor:v0.49.1
Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
PodmanArgs=--pid=host --cgroupns=host
PublishPort=127.0.0.1:8080:8080
PublishPort=192.168.0.235:8080:8080
@@ -37,7 +37,7 @@ Wants=network-online.target
[Container]
ContainerName=node-exporter
Image=docker.io/prom/node-exporter:latest
Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
Network=host
PodmanArgs=--pid=host
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100
+1 -1
View File
@@ -4,7 +4,7 @@ After=network-online.target
Wants=network-online.target
[Container]
Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329@sha256:8f655bad7dcbc5398a78f70a5af12600b4ed3424f901566ed1eaa6710e9e2334
ContainerName=plex
Network=dankserver_appnet:alias=plex
Environment=PUID=0
@@ -14,7 +14,7 @@ After=network.target
[Container]
ContainerName=romm-mariadb
Image=docker.io/library/mariadb:11
Image=docker.io/library/mariadb:11@sha256:f16d7a344413e519dff518f27bdb221388f84b0d274c97c4fb5f2cb0dc1b34fa
Network=dankserver_appnet:alias=mariadb
EnvironmentFile=/home/dank/Containers/romm/env
Environment=MARIADB_DATABASE=romm
+1 -1
View File
@@ -12,7 +12,7 @@ After=network.target
[Container]
ContainerName=romm-redis
Image=docker.io/library/redis:7-alpine
Image=docker.io/library/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
Network=dankserver_appnet:alias=redis
Volume=/home/dank/Containers/romm/redis:/data
HealthCmd=redis-cli ping
+1 -1
View File
@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
[Container]
ContainerName=romm
Image=ghcr.io/rommapp/romm:latest
Image=ghcr.io/rommapp/romm:latest@sha256:03295f95a6290e752641f9553ac4198b0e5b71b879d57f7b2371b98f608feb54
Network=dankserver_appnet
EnvironmentFile=/home/dank/Containers/romm/env
Environment=DB_HOST=mariadb
@@ -4,7 +4,7 @@ After=network-online.target
Wants=network-online.target
[Container]
Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245@sha256:c66d3afa8f7a701a48dd1b1f57b22eca0483d895d670fe11ecff783f6a9bbd73
ContainerName=tautulli
Network=dankserver_appnet:alias=tautulli
Environment=PUID=0
@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
[Container]
ContainerName=gluetun
Image=docker.io/qmcgaw/gluetun:latest
Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
AddCapability=NET_ADMIN
AddDevice=/dev/net/tun:/dev/net/tun
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -15,7 +15,7 @@ After=gluetun.service
[Container]
ContainerName=lidarr
Image=lscr.io/linuxserver/lidarr:latest
Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -24,7 +24,7 @@ After=gluetun.service
[Container]
ContainerName=prowlarr
Image=lscr.io/linuxserver/prowlarr:latest
Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -29,7 +29,7 @@ Network=gluetun.container
# Pinned to the last 4.x line: Readarr's retired qB client requires the
# qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
# empty body, which fails Readarr's auth check. All other clients accept 4.x.
Image=lscr.io/linuxserver/qbittorrent:4.6.7
Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -15,7 +15,7 @@ After=gluetun.service
[Container]
ContainerName=radarr
Image=lscr.io/linuxserver/radarr:latest
Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -19,7 +19,7 @@ After=gluetun.service
[Container]
ContainerName=readarr
Image=lscr.io/linuxserver/readarr:amd64-1.0.980
Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -16,7 +16,7 @@ After=gluetun.service
[Container]
ContainerName=sonarr
Image=lscr.io/linuxserver/sonarr:latest
Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
+32 -14
View File
@@ -1,27 +1,45 @@
#!/usr/bin/env bash
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
# derive-protected-experts.py over the calib dir, splice its cast lines into
# the marked section of the recipe in place, keep the pre-edit recipe in the
# campaign trash. Refuses to run twice (marker text says derived).
# campaign trash. Refuses to run twice (marker text says derived), and refuses
# when a requested target count was not reached. Run it on a COPY of the
# template so the committed recipe stays the clean, reviewable baseline.
set -euo pipefail
CALIB=${1:-/home/dank/corpora/calib}
RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
PY=/home/dank/venvs/radcorpus/bin/python
DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
TARGET_COUNT=${3:-0}
PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
TS=$(date +%Y%m%d-%H%M%S)
# OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
# scratch directory; on the box they are the campaign's usual places.
OUT=${OUT:-/tmp/protected-cast-$TS.txt}
TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
ARGS=("$DERIVE" "$CALIB")
if [ "$TARGET_COUNT" -gt 0 ]; then
ARGS+=(--target-count "$TARGET_COUNT")
fi
"$PY" "${ARGS[@]}" > "$OUT"
N=$(grep -c 'cast dtype=bf16' "$OUT")
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
echo " (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
exit 1
fi
echo "derived $N protected experts -> $OUT"
cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
mkdir -p "$TRASH_DIR"
cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
"$PY" - "$RECIPE" "$OUT" <<'PYEOF'
import sys
recipe, castfile = sys.argv[1], sys.argv[2]
lines = open(recipe).read().splitlines()
@@ -35,7 +53,7 @@ for l in lines:
elif 'PROTECTED-END' in l:
insec = False
out.append(l)
out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
elif not insec:
out.append(l)
open(recipe, 'w').write('\n'.join(out) + '\n')
+220
View File
@@ -0,0 +1,220 @@
#!/usr/bin/env bash
# build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
# checkpoint /home/dank/models/Qwen3.8-Flash-Next into
# /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
# no rule matches keeps the checkpoint's encoding, so every tensor stays the
# bf16 it is in the safetensors and the container is the checkpoint's own
# numbers behind a .rad header.
#
# This file is the campaign's measuring stick, twice over: scripts/
# run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
# arm's --kld-ref is scored against that record) and the same container serves
# with the calibration tap to write the Grams that derive the protected-expert
# list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
# byte in bf16, silently moves every gate in the campaign — hence no recipe, no
# --reuse, and the four refusal gates below.
#
# Why a container at all: a checkpoint dir has no mapped/file tiers, and a
# weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
# served from one; rad-convert also bakes the tokenizer. The convert runs on
# the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
# kernels on CPU and production serving is untouched for the whole run (the
# precedent that proved it: scripts/convert-swift15-w4dn8.sh).
#
# Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
# written, cheapest first — the verifier runs last of them because its Hub gate
# hashes 343 GiB and must not be paid for by a run that was going to refuse on
# the output file anyway.
# checkpoint the tree must be on disk; an absent dir names the command that
# makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
# free space the convert writes ~343,323 MiB next to the ~343,323 MiB
# checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
# plus 6.5 GiB of slack) is the floor. The volume that cannot
# hold both is discovered at minute 2, not hour 5.
# output the campaign converts the reference ONCE. An existing
# qwen38-bf16.rad is refused; --force retires it to
# ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
# deleted, and the move is a rename on the same volume).
# verified scripts/download-qwen38-bf16.sh's verify mode must pass. A
# failed verify exits 1, not 2: the tree being the wrong payload
# is a result, a missing environment is a refusal.
# plan rad-convert --plan-only -v first — the oracle for "does this
# shape boot". A refused plan costs seconds and saves a convert.
#
# usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
# --dry-run print the argv it would exec (podman wrapping rad-convert),
# one shell-quoted argument a line, and exit 0. No gate runs, no
# container starts, nothing is written — the offline test seam.
# It does resolve the checkpoint and output paths through the
# /models mount, because an argv that carries host paths would
# not run either.
# --plan-only run the plan and every gate except the disk and output ones —
# those two protect bytes the plan never writes, and this is the
# probe to run when the disk gate itself is what refuses
# --force retire an existing output container to the trash dir first
#
# knobs: FRANK_QWEN38_MODELS the host dir mounted at /models (default
# /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
# Qwen3.8-Flash-Next, the same knob the verifier reads);
# FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
# QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
# verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
# reach it through the environment untouched.
# exit: 0 converted (or planned), 1 the checkpoint failed verification,
# 2 refused before anything was written
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# ONE owner of the engine pin: the digest frank-archon serves on. The reference
# must be built by the SAME engine build that scores it, or a KLD delta is
# partly a version delta.
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-archon.env"
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
LOG="/home/dank/logs/qwen38_bf16_convert.log"
DRY=0
PLAN_ONLY=0
FORCE=0
while [ $# -gt 0 ]; do
case "$1" in
--dry-run) DRY=1 ;;
--plan-only) PLAN_ONLY=1 ;;
--force) FORCE=1 ;;
*) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
exit 2 ;;
esac
shift
done
# The convert sees the tree only through the /models mount, so both paths must
# be under it; the container-side spelling is what goes into the argv.
case "$CKPT" in
"$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
*) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
exit 2 ;;
esac
case "$OUT" in
"$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
*) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
exit 2 ;;
esac
# Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
# bound this container will actually be served at. Both flags come off the pinned
# image's own help: --kv-cache-dtype at convert "only picks the kernels it
# checks" and the container is identical either way, so naming fp8 is free
# insurance — if the fp8 KV kernels the fleet serves would not resolve, the
# convert refuses here instead of the boot failing on a drained window.
# --max-ctx is the bound declare is run at, whose default is the checkpoint's
# training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
# machine, so the default is not an option). 32,768 is set by the corpus, not by
# taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
# pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
# corpus — so the window the rig boots at clears every record 2.6x over and the
# reference boot cannot die on a sequence that will not fit. The rig's window is
# one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
# 32,768) and that script refuses a corpus its window cannot hold, so raising one
# without the other is caught before the drained window, not inside it. The
# payload containers the gate scores declare at the training context because
# production must hold the 200k shape; a declared bound is a declare-time check,
# not a weight change, so the reference and the payloads carry the same numbers
# whatever each declared at — which is why one can be measured against the other.
CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
if [ "$DRY" = "1" ]; then
echo "dry run: no gate checked, no container started, nothing written"
echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
echo "# plan argv:"
printf '%q\n' "${PLAN_CMD[@]}"
echo "# convert argv:"
printf '%q\n' "${CONV_CMD[@]}"
exit 0
fi
# ---- gate: the checkpoint is on disk ---------------------------------------
if [ ! -d "$CKPT" ]; then
echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
exit 2
fi
# ---- gate: peak disk --------------------------------------------------------
# The deepest existing ancestor is the right thing to ask: a fresh
# /home/dank/models/rad has no free space of its own to report.
OUT_DIR="$(dirname "$OUT")"
PROBE="$OUT_DIR"
while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
exit 2
fi
if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
exit 2
fi
# ---- gate: one conversion ---------------------------------------------------
if [ -e "$OUT" ]; then
if [ "$PLAN_ONLY" = "1" ]; then
echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
elif [ "$FORCE" = "1" ]; then
mkdir -p "$TRASH"
DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
mv "$OUT" "$DEST"
echo "TRASHED: retired the existing reference container $OUT -> $DEST"
else
echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
exit 2
fi
fi
# ---- gate: the tree really is the checkpoint (the full verify) -------------
echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
rc=0
bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
if [ "$rc" -ne 0 ]; then
if [ "$rc" -eq 2 ]; then
echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
exit 2
fi
echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
exit 1
fi
# ---- plan, then convert ------------------------------------------------------
mkdir -p "$(dirname "$LOG")"
# rad-convert opens -o during declare as well, so the output directory exists
# before the plan; that empty directory is all this creates ahead of it.
mkdir -p "$OUT_DIR"
echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
"${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
rc=${PIPESTATUS[0]}
if [ "$rc" -ne 0 ]; then
echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
exit 2
fi
if [ "$PLAN_ONLY" = "1" ]; then
echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
exit 0
fi
echo "== convert: $CKPT -> $OUT ($(date -Is))"
"${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
rc=${PIPESTATUS[0]}
if [ "$rc" -ne 0 ]; then
echo "rad-convert rc=$rc; log: $LOG" >&2
exit "$rc"
fi
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env bash
# convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
# campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
# `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
#
# Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
# converter runs on the host backend and the fleet can keep serving through the whole script.
# The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
# boundary between measurement and serving.
#
# Three arms, because the campaign's question is which search knob pays for its hours
# (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
#
# cd3 the recipe as written — coordinate descent three sweeps. The control.
# cd6 six sweeps: the same search, twice the passes.
# ao1 act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
# stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
# cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
# act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
# the two knobs cannot be combined into a best-of-both container.
#
# The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
# recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
# a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
# expert line and the other 60 lines cannot drift between arms.
#
# Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
# (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
# a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
#
# Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
# inputs checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
# directory converts UNCALIBRATED with one stderr note, which is silently worse
# than failing here.
# reference the 335.3 GiB BF16 reference container must be GONE before the converts start.
# It and the checkpoint together eat the volume; its job (the KLD recording and
# the calibration grams) is finished by then, and both survive on disk without the
# container beside it. The FP8-native anchor is FETCHED after the converts (disk
# sequencing), so nothing here waits on it.
# plan --plan-only runs FIRST, per arm, and prints "about X would be written". The
# number is the bill: free space must cover it plus the 128k-prompt overshoot
# margin, and the finished artifact is checked against it.
# outputs no arm overwrites anything; move the old one to trash.
#
# Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
# [--target-count N] [--keep-recipe]
set -u -o pipefail
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed}
CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
CALIB=${CALIB:-$CAMPAIGN/calib}
IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
OUTDIR=${OUTDIR:-/home/dank/models/rad}
BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad} # the reference container, not an arm output
STATE=${STATE:-$CAMPAIGN} # the dir the container mounts read-only: the
# spliced recipe and the imatrix live here
TARGET_COUNT=${TARGET_COUNT:-48}
MIN_GRAMS=${MIN_GRAMS:-24576} # per-expert DOWN grams: 48 recorded layers x 512
# experts. The checkpoint has 49 MoE layers, but the
# MTP layer is never recorded, so its 512 experts are
# absent from the tap's output by design
MARGIN_MIB=${MARGIN_MIB:-40960} # the long-prompt overshoot, off the plan number
TS=$(date +%Y%m%d-%H%M%S)
ARMS=()
DRY=0
PLAN_ONLY=0
KEEP_RECIPE=0
while [ "$#" -gt 0 ]; do
case "$1" in
--arm) ARMS+=("$2"); shift 2 ;;
--dry-run) DRY=1; shift ;;
--plan-only) PLAN_ONLY=1; shift ;;
--target-count) TARGET_COUNT="$2"; shift 2 ;;
--keep-recipe) KEEP_RECIPE=1; shift ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
done
[ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
for a in "${ARMS[@]}"; do
case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
done
# The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
# every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
# container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
# it for empty).
expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
arm_quant() {
case "$1" in
# The control is the template itself: no override, so the recipe's own line decides.
cd3) return 0 ;;
cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
# act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
# columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
# this arm is the interleave knob alone, not cd=3 plus it.
ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
esac
}
# --- gates ------------------------------------------------------------------------------------
[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
[ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
[ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
[ "$GRAMS" -ge "$MIN_GRAMS" ] || {
echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
[ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
# The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
# no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
# converter fail halfway through a planned arm.
[ "$(dirname "$IMATRIX")" = "$STATE" ] || {
echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
if [ -e "$BF16_RAD" ]; then
echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
"the calibration grams are already taken, so move it to trash — the converts need its" \
"space (peak-disk law)" >&2
exit 2
fi
for a in "${ARMS[@]}"; do
[ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
done
# --- the protected plane, spliced into a per-run copy of the template --------------------------
mkdir -p "$STATE"
WORK="$STATE/recipe-$TS.recipe"
cp "$RECIPE" "$WORK"
if [ "$DRY" = 1 ]; then
echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
else
# Positional, because the tool reads its recipe and calib dir by position: passing them as
# environment variables would be silently ignored and it would rewrite the COMMITTED
# template in place instead of this run's copy.
"$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
|| { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
fi
run_convert() { # run_convert ARM [extra flags...]
local arm="$1"; shift
local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
local argv=(--rm --name "qwen38-convert-$arm"
-e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
-v "$(dirname "$CKPT"):/models:ro"
-v "$CALIB:/data/calib:ro"
-v "$STATE:/campaign:ro"
-v "$OUTDIR:/out")
local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
--tokenizer "/models/${CKPT##*/}/tokenizer.json"
--imatrix /campaign/"${IMATRIX##*/}"
# No --max-ctx: the context declare then runs at the checkpoint's training context
# (262144), which is what the shipped container did — rad-info --meta on it carries
# no override. A container declared at a smaller bound cannot be expected to serve
# the 200k window D2 holds, and the convert is not where to discover that.
# --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
# checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
# to by the convert, not by a boot on a drained window.
--kv-cache-dtype fp8 -o "/out/$(basename "$out")")
local q; q=$(arm_quant "$arm")
[ -n "$q" ] && conv+=(--quant "$q")
if [ "$DRY" = 1 ]; then
echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
return 0
fi
podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
2>&1 | tee "$log"
return "${PIPESTATUS[0]}"
}
# --- plan, then bill, then convert -------------------------------------------------------------
overall=0
for arm in "${ARMS[@]}"; do
echo "== arm $arm =="
if [ "$DRY" = 1 ]; then
# Both lines, in order: the plan runs before the convert, and the convert is what the
# operator is about to pay an afternoon for.
run_convert "$arm" --plan-only
run_convert "$arm"
continue
fi
rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
[ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
echo "$plan_log"
# "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
# hardcoding a size, so the gate still works after any recipe edit.
want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
[ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
overall=1; continue
fi
if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
rc=0; run_convert "$arm" || rc=$?
[ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
out="$OUTDIR/qwen38-w5nl64-$arm.rad"
got_mib=$(( $(stat -c %s "$out") / 1048576 ))
# The receipt: the artifact is what the plan predicted, within the metadata the plan does not
# count. An arm that came out materially smaller lost a plane somewhere.
lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
overall=1; continue
fi
echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
done
if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
mkdir -p "$CAMPAIGN/.AGENTTRASH"
mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
fi
exit "$overall"
+7 -2
View File
@@ -34,7 +34,6 @@ SERVICE_ORDER = (
"hermes",
"hermes-dashboard",
"hermes-webui",
"prometheus",
"grafana",
"loki",
"alloy",
@@ -42,6 +41,9 @@ SERVICE_ORDER = (
"cadvisor",
"gpu-exporter",
"alertmanager",
# The prober must answer before prometheus re-runs the services job.
"blackbox-exporter",
"prometheus",
"authentik-postgres",
"authentik-server",
"authentik-worker",
@@ -97,6 +99,7 @@ DANKCLAW_SERVICES = {
"cadvisor",
"gpu-exporter",
"alertmanager",
"blackbox-exporter",
"authentik-postgres",
"authentik-server",
"authentik-worker",
@@ -277,6 +280,8 @@ def services_for_path(path: str, target: str) -> set[str]:
# Provisioned datasources and dashboards: git is the source of
# truth, so a change here must restart the service that reads them.
_add(services, "grafana")
elif group == "blackbox":
_add(services, "blackbox-exporter")
elif group == "homepage":
# Homepage is a Dankclaw-owned deployment. Shared homepage files
# are intentionally not copied during a Dankserver-only run.
@@ -380,7 +385,7 @@ def services_for_path(path: str, target: str) -> set[str]:
# consumer (prometheus before grafana, loki before alloy).
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
_add(services, name)
return services
if len(parts) >= 2 and parts[1] == "gitea":
+80 -17
View File
@@ -86,7 +86,7 @@ service_is_allowed() {
dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
dankclaw:comfyui|dankclaw:hermes|\
dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -252,16 +252,24 @@ install_signal_relay() {
# The relay binds the host's LAN IP (the one address a bridge container
# can reach as host.containers.internal), so keep the unauthenticated
# endpoint off the rest of the LAN: admit loopback, the host itself
# (its own health probe) and the podman bridge subnet only. Idempotent:
# -C before -I.
# (its own health probe) and the podman bridge subnet only.
# ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
# shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
# ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
# (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
# dankclaw drain aborted at the alertmanager gate twice before the cause
# was read off `nft list ruleset`). Rebuild the tail instead of inserting
# it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
# at the head (-C before -I, unchanged), then APPEND the DROP so it is
# last and the ACCEPTs decide first. Converges on poisoned hosts.
run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
# The interpreter needs a moment to bind after restart; a one-shot curl
# raced the socket on 2026-09-17 and failed a healthy deploy.
local i
@@ -311,8 +319,45 @@ copy_loki_config() {
}
copy_alloy_config() {
run_privileged mkdir -p /opt/alloy/config
run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
# The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
# carries the host's name, so Loki streams say WHICH machine's journal a
# line came from without a second config file per host.
# *_CONF_ROOT are test seams: the bats cases execute these copiers for
# real against a sandbox with a stubbed run_privileged.
local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
tmp="$(mktemp)"
sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
echo "alloy config: host-label substitution failed" >&2
rm -f "$tmp"
return 1
fi
# Syntax gate: alloy's config language has no schema — a misplaced
# attribute (e.g. `on_error` outside the json stage) parses only at RUN
# time, and the unit then crash-loops with a healthy-looking deploy.
# Validate against the very image that will run it, before installing.
# Skipped only in the bats sandbox (no image there).
if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
if ! podman run --rm -v "$tmp:/c.alloy:ro" \
--entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
validate /c.alloy; then
echo "alloy config: validate FAILED against the pinned image" >&2
rm -f "$tmp"
return 1
fi
fi
run_privileged mkdir -p "$root/config"
run_privileged cp "$tmp" "$root/config/config.alloy"
rm -f "$tmp"
}
copy_blackbox_config() {
# Single-file mount target: the unit binds this exact path into
# /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
# BLACKBOX_CONF_ROOT is the same test seam.
local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
run_privileged mkdir -p "$root"
run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
}
copy_grafana_provisioning() {
@@ -321,14 +366,25 @@ copy_grafana_provisioning() {
# datasource would be invisible to review, lost with the data volume, and
# unknown to anyone reading the repository. The unit already mounts
# /opt/grafana/provisioning read-only, so this only has to populate it.
run_privileged mkdir -p /opt/grafana/provisioning/datasources \
/opt/grafana/provisioning/dashboards
# GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
# copy runs unprivileged (the bats case executes this function for real).
local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
run_privileged mkdir -p "$root/provisioning/datasources" \
"$root/provisioning/dashboards"
run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
/opt/grafana/provisioning/datasources/prometheus.yml
"$root/provisioning/datasources/prometheus.yml"
run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
/opt/grafana/provisioning/dashboards/provider.yml
run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
/opt/grafana/provisioning/dashboards/llm-fleet.json
"$root/provisioning/dashboards/provider.yml"
# EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
# turns each file in the directory into a dashboard, so a copier that names
# files one by one silently drops new boards (measured 2026-10-21: the
# containers board existed in the repo since its commit and was NEVER
# provisioned — the panel-census browser check could not find it while the
# hand-placed older boards happened to be on disk). Same rule as the quadlet
# tier: the directory is the contract, the copy is generated state.
for dash in shared/grafana/provisioning/dashboards/*.json; do
run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
done
}
render_caddyfile() {
@@ -552,6 +608,9 @@ deploy_observability() {
;;
cadvisor)
;;
blackbox-exporter)
copy_blackbox_config
;;
*)
echo "unknown observability service: $service" >&2
return 2
@@ -566,6 +625,9 @@ deploy_observability() {
loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
# Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
# is not bindable from the host netns — see the unit header).
blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
esac
}
@@ -1401,9 +1463,10 @@ deploy_anubis() {
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
# qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
# + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
# run a 200,000 window with its ~45k fit gap; the pool still carries two
# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
# ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
# relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
# file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -2033,7 +2096,7 @@ deploy_service() {
plex) deploy_plex ;;
tautulli) deploy_tautulli ;;
alloy) deploy_alloy ;;
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
gpu-exporter) deploy_gpu_exporter ;;
comfyui) deploy_comfyui ;;
*) deploy_compose_service "$service" ;;
+6 -4
View File
@@ -133,13 +133,15 @@ fi
valid_service() {
local service="$1"
case "$TARGET:$service" in
# dankclaw:searxng-network is client-only: a network quadlet with no
# remote dispatch handler, so the remote gate does not (must not) list it.
dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
dankclaw:gitea-server|dankclaw:homepage|\
dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
dankclaw:alertmanager|\
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:alertmanager|dankclaw:blackbox-exporter|\
dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
dankclaw:open-notebook|\
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
dankserver:frank-archon|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
dankserver:prowlarr|dankserver:frank-archon|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
+151 -24
View File
@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
[--min-route-mult 3.0]
[--min-route-mult 3.0] [--target-count 0]
--target-count N fills the byte bill: the bars pick the base set exactly as
the bar-only pass does, then the energy ranking admits experts one at a time
(each with its parity partner) until the count lands. The energy bar NEVER
relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
walks down only when the ranking cannot produce that many candidates.
0 takes the bars as given.
"""
import argparse
import collections
import glob
import os
import re
@@ -32,6 +39,7 @@ import sys
import numpy as np
NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
def read_gram(path):
@@ -49,27 +57,8 @@ def read_gram(path):
return k, rank, int(rows), tr
def main():
ap = argparse.ArgumentParser()
ap.add_argument("calib_dir")
ap.add_argument("--min-share", type=float, default=0.03)
ap.add_argument("--min-route-mult", type=float, default=3.0)
a = ap.parse_args()
# layer -> global expert -> (rank, rows, energy)
data = {}
for path in sorted(glob.glob(os.path.join(
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
m = NAME_RE.match(os.path.basename(path))
if not m:
continue
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
k, _, rows, tr = read_gram(path)
data.setdefault(layer, {})[expert] = (rank, rows, tr)
if not data:
raise SystemExit("no per-expert down grams found")
def select(data, min_share, min_route_mult):
"""(cast lines, report lines, count) for one pair of bars."""
lines, report = [], []
total = 0
for layer in sorted(data):
@@ -83,7 +72,7 @@ def main():
for e, (rank, rows, tr) in experts.items():
es, rs = tr / tot_e, rows / max(tot_r, 1)
rec = (e, rank, es, rs)
if es >= a.min_share and es >= a.min_route_mult * rs:
if es >= min_share and es >= min_route_mult * rs:
qual.append(rec)
elif es > 0:
pool.append(rec)
@@ -107,10 +96,148 @@ def main():
f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
f"ratio {es/max(rs,1e-12):6.1f}")
total += len(chosen)
return lines, report, total
def parity_repair(chosen, by_half, report, only=None):
"""Add the next-best candidate to every half — or to `only` — holding an odd count.
The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
of bf16 experts on a half is not a style problem: the two tables would not match.
`by_half` is energy-descending, so the first unchosen candidate is the next most
energetic, which is the expert worth keeping. Returns how many were added.
"""
added = 0
for key in ([only] if only is not None else sorted(by_half)):
layer, half = key
n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
if n % 2 == 0:
continue
for e, es, rs in by_half.get(key, []):
if (layer, e) not in chosen:
chosen[(layer, e)] = (es, rs)
added += 1
report.append(f"blk.{layer}: half {half} odd -> added e{e} "
f"(energy {es*100:.2f}%)")
break
else:
report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
f"left to make it even")
return added
def select_to_count(data, min_share, min_route_mult, target):
"""(lines, report, total, cutoff) for the set that LANDS the count.
The bars decide the base set; the count is then filled from the energy ranking,
one expert at a time, each addition taking its half's next-best candidate with it
so every rank's half stays even. Only the RATIO bar walks down to find candidates
— it is the law that says an expert is disproportionate and not merely busy, and
the caller's byte bill has to be filled — while the ENERGY bar holds, because that
bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
the set bottoms out at: the bar this campaign actually converted with.
"""
report = []
shares, ranked = {}, []
for layer in sorted(data):
experts = data[layer]
tot_e = sum(v[2] for v in experts.values())
tot_r = sum(v[1] for v in experts.values())
if tot_e <= 0 or tot_r <= 0:
report.append(f"blk.{layer}: empty recording, skipped")
continue
for e, (_rank, rows, tr) in experts.items():
es, rs = tr / tot_e, rows / max(tot_r, 1)
if es <= 0:
continue
shares[(layer, e)] = (es, rs)
ranked.append((es, layer, e, rs))
ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
by_half = {}
for es, layer, e, rs in ranked:
by_half.setdefault((layer, e // 256), []).append((e, es, rs))
chosen = {k: v for k, v in shares.items()
if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
total = len(chosen)
total += parity_repair(chosen, by_half, report)
mult = min_route_mult
while True:
pool = [(es, l, e, rs) for es, l, e, rs in ranked
if es >= mult * rs and (l, e) not in chosen]
if len(pool) + total >= target or mult <= 0.0:
break
mult -= 0.5
report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
f"for {target - total} slots; relaxing toward 0")
for es, layer, e, rs in pool:
if total >= target:
break
if (layer, e) in chosen:
continue
chosen[(layer, e)] = (es, rs)
total += 1
report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
lines, cutoff = [], None
for (layer, e) in sorted(chosen):
es, rs = chosen[(layer, e)]
cutoff = es if cutoff is None else min(cutoff, es)
lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight cast dtype=bf16")
return lines, report, total, cutoff
def main():
ap = argparse.ArgumentParser()
ap.add_argument("calib_dir")
ap.add_argument("--min-share", type=float, default=0.03)
ap.add_argument("--min-route-mult", type=float, default=3.0)
ap.add_argument("--target-count", type=int, default=0)
a = ap.parse_args()
# layer -> global expert -> (rank, rows, energy)
data = {}
for path in sorted(glob.glob(os.path.join(
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
m = NAME_RE.match(os.path.basename(path))
if not m:
continue
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
k, _, rows, tr = read_gram(path)
data.setdefault(layer, {})[expert] = (rank, rows, tr)
if not data:
raise SystemExit("no per-expert down grams found")
if a.target_count > 0:
lines, report, total, cutoff = select_to_count(
data, a.min_share, a.min_route_mult, a.target_count)
bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
else:
lines, report, total = select(data, a.min_share, a.min_route_mult)
bar = f"min-share {a.min_share:.3g}"
# The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
# every layer must hold an even number of bf16 experts. Judged on the output and
# not on the internal sets, so a slip in either pass is caught here rather than at
# engine load time.
halves = collections.Counter()
for rule in lines:
m = CAST_RE.match(rule)
if m:
halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
odd = {key: n for key, n in halves.items() if n % 2}
if odd:
print("PARITY LAW BROKEN: " + ", ".join(
f"blk.{lay} half {half} holds {n} protected experts"
for (lay, half), n in sorted(odd.items())), file=sys.stderr)
raise SystemExit(2)
print("\n".join(lines))
print("\n".join(report), file=sys.stderr)
print(f"total protected: {total}", file=sys.stderr)
print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
if __name__ == "__main__":
+249
View File
@@ -0,0 +1,249 @@
#!/usr/bin/env bash
# download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
# BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
# converts, and the source the KLD reference is recorded from.
#
# Why the base and not a finetune: the campaign converts to compare against the
# publisher's own iq4r container, and that container was built from this
# checkpoint. Base-in, base-out; anything else confounds the gate. (The
# predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
# set is identical to this repo's but whose weights are not — none of its
# calibration grams, KLD reference or protected list carries over.)
#
# Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
# in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
# 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
# verified the key set against this repo, so no gate letter is needed).
#
# Verification is four gates deep, because the failure this repo actually lived
# through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
# HTTP 401/403 and writes the error body as the artifact — three shards of
# identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
#
# 1. shard census: the file count must equal the expected shard count, so a
# partially fetched fragment tree (the .partNofM trap) cannot pass.
# 2. per-shard completeness, from the shard's OWN safetensors header: the
# header declares where the tensor data ends, so every shard's true size is
# known offline. A torn tail is named here, by file, instead of surfacing
# as a load error five hours later.
# 3. index coverage: every shard model.safetensors.index.json references
# exists and is named.
# 4. total bytes against the number the Hub declared, then, unless --no-hub,
# every file's sha256 against the Hub's LFS record — including the shards
# already on disk, so a previous partial download is never trusted.
#
# The token never enters a command line (the 2026-09-16 leak law): it is read
# into HF_TOKEN in the environment, where a local process listing is the only
# reader. This repo is ungated, so the fetch simply runs without one when no
# token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
# hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
#
# usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
# [--shards N] [--bytes B] [--repo ID]
# --fetch download first (resumable: re-run to continue), then verify
# --no-hub skip the Hub-side sha256 sweep; the other gates always run
# exit: 0 verified, 1 verification failed, 2 environment / usage error
set -euo pipefail
REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
# Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
# 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
# shards only, so a config file edited on the Hub cannot fail it.
EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
FETCH=0
HUB=1
while [ $# -gt 0 ]; do
case "$1" in
--fetch) FETCH=1 ;;
--no-hub) HUB=0 ;;
--dir) LOCAL_DIR="$2"; shift ;;
--shards) EXPECTED_SHARDS="$2"; shift ;;
--bytes) EXPECTED_BYTES="$2"; shift ;;
--repo) REPO="$2"; shift ;;
*) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
exit 2 ;;
esac
shift
done
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
FAILS=0
note() { echo "$@"; FAILS=$((FAILS + 1)); }
if [ "$FETCH" = "1" ]; then
# Every fetch gate is a hard stop: skipping the download silently and then
# verifying a half tree is how a campaign ships a stale payload.
HFGET="$SELF_DIR/hfget.sh"
if [ ! -f "$HFGET" ]; then
echo "REFUSED: the fetch helper is not here: $HFGET" >&2
exit 2
fi
if ! command -v jq >/dev/null 2>&1; then
echo "REFUSED: hfget.sh needs jq on PATH" >&2
exit 2
fi
mkdir -p "$LOCAL_DIR"
# The campaign's peak-disk law starts at the fetch: the checkpoint and the
# reference container built from it are ~335 GiB each, and a tree that does
# not fit is discovered hours into a transfer, when it costs the most.
free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
echo " (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
echo " nothing is left half-written)" >&2
exit 2
fi
echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
if [ -z "${HF_TOKEN:-}" ]; then
CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
if [ -f "$CREDSTORE" ]; then
HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
elif [ -f "$HOME/.cache/huggingface/token" ]; then
# The legacy single-line token file the hub library writes.
HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
elif [ -f "$HOME/.hf_token" ]; then
HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
fi
fi
if [ -n "${HF_TOKEN:-}" ]; then
export HF_TOKEN
fi
bash "$HFGET" "$REPO" "$LOCAL_DIR" main
unset HF_TOKEN
fi
if [ ! -d "$LOCAL_DIR" ]; then
echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
exit 1
fi
INDEX="$LOCAL_DIR/model.safetensors.index.json"
if [ ! -f "$INDEX" ]; then
echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
exit 1
fi
echo "== gate 1: shard census ($LOCAL_DIR)"
SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
else
echo " ok: $SHARDS shards"
fi
echo "== gate 2: every shard is whole, judged by its own header"
# A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
# declares where the data ends, so the file's true size is computable offline:
# 8 + header + the largest end offset. Truncation and error bodies both fall
# here, and each is reported with the file that caused it.
while IFS= read -r f; do
state=$(python3 - "$f" <<'PY'
import json, os, struct, sys
path = sys.argv[1]
size = os.path.getsize(path)
try:
with open(path, "rb") as fh:
(hlen,) = struct.unpack("<Q", fh.read(8))
if hlen == 0 or hlen > size:
print("NOT_SAFETENSORS")
raise SystemExit
header = json.loads(fh.read(hlen))
except Exception:
print("NOT_SAFETENSORS")
raise SystemExit
if not header:
print("NOT_SAFETENSORS")
raise SystemExit
end = max((v["data_offsets"][1] for v in header.values()
if isinstance(v, dict) and "data_offsets" in v), default=0)
want = 8 + hlen + end
print("OK" if want == size else f"TRUNCATED want={want} have={size}")
PY
)
case "$state" in
OK) ;;
NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
esac
done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
if [ "$FAILS" -eq 0 ]; then
echo " ok: every shard's data extent matches its size on disk"
fi
echo "== gate 3: the index's shards are all present"
INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
import json, sys
map_ = json.load(open(sys.argv[1]))["weight_map"]
print("\n".join(sorted(set(map_.values()))))
PY
)
while IFS= read -r name; do
[ -n "$name" ] || continue
if [ ! -f "$LOCAL_DIR/$name" ]; then
note "FAIL MISSING: $name is named by the index and is not on disk"
fi
done <<< "$INDEX_SHARDS"
if [ "$FAILS" -eq 0 ]; then
echo " ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
fi
echo "== gate 4: total bytes vs the Hub's declaration"
TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
else
echo " ok: $TOTAL bytes"
fi
if [ "$HUB" = "1" ]; then
echo "== gate 5: every file's sha256 vs the Hub's LFS record"
# size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
TMPD="$(mktemp -d)"
trap 'rm -rf "$TMPD"' EXIT
MANIFEST="$TMPD/tree.json"
PLAN="$TMPD/plan"
curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
-o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
python3 - "$MANIFEST" > "$PLAN" <<'PY'
import json, sys
for e in json.load(open(sys.argv[1])):
if e.get("type") != "file":
continue
sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
if sha:
print(e["path"], e["size"], sha)
PY
BAD=0
while read -r rel size sha; do
[ -n "${rel:-}" ] || continue
if [ ! -f "$LOCAL_DIR/$rel" ]; then
note "FAIL MISSING: $rel is on the Hub and not on disk"
continue
fi
have=$(stat -c %s "$LOCAL_DIR/$rel")
if [ "$have" != "$size" ]; then
note "FAIL SIZE: $rel is $have bytes, Hub says $size"
continue
fi
got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
if [ "$got" != "$sha" ]; then
note "FAIL SHA256: $rel got $got want $sha"
BAD=$((BAD + 1))
fi
done < "$PLAN"
if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
echo " ok: sha256 matches for every file"
fi
rm -rf "$TMPD"
trap - EXIT
fi
if [ "$FAILS" -gt 0 ]; then
echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
exit 1
fi
echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"
+22 -12
View File
@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
# stays at the image default, which is what was qualified) ------------------
# Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
HALOGEN_MODEL_ID=frank-dragoon
# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
# the page) — a window equal to the ceiling makes every long request a 400.
HALOGEN_CTX=200704
# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
# at once. Pool positions buy concurrency; the per-request window never moves
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
# RAM the pool takes is RAM the n-gram table's page cache loses.
# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
# longer prompt is a 400 naming the limit, never a truncation; stretching
# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
# page): with the window at 200,000 the gap is 62,144, which absorbs the
# chars/4 under-count. A window at or near the ceiling makes every long
# request a guaranteed 400.
HALOGEN_CTX=262144
# ONE pool behind both slots. The harness window is 200,000, so 401,408
# still carries TWO maximum harness-length conversations at once
# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
# ceiling-length requests would queue instead — buying those means raising
# the pool, and RAM the pool takes is RAM the n-gram table's page cache
# loses. Pool positions buy concurrency; the per-request window never moves
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
HALOGEN_KV_POOL_POSITIONS=401408
# Two concurrent conversations: the MTP drafter is not slot-locked on this
# engine, so the second slot is capacity, not a speculation tax.
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
# Node paths and the XRT directory are variables so the bats seam can drive
# these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
# once at its own path, because those libraries find each other at their install
# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
# system dir, which dangle inside the container.
# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
# searches that third prefix; without it the lane dies naming it and the
# supervisor takes the container down — 2026-10-21 boot loop). The basename of
# this directory IS the multiarch name. The vendor forbids mounting an
# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
# sysfs root the fabric-clock preflight reads (same knob the vendor's
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
After=network-online.target
Wants=network-online.target
+6
View File
@@ -32,6 +32,7 @@ import glob
import os
import subprocess
import sys
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
FAMILIES = [
@@ -198,6 +199,11 @@ def main(argv):
return 2
servers.append(ThreadingHTTPServer((b, args.port), Handler))
sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
# EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
# healthy in `ss` yet times out every scrape once its SYN backlog fills —
# and the loopback deploy gate cannot see it. One thread per extra bind.
for s in servers[1:]:
threading.Thread(target=s.serve_forever, daemon=True).start()
servers[0].serve_forever()
return 0
+418
View File
@@ -0,0 +1,418 @@
#!/usr/bin/env python3
"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
beside the recording. Python 3 stdlib only: numpy must NOT be required, because
this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
WHY. `rad-convert` takes its per-column activation second moments and its
expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
Only the GGUF form carries a per-expert dimension — with the legacy flat form,
or with none, every expert in a layer is weighted by the layer's aggregate,
which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
already measured everything; this tool only re-voices it, so the imatrix and the
protected-expert census (derive-protected-experts.py) are the SAME recording:
one measurement, two consumers, neither able to disagree about which experts are
hot.
INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
(arch/common/rad_arch.h, calib_write). Header law, same parse as
scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
numbering (rad_block_moe_fp8.h: ge = expert_base + e):
gram.r<rank>.blk.L.ffn_down_exps.bin the layer's pooled gram
gram.r<rank>.blk.L.ffn_down_exps.<e>.bin expert e's own gram
`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
rows, so **the diagonal is exactly the per-column activation second moment an
imatrix wants**, and `rows` is the count the reader hands back from `count()`.
Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
put a full matrix, and the quantiser asks for columns.
OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
layer's experts folded into ONE entry, so expert 9 is slice 9 of
`blk.L.ffn_down_exps`):
blk.L.ffn_down_exps in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
gate_up input is the same hidden-state rows as its down
input, so the second moment over them is identical and
so is the routing count. Two tensor infos over one
payload: the reader maps each half independently
(imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
by base name), and copying ~200 MB to state one fact
twice buys no correctness.
blk.L.ssm_inz [k] and [1], from the pooled gram: that row's input is
the same hidden state, so its columns take the pooled
diagonal and its count is the layer's pooled rows.
Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
entry's cols do not equal the weight's width and the convert proceeds without
it: an entry the reader silently ignores is a lie in the artifact, worse than
an absent one. The corollary to check before trusting the two hidden-width
entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
expert width, so the gram's k must equal the checkpoint's hidden width (its
config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
its count stays its own routing rows — the rule the engine itself applies when it
picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
is worse than round-to-nearest, and those experts carry a fraction of a percent
of the routing). The count never inherits: popularity is `count`'s job. The count
of inherited experts is reported on stderr.
RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
so the ranks describe DISJOINT experts and the fold is a UNION into the one
[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
calib_rows). The same global expert in two per-expert files is legal only when
the numbers agree exactly; otherwise the recording is corrupt and the entry would
be whichever file sorted last, so refuse.
GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
relative to the aligned data section. Every tensor offset is padded up to
`general.alignment` and the data section itself starts aligned. That is the
classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
arithmetic point into the middle of the previous tensor's floats, and it reads
them without complaint.
chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
row total, not a sum over layers: every layer's tap accumulated over the whole
corpus, so adding the layers counts each token once per layer and inflates the
figure by the layer count. chunk_count = that row total // chunk_size, the
"chunks of chunk_size" line imatrix.cpp prints for itself.
REFUSALS, each exit 2 with a message naming what is missing:
NO_GRAMS the directory holds no RADGRAM2 down-projection gram
NOT_RADGRAM2 a gram whose magic is not RADGRAM2
SHORT a gram whose length is not 32 + k*k*4
NO_POOLED a layer with expert grams and no pooled gram to inherit from
K_MISMATCH a layer whose files disagree about k (rank pooled grams, or an
expert gram against its layer's pooled gram)
EXPERT_CONFLICT a global expert recorded twice with different numbers
EXISTS the -o path already holds a file (the trash law: move it)
CHUNK_COUNT the derived chunk count does not fit the u32 metadata field
USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
[--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
"""
import argparse
import os
import re
import struct
import sys
MAGIC = b"RADGRAM2"
HDR = 32 # 8s I I Q I + pad, rad_arch.h
KOWN_ROWS_PER_K = 4 # libquant/lq_hess.h kOwnRowsPerK
ALIGNMENT = 32 # general.alignment, and the pad unit
POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
GT_U32, GT_STR, GT_ARR = 4, 8, 9
GGML_F32 = 0
def refuse(msg):
sys.stderr.write("REFUSED %s\n" % msg)
sys.exit(2)
def read_gram(path):
"""(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
try:
with open(path, "rb") as f:
raw = f.read()
except OSError as e:
refuse("UNREADABLE %s: %s" % (path, e))
if len(raw) < HDR:
refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
% (path, len(raw), HDR))
magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
if magic != MAGIC:
refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
if k <= 0:
refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
need = HDR + k * k * 4
if len(raw) != need:
refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
% (path, len(raw), need, k))
# The diagonal only, read by stride out of the row-major matrix.
diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
for i in range(k)]
return k, rank, rows, domain, diag
def load_dir(d, verbose):
"""{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
Files outside the ffn_down_exps stem are not ours to fold: the tap names a
pooled gram after every projection it taps, and those carry other widths."""
if not os.path.isdir(d):
refuse("NO_GRAMS %s: not a directory" % d)
layers, rotated = {}, 0
for name in sorted(os.listdir(d)):
m = POOLED_RE.match(name) or EXPERT_RE.match(name)
if not m:
continue
path = os.path.join(d, name)
if not os.path.isfile(path):
continue
pooled = bool(POOLED_RE.match(name))
layer = int(m.group(2))
k, rank, rows, domain, diag = read_gram(path)
if domain != 0:
rotated += 1
L = layers.setdefault(layer, {"pooled": None, "experts": {}})
if pooled:
P = L["pooled"]
if P is None:
L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
elif P["k"] != k:
refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
" disagree about the width"
% (layer, P["paths"][0], P["k"], path, k))
else:
# calib_sum: each rank summed the same stem over its own rows.
P["rows"] += rows
P["paths"].append(path)
for i in range(k):
P["diag"][i] += diag[i]
if verbose:
sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
% (layer, name, k, rows))
continue
e = int(m.group(3))
own = L["experts"].get(e)
if own is None:
L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
"path": path, "ranks": [rank]}
if verbose:
sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
% (layer, rank, e, rows))
continue
if own["rows"] != rows or own["diag"] != diag:
refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
" (rows=%d) record the same global expert with different"
" numbers" % (layer, e, own["path"], own["rows"], path, rows))
own["ranks"].append(rank)
sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
" and %s; folded once\n"
% (layer, e, own["path"], name))
if not layers:
refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
"[.<expert>].bin files in the directory" % d)
return layers, rotated
def build_entries(layers, min_rows, verbose):
"""([ (name, dims, payload|None, key) ], census, layers emitted)."""
entries = []
census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
for layer in sorted(layers):
P, E = layers[layer]["pooled"], layers[layer]["experts"]
if E and P is None:
refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
" expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
" under the row threshold would have nothing to inherit"
% (layer, len(E), layer))
if P is None:
continue
k = P["k"]
census["layers"] += 1
census["rows"] = max(census["rows"], P["rows"])
dn = "blk.%d.ffn_down_exps" % layer
# A staging key names ONE payload. A gate_up entry reuses the down
# layer's bytes by carrying the same key with no payload of its own;
# in_sum2 and counts are different bytes and must never share a key.
ksum, kcnt = (layer, "sum2"), (layer, "counts")
kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
# One entry holds one width: an expert gram recorded at another k would
# inherit (or be inherited by) a diagonal of the wrong length, and the
# reader would then answer `cols` from whichever file set dims[0].
for e in sorted(E):
if E[e]["k"] != k:
refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
" layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
E[e]["k"], k))
if not E:
# No per-expert recording for this layer: emit the dense form
# ([k], [1]) rather than 512 identical slices, which would claim a
# per-expert measurement that was never taken.
entries.append((dn + ".in_sum2", [k],
struct.pack("<%df" % k, *P["diag"]), ksum))
entries.append((dn + ".counts", [1],
struct.pack("<f", float(P["rows"])), kcnt))
sys.stderr.write("note: layer %d has no per-expert grams; %s is"
" dense over the pooled gram\n" % (layer, dn))
else:
floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
n_expert = max(E) + 1
sum2, counts = bytearray(), bytearray()
for e in range(n_expert):
own = E.get(e)
if own is not None and own["rows"] >= floor:
src = own["diag"]
rows = own["rows"]
else:
src = P["diag"]
rows = own["rows"] if own is not None else 0
census["inherited"] += 1
sum2 += struct.pack("<%df" % k, *src)
counts += struct.pack("<f", float(rows))
census["experts"] += 1
census["routing"] += rows
if min(E) != 0:
sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
" those slices carry the pooled gram at count 0\n"
% (layer, min(E) - 1))
entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
gu = "blk.%d.ffn_gate_up_exps" % layer
entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
entries.append((gu + ".counts", [1, n_expert], None, kcnt))
if verbose:
sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
" own-gram floor=%d\n"
% (layer, k, n_expert, P["rows"], floor))
inz = "blk.%d.ssm_inz" % layer
entries.append((inz + ".in_sum2", [k],
struct.pack("<%df" % k, *P["diag"]), kpgm))
entries.append((inz + ".counts", [1],
struct.pack("<f", float(P["rows"])), kpcnt))
return entries, census
def enc_str(v):
b = v.encode("utf-8")
return struct.pack("<Q", len(b)) + b
def write_gguf(path, entries, datasets, chunk_size, chunk_count):
"""Write GGUF v3 and return its size in bytes.
One payload per key: entries that share a key share the bytes and the
offset, so a second tensor info states the same fact without copying it."""
kv = bytearray()
kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
+ struct.pack("<I", ALIGNMENT)
kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
+ struct.pack("<I", chunk_size)
kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
+ struct.pack("<I", chunk_count)
kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
+ struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
for d in datasets:
kv += enc_str(d)
head = bytearray()
head += b"GGUF" + struct.pack("<I", 3)
head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
head += kv
payloads = {}
for name, _dims, payload, key in entries:
if payload is None:
continue
prev = payloads.get(key)
if prev is not None and prev != payload:
# Two different payloads on one key would ship whichever was
# appended last while BOTH tensor infos pointed at it: the counts
# payload landing over the in_sum2 payload is exactly how a
# hand-rolled GGUF writer loses half its data without an error.
refuse("INTERNAL %s: two different payloads share one staging key"
% name)
payloads.setdefault(key, payload)
infos = bytearray()
offsets, order, size = {}, [], 0
for name, dims, payload, key in entries:
if key not in payloads:
refuse("INTERNAL %s: no payload staged for its key" % name)
if key not in offsets:
offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
size = offsets[key] + len(payloads[key])
order.append(key)
infos += enc_str(name) + struct.pack("<I", len(dims))
for d in dims:
infos += struct.pack("<Q", d)
infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
body = bytes(head) + bytes(infos)
data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
with open(path, "wb") as f:
f.write(body)
f.write(b"\0" * (data_start - len(body)))
# The offsets in the tensor infos are the file's law, not a hint: each
# payload must land at its own padded offset. Writing them back to back
# instead leaves every reader that trusts the info — imatrix.cpp does —
# reading its neighbour's floats: plausible values, no error, wrong
# quantisation.
here = data_start
for key in order:
f.write(b"\0" * (data_start + offsets[key] - here))
f.write(payloads[key])
here = data_start + offsets[key] + len(payloads[key])
return f.tell()
def main():
ap = argparse.ArgumentParser(
description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
ap.add_argument("calib_dir")
ap.add_argument("-o", "--out", required=True)
ap.add_argument("--datasets", default="",
help="comma-separated names for imatrix.datasets")
ap.add_argument("--chunk-size", type=int, default=4096,
help="rows per chunk for imatrix.chunk_size (default 4096)")
ap.add_argument("--min-rows", type=int, default=0,
help="own-gram row threshold (default 4*k, the engine's"
" kOwnRowsPerK law)")
ap.add_argument("-v", "--verbose", action="store_true")
a = ap.parse_args()
if os.path.exists(a.out):
# The repo's deletion law: an artifact that exists is moved to trash by
# a human, never silently overwritten by a re-run of the tool that made
# it, so a second convert cannot destroy the imatrix the first one fed.
refuse("EXISTS %s: an imatrix is already there; move it to"
" .AGENTTRASH/ or name another -o" % a.out)
if a.chunk_size <= 0:
refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
layers, rotated = load_dir(a.calib_dir, a.verbose)
entries, census = build_entries(layers, a.min_rows, a.verbose)
if not entries:
refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
if rotated:
sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
" diagonal is the rotated activation's second moment,"
" not the model's own\n" % rotated)
chunk_count = census["rows"] // a.chunk_size
if chunk_count > 0xFFFFFFFF:
# imatrix.chunk_count is a u32 in the reader's grammar, so there is no
# legal way to record a bigger one. Reaching this means either a chunk
# size picked for a far larger run or a row total read out of the wrong
# header offset — rows is a u64 and a u32 parse is off by 2**32 — and
# the reader would trust whatever number landed in the field.
refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
" imatrix.chunk_count field; raise --chunk-size"
% (chunk_count, census["rows"], a.chunk_size))
nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
a.chunk_size, chunk_count)
sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
" rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
% (len(entries), census["layers"], census["experts"],
census["inherited"], census["rows"], census["routing"],
chunk_count, a.chunk_size, nbytes, a.out))
if __name__ == "__main__":
main()
+122
View File
@@ -0,0 +1,122 @@
#!/usr/bin/env bash
# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
# acceptance gate: three KLD reports against ONE reference recording, one verdict.
#
# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
# kldref, in the same drained window that records the calibration grams). Every scored
# container is measured against that one recording, so the KL scale runs from 0 = the
# checkpoint itself downward in faithfulness. Three containers ride it:
#
# fp8.json the published FP8-native release — the achievable end of the interval
# iq4r.json the payload frank-archon serves — the bar to beat
# cand.json the arm under test — the subject
#
# The FP8 number is what makes the interval absolute instead of relative: "half the gap"
# is half of the distance between the 4-bit payload and the near-lossless release, not half
# of an invented distance to the checkpoint. That is also why the FP8 payload is FETCHED and
# scored, never converted: it is a second opinion about what this checkpoint tolerates, not a
# step of the campaign. It lands LAST (disk sequencing) — by then the 335 GiB bf16 container
# is gone, so its 172.8 GiB fits.
#
# The law, in order of severity, on the mean KL of one slice (--slice all | a corpus source):
# 1. KL(fp8) < KL(iq4r) the anchors are ordered. If the near-lossless
# release does NOT sit closer to the checkpoint than
# the 4-bit payload, the corpus, the tokenizer or the
# boot shape is wrong and no arm number means anything.
# 2. KL(fp8) < KL(cand) < KL(iq4r) the candidate lies strictly inside the interval —
# outside it, something is mis-measured, not better
# 3. KL(cand) <= KL(iq4r) - R x gap it closes at least R (--recover, default 0.5) of the
# gap = KL(iq4r) - KL(fp8): at least half the way from
# what is served toward the release (agreed 2026-10-27)
# 4. top-1(cand) >= top-1(iq4r) the agreement the harness keys on does not fall
#
# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
# same step size, --deterministic, no speculation) that makes the three numbers comparable.
# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
#
# Usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
set -uo pipefail
[ "$#" -ge 3 ] || { echo "usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2; exit 2; }
python3 - "$@" <<'PY'
import json, sys
a = sys.argv[1:]
opts, files = [], []
i = 0
while i < len(a):
if a[i].startswith("--"):
opts += [a[i], a[i + 1]]
i += 2
else:
files.append(a[i])
i += 1
if len(files) != 3:
sys.exit("usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
o = dict(zip(opts[0::2], opts[1::2]))
slice_ = o.get("--slice", "all")
recover = float(o.get("--recover", 0.5))
if not 0.0 < recover <= 1.0:
sys.exit(f"--recover must be in (0,1], got {recover}")
def die(msg):
print(msg, file=sys.stderr)
raise SystemExit(2)
def read(path):
try:
d = json.load(open(path))
except Exception as e:
die(f"cannot read the KLD report {path}: {e}")
if slice_ == "all":
g = d.get("all")
else:
g = (d.get("by_source") or {}).get(slice_)
if g is None:
die(f"{path} has no slice {slice_!r}; it has: "
f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
if g is None or "kld" not in g or "top1_agreement" not in g:
die(f"{path} has no {slice_!r} summary — wrong report version?")
return d, g
rows = []
for label, path in zip(("fp8", "iq4r", "cand"), files):
d, g = read(path)
rows.append((label, path, d.get("model", "?"), g["kld"]["mean"], g["kld"]["p99_9"],
g["top1_agreement"], g["positions"]))
print(f"reference: BF16 container; slice: {slice_}")
print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8} positions")
for label, path, model, kl, p999, top1, n in rows:
print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}% {n}")
r = {label: (kl, top1) for label, _p, _m, kl, _q, top1, _n in rows}
kl_fp8, kl_iq4r, kl_cand = r["fp8"][0], r["iq4r"][0], r["cand"][0]
t_iq4r, t_cand = r["iq4r"][1], r["cand"][1]
gap = kl_iq4r - kl_fp8
ok = True
def law(n, text, passed):
global ok
ok = ok and passed
print(f" [{'PASS' if passed else 'FAIL'}] {n}. {text}")
law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
gap > 0)
law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
kl_fp8 < kl_cand < kl_iq4r)
need = kl_iq4r - recover * gap
law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
if gap > 0 else "gap is not positive; cannot score recovery",
gap > 0 and kl_cand <= need)
law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
t_cand >= t_iq4r)
print("KLD GATE " + ("PASS" if ok else "FAIL"))
sys.exit(0 if ok else 1)
PY
@@ -0,0 +1,80 @@
# qwen4exp-qwen38-w5nl64-bf16trunk.recipe — every routed expert, BOTH projections, at five bits:
# u5 codes into libquant's 32-level w5nl table, an E4M3 scale a group of 64 under the fixed 2^-13
# second level, rotated by the 128-wide Hadamard, chosen by GPTQ against $CALIB and swept three
# times by coordinate descent. This is `w5nl64a8h` in the engine's served-encoding table — the top of
# the four-bit class — applied wider than any published radiance container of this checkpoint: the
# author's own w5 rung (`qwen4exp-w5dn64-i8-hc8m.recipe`) moves only `down`, because that rung was
# priced as a byte-for-error trade against prefill; this campaign buys the whole rung and pays for
# it out of the pinned host pool (+15.3 GiB of expert plane, `--host-pool-mib 61440` per rank), not
# out of the KV pool.
#
# Every other line is the served container's recipe (dump it with
# `rad-info --recipe /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`), minus the rows this
# campaign leaves at the checkpoint:
#
# * the trunk's block linears, the shared expert and the hyper-connection mixers are NOT listed —
# dropping their i8/fp8 lines leaves them bf16, which is what the checkpoint holds. Measured bill
# +1.978 GiB of always-resident weight = ~131k tokens of KV off the pool.
# * `ssm_inz`/`ssm_out` STAY int8: the delta-net gates are the largest trunk rows (+1.9 GiB, 126k
# tokens) and the least sensitive per byte.
# * `mtp.draft_head` STAYS 2-bit and the PLE n-gram table STAYS one fixed E4M3 scale: the draft
# head's rung is cheap and its acceptance cost is a decode question this campaign does not touch,
# and bf16 for the n-gram table would pin +47.69 GiB of host RAM for a lookup.
#
# Protected experts: none in this file. `scripts/apply-protected-cast.sh` appends the derived set
# (48, two per layer so each rank's 256-expert half stays even — the four-bit GEMM takes the rest of
# a half as two equal tables by parity) from the fresh 12M-token energy/routing census, and they must
# sit ABOVE the expert row because the first rule that matches a weight decides it.
#
# CALIB=/home/dank/campaign/qwen38-w5nl64/calib \
# rad-convert /home/dank/models/Qwen3.8-Flash-Next \
# --recipe this.file --imatrix .../imatrix.gguf -o out.rad
#
# The arm driver (`scripts/convert-qwen38-w5nl64.sh`) generates the per-arm variants of the expert
# row from this template — cd=3 (control), cd=6, and `act_order=1` with NO cd= (libquant refuses the
# pair, lq_registry.cpp:105) — rather than keeping three near-duplicate recipes that can drift apart.
#
# The whole campaign, in the order that fits on one disk (every step runs on dankserver):
#
# 1. scripts/download-qwen38-bf16.sh --fetch the vendor BF16 checkpoint, 335.3 GiB
# 2. scripts/build-qwen38-bf16-rad.sh the BF16 reference container, another 335.3 GiB
# 3. one drained window, two boots:
# scripts/run-qwen38-bf16.sh kldref the KLD reference recording (the zero of scale)
# scripts/run-qwen38-bf16.sh calib the 12M-position calibration tap -> grams
# 4. trash qwen38-bf16.rad the peak-disk law: the converts need its room
# 5. scripts/grams-to-imatrix.py grams -> imatrix.gguf in the campaign dir
# 6. scripts/convert-qwen38-w5nl64.sh the arms, planned and billed before each one runs
# 7. fetch the published FP8-native release, then per container:
# scripts/run-qwen38-bf16.sh score RAD OUT fp8 anchor, served iq4r, each arm
# then scripts/kld-gate-qwen38.sh fp8.json iq4r.json cand.json
# 8. a winner is a trio edit plus `scripts/deploy.sh dankserver --service frank-archon`; the
# rollback is the commit before it. If the win needs the bigger pinned plane, raise
# FRANK_ARCHON_HOST_POOL_MIB and FRANK_ARCHON_TTM_TARGET_MIB in the SAME commit: the preflight
# floor is 2xHOST_POOL + 48,824 + prefix cache + MNBT + the store, and a pool raised without
# the TTM target is a boot refusal that Restart=always repeats forever, invisible to deploy.sh.
# PROTECTED-BEGIN — scripts/apply-protected-cast.sh splices the derived cast lines in here.
# They MUST sit above the expert rule: the first rule whose glob matches a weight decides it, so a
# cast line below the five-bit rule would never win. The set is derived from the campaign's own
# 12M-token recording (`derive-protected-experts.py --target-count 48` over the fresh grams), and the
# driver splices it into a per-arm COPY so this template stays the clean, reviewable baseline.
# PROTECTED-END
# Every routed expert, trunk and MTP layer alike, both projections, at five bits. The experts' scales
# sit between 2^-15 and 2^-10.6 under w4nl's top level of 22 and half a binade lower under w5nl's 30
# — inside E4M3's normal range under 2^-13 either way. The MTP layer has no recording and is rounded
# to nearest, which libquant says on stderr; it is one layer of 49 and only ever changes drafts.
blk.*.ffn_*_exps.*.weight gptq table=w5nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
# The delta-net's in/out gates, int8 with a bf16 scale a group of 128 searched for least error — the
# served recipe's lines, unchanged.
blk.*.ssm_inz.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ssm_out.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
# The MTP head's two mixing rows, E4M3 as served. They only ever change the drafts.
mtp.fc_hidden.weight rtn codes=fp8_e4m3 group=128 scale=f32
mtp.fc_embedding.weight rtn codes=fp8_e4m3 group=128 scale=f32
# The n-gram table at one fixed E4M3 scale, and the 2-bit draft head — both the served lines.
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
mtp.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
+346
View File
@@ -0,0 +1,346 @@
#!/usr/bin/env bash
# run-qwen38-bf16.sh — RUNS ON DANKSERVER. Boot a container over the campaign's
# ONE measuring shape for the phases that need both R9700s. The engine owns the
# cards, so frank-archon is stopped first and restored on every exit path — the
# autoresearch pattern: stop, never mask (the unit file in /etc/systemd/system
# is a symlink into the checkout and mask fails on a relinked unit; the deploy
# restores it).
#
# kldref --kld-record the reference log-probabilities: the engine
# runs /home/dank/corpora/out/kld.jsonl (197 prompts, 50,432
# scored positions, score tail 256) through prefill, writes
# /home/dank/campaign/qwen38-w5nl64/kld-ref, and EXITS. The
# model behind it is the BF16 container from
# scripts/build-qwen38-bf16-rad.sh — the vendor checkpoint's
# own numbers, which is what makes it the zero point.
# score RAD OUT --kld-ref score ONE container against that recording and
# EXIT: the FP8-native anchor, the served iq4r, and every
# arm. Deliberately the SAME engine pin, the same TP, the
# same step size, the same fp8 KV, --deterministic and no
# speculation as kldref — KL and top-1 agreement are only
# comparable across boots when everything except the weights
# is identical, so this mode is one argv with --model and
# the two --kld-* flags changed, never a second shape.
# RAD must sit under the models dir (the container reads it
# through the ro /models mount); OUT lands on a writable
# mount because --kld-out F also writes F.rows beside it.
# calib serve on :7770 with the calibration tap open
# (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
# separate driver — scripts/calib-push.py --port 7770
# --container qwen38-bf16 --corpus
# /home/dank/corpora/out/calib.jsonl — pushes the corpus; at
# 12M pooled positions the engine writes the Grams and the
# tap switches itself off (calib-push prints TAP_DONE). Stop
# this script after that; the trap brings frank-archon back.
#
# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
# weights disk tier, deterministic (the resident set pinned to the loader's plan,
# so a recorded reference and a scored candidate see the same one), no
# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
# and the production chat template rather than the one baked into the container —
# every scored container is asked the same tokenised questions the reference
# answered. The window is 32,768: kld.jsonl is 197 records with a longest of
# 38,263 characters (longest prompt FIELD 36,855, median record 27,228). Its
# token length — 10,554 for that longest record, counted by the engine itself —
# is in the funding paragraph below; the kldref gate re-checks the file against
# the window before it boots anything.
#
# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
# removed: the card is split between the weights and the cache from what the
# model needs." The resolver derives the split from --host-pool-mib,
# --gpu-headroom-mib and the activation arena; the rig inherits the first two from
# frank-archon.env, the file that owns them. On THIS payload the host pool is not
# the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch,
# where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue.
# Measured twice — 12,288 and 49,152 MiB of host pool both printed the same
# "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything
# (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names
# is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena
# scales with it)" — which is what the step size below now does.
#
# The pool, not the window, decides what the recorder accepts: it refuses any
# document longer than the pool, so the 32,768-token window was decorative while the
# cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds
# 7436, so it can never be scored" on all 197 records, exit 1. The longest record
# is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the
# chars/4.97 this repo measured on the K3 corpus).
#
# The step size is ONE value for the recording and every scoring pass:
# FRANK_QWEN38_MNBT, default 1,024. The gate's whole claim is that the reference,
# the served iq4r and every arm rode one shape, so the ceiling is set by the
# TIGHTEST payload the gate boots rather than by the bf16 reference: the quantised
# expert FILE tiers (the served iq4r, the arms) stream only at MNBT<=2048 and
# refuse the widened form above it. 2,048 was still the wrong default on this
# payload, because the arena the engine charges for a step comes straight out of
# the cache — 1.08 GiB at a 2,048-token step, 504.39 MiB at a 1,024-token one, and
# the experts keep their share either way. At 1,024 the cache is 866 MiB: 24,300
# tokens, which scores every one of the 197 documents (longest 11,970).
# Calibration keeps the wide 65,536 — grams are not logits, nothing downstream is
# compared against a step size, and that is where a big step pays.
#
# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
# (RAD and OUT are score's two positionals: RAD is a container under the models
# dir, OUT the report path whose directory is created if missing)
# --dry-run print the exact `podman run` argv, one shell-quoted argument a
# line, and exit 0: the service is NOT stopped, nothing is
# mounted, nothing is written — the offline test seam. It still
# validates the argv's own shape (paths through the /models mount,
# a step size that is a number), because an argv that could not
# boot is not worth printing.
# --image REF override the digest pin (a probe boot, or the test seam)
#
# knobs: FRANK_QWEN38_MODELS (the dir mounted at /models), FRANK_QWEN38_RAD (the
# bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
# calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
# the container reads through /models), FRANK_QWEN38_LOG_DIR, and
# FRANK_QWEN38_MNBT — the step size shared by kldref and score (default
# 1,024: the file tier's ceiling is 2,048 and production runs there, but on
# this payload the arena charged for a wider step is taken from the cache),
# and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
# see the corpus census above, and raise it together with the corpus).
# kldref refuses (exit 2) when the longest record of the corpus it is about to
# read is longer than MAX_MODEL_LEN x 3 characters — chars/3 being the
# pessimistic token:char ratio, so the script says no before the engine finds out
# inside a drained window, after the checkpoint work is already done. score
# refuses when the reference dir is absent, when RAD is not on disk or not under
# the models mount, or when OUT already exists: the trash law means a second
# score over the same name is a deliberate act, and two numbers in one file is
# how a campaign loses its own history.
# Engine stdout goes to $LOG_DIR/qwen38_bf16_<mode>.log — the boot ledger is the
# record of what the resolver actually gave the cards; the journal never sees it.
# exit: the engine's code; 2 = refused before the service was touched
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-archon.env"
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
MODE=""
POS=()
DRY=0
while [ $# -gt 0 ]; do
case "$1" in
kldref|calib|score) MODE="$1" ;;
--dry-run) DRY=1 ;;
--image) IMAGE="$2"; shift ;;
*)
# score's two positional arguments, and nothing else may be bare.
if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
POS+=("$1")
else
echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF] (unknown argument: $1)" >&2
exit 2
fi ;;
esac
shift
done
if [ -z "$MODE" ]; then
echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF] (no mode given)" >&2
exit 2
fi
if [ "$MODE" = score ] && [ "${#POS[@]}" -ne 2 ]; then
echo "usage: ${0##*/} score RAD OUT — RAD is the container to score, OUT the report to write (got ${#POS[@]} argument(s))" >&2
exit 2
fi
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
KLD_CORPUS="${FRANK_QWEN38_KLD_CORPUS:-/home/dank/corpora/out/kld.jsonl}"
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
TPL="${FRANK_QWEN38_TPL:-$MODELS/tpl.jinja}"
TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
KLD_REF_DIR="$CAMPAIGN/kld-ref"
CALIB_DIR="$CAMPAIGN/calib"
C=qwen38-bf16
SERVICE=frank-archon
# The card split, taken from the file that owns it instead of a private pair:
# --host-pool-mib also funds the KV cache (see the header), so hardcoding it
# here once starved the recorder. The `:?` guards keep an empty env fatal.
HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
# ONE step size for both logits modes. The file tier streams only at <=2,048 and
# the gate scores file-tier payloads (the served iq4r, the arms) against the
# reference, so the reference is recorded at a step the payloads can meet — never
# two numbers that a KL between them would have to forgive. 1,024 rather than the
# 2,048 ceiling: on this payload the arena charged for the step is the cache's only
# remaining supplier, and 1,024 is the rung that funds every corpus document.
MNBT="${FRANK_QWEN38_MNBT:-1024}"
if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
echo "REFUSED: FRANK_QWEN38_MNBT must be a token count (got: $MNBT)" >&2
exit 2
fi
# The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
# env file says so for production), so this is the number the corpus must fit in.
MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
echo "REFUSED: FRANK_QWEN38_MAX_MODEL_LEN must be a token count (got: $MAX_MODEL_LEN)" >&2
exit 2
fi
# Every container this campaign boots is read through the ro /models mount, so
# the host path must live under it; the container-side spelling is what goes
# into the argv. Same rule for the container a score pass opens.
under_models() {
case "$1" in
"$MODELS"/*) printf '/models/%s' "${1#"$MODELS"/}" ;;
*) return 1 ;;
esac
}
# same_dir A B — the two name one directory, judged only when BOTH exist: a dry
# run has no fixtures on disk, and two absent paths are not the same place.
same_dir() {
[ -d "$1" ] && [ -d "$2" ] && [ "$(cd "$1" && pwd)" = "$(cd "$2" && pwd)" ]
}
if [ "$MODE" = score ]; then
SCORE_RAD="${POS[0]}"
SCORE_OUT="${POS[1]}"
OUT_DIR="$(dirname "$SCORE_OUT")"
MODEL_IN="$(under_models "$SCORE_RAD")" || {
echo "REFUSED: container to score $SCORE_RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
exit 2
}
else
MODEL_IN="$(under_models "$RAD")" || {
echo "REFUSED: reference container $RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
exit 2
}
fi
# The measuring shape. Identical for kldref and score by construction — this
# array is the only place the campaign's KL numbers are defined.
ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
--host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
--deterministic --num-speculative-tokens 0
--kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
--ngram-placement ram
--override-chat-template /models/tpl.jinja)
MOUNTS=(-v "$MODELS:/models:ro")
TAPENV=()
case "$MODE" in
kldref)
# Recording is prefill-only over four slots at the campaign's single step size
# — 1,024, not production's 2,048 (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4,
# FRANK_ARCHON_MAX_BATCHED_TOKENS=2048): the wider step costs the cache the
# room the corpus needs. Corpus read-only, record dir rw.
ARGS+=(--model "$MODEL_IN"
--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
--max-num-seqs 4 --max-num-batched-tokens "$MNBT")
MOUNTS+=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
-v "$KLD_REF_DIR:/data/kld-ref")
;;
score)
# The same argv with three words changed: which container, and that this
# time the engine reads a recording and writes a report instead of the
# reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
# able to touch the zero point — and the report gets its own writable bind.
if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR"; then
echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
exit 2
fi
ARGS+=(--model "$MODEL_IN"
--kld-ref /data/kld-ref --kld-out "/data/out/$(basename "$SCORE_OUT")"
--max-num-seqs 4 --max-num-batched-tokens "$MNBT")
MOUNTS+=(-v "$KLD_REF_DIR:/data/kld-ref:ro"
-v "$OUT_DIR:/data/out")
;;
calib)
# Calibration is exempt from the shared step: the tap counts pooled
# positions and nothing downstream is compared against a step size, so the
# wide step is free throughput (a disk-tier step costs the same at 1 token
# as at 64k).
ARGS+=(--model "$MODEL_IN" --port 7770
--max-num-seqs 8 --max-num-batched-tokens 65536)
MOUNTS+=(-v "$CALIB_DIR:/data/calib")
TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
-e RADIANCE_CALIB_TOKENS=12000000)
;;
esac
CMD=(podman run --rm --name "$C"
--network host --init
--security-opt seccomp=unconfined
--ulimit memlock=-1:-1
--device /dev/kfd --device /dev/dri
"${MOUNTS[@]}" "${TAPENV[@]}"
"$IMAGE" "${ARGS[@]}")
# ---- gate: the window must hold the corpus (kldref alone can meet it) --------
# score reads a recording instead of a corpus, and every arm declares at the
# checkpoint's training context, so neither can meet a sequence too long to boot.
# Measured on the raw JSON line — the pessimistic direction, and the same figure
# the census in the header quotes. It runs on a dry run too, because a refusal
# this cheap should never have to wait for the drained window to be discovered.
if [ "$MODE" = kldref ] && [ -f "$KLD_CORPUS" ]; then
read -r REC_NO REC_CHARS < <(awk '{ if (length($0) > best) { best = length($0); where = NR } } END { print where + 0, best + 0 }' "$KLD_CORPUS")
LIMIT_CHARS=$(( MAX_MODEL_LEN * 3 ))
if [ "$REC_CHARS" -gt "$LIMIT_CHARS" ]; then
echo "REFUSED: record $REC_NO of $KLD_CORPUS is $REC_CHARS chars, past the $LIMIT_CHARS a $MAX_MODEL_LEN-token window promises at the pessimistic chars/3 — raise FRANK_QWEN38_MAX_MODEL_LEN, and the container's declared --max-ctx with it, before booting the reference" >&2
exit 2
fi
echo "corpus fits: longest record $REC_CHARS chars (number $REC_NO) inside the $MAX_MODEL_LEN-token window (limit $LIMIT_CHARS chars)"
fi
if [ "$DRY" = 1 ]; then
echo "dry run: $MODE — the production unit is left running, nothing is mounted, nothing is written"
printf '%q\n' "${CMD[@]}"
exit 0
fi
# ---- gates that only a real run needs ---------------------------------------
if [ "$MODE" = score ]; then
if [ ! -d "$KLD_REF_DIR" ]; then
echo "REFUSED: no KLD reference at $KLD_REF_DIR — record it first in a drained window: scripts/run-qwen38-bf16.sh kldref" >&2
exit 2
fi
if [ ! -f "$SCORE_RAD" ]; then
echo "REFUSED: no container to score at $SCORE_RAD" >&2
exit 2
fi
if [ -e "$SCORE_OUT" ]; then
echo "REFUSED: $SCORE_OUT already exists — one report per name; move the old one to trash before re-scoring" >&2
exit 2
fi
mkdir -p "$OUT_DIR"
else
if [ ! -f "$RAD" ]; then
echo "REFUSED: no reference container at $RAD — build it first: scripts/build-qwen38-bf16-rad.sh" >&2
exit 2
fi
[ "$MODE" = kldref ] && [ ! -f "$KLD_CORPUS" ] && {
echo "REFUSED: no KLD corpus at $KLD_CORPUS — the record would be empty" >&2
exit 2
}
fi
if [ ! -f "$TEMPLATE" ]; then
echo "REFUSED: no chat template at $TEMPLATE" >&2
exit 2
fi
mkdir -p "$LOG_DIR"
# The prod template, staged where the ro /models mount reaches it: a reference
# recorded with the container's baked template scores a different tokenisation
# than the arms served with froggeric's, and the KL would be measuring that.
cp -f "$TEMPLATE" "$TPL" || {
echo "REFUSED: cannot stage the chat template at $TPL (the container reads it through /models)" >&2
exit 2
}
[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
[ "$MODE" = calib ] && mkdir -p "$CALIB_DIR"
restore() { sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true; }
trap restore EXIT
echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
podman rm -f "$C" >/dev/null 2>&1 || true
"${CMD[@]}" 2>&1 | tee "$LOG"
rc=${PIPESTATUS[0]}
echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
exit "$rc"
+14 -5
View File
@@ -136,10 +136,17 @@ fi
# The engine re-checks size+sha256 against that catalog at boot; it cannot
# fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
# stays unset — no outbound connection at boot),
# 2. the distro XRT's three libraries, each mounted twice (where the image
# looks for XRT, and at its own path — the libraries find each other there;
# an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
# container, which the vendor calls out explicitly),
# 2. the distro XRT's three libraries, each mounted THREE times: where the
# image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
# libraries find each other there), and at /opt/xilinx/lib/<multiarch>
# — the NPU child searches that third prefix too, and since the host's
# libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
# names: omitting it crashes the lane with `No such library
# '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
# supervisor takes the dead lane down on the whole container (measured
# 2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
# of symlinks into the system dir dangles inside the container, which the
# vendor calls out explicitly),
# 3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
# hang the machine while FCLK changes speed, and the engine refuses to
# start when it moves — it cannot read host sysfs from inside the
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
exit 1
fi
REAL="$(readlink -f "$XRT_DIR/$L")"
NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
MULTIARCH="${XRT_DIR##*/}"
NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
-v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
done
HELD=0
for D in "$SYSFS"/class/drm/card*/device; do
+18 -14
View File
@@ -52,28 +52,31 @@ Where the contract lives:
The arithmetic (the coupled contract every endpoint's maxTokens comment
encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
follows is a **worked example** recomputed from the live files on 2026-10-18 —
a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
follows is a **worked example** recomputed from the live files on 2026-10-21 —
a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
bundle clamps the percentage to 1–99 and converts it to a token threshold.
2. **Output budget** = capped at the target model's `maxTokens` — every role
here compacts to frank-zealot, so the budget is its 16,384.
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
200,704. The registry's own comments design around an 85 % trigger (archon:
170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
does hold. While `config.yml` and those comments disagree, treat 85 % as the
design point and re-run this arithmetic after ANY window change — a 400 from
the compaction target mid-conversation is the symptom.
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
dragoon session the fitting window is the ENGINE ceiling, not a slot: the
190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
error plus the 45,056 request budget lands inside — that headroom is the
reason the ceiling moved to native on 2026-10-21. For a session compacting
INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
200,704. Treat 85 % as the design point until config.yml and the registry
comments agree, and re-run this arithmetic after ANY window change — a 400
from the compaction target mid-conversation is the symptom.
(Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
32,768 ≤ 200,704 zealot.)
Procedure (recompute from the LIVE files every time — windows rotate with
every swap; the block above is the 2026-10-18 state):
every swap; the block above is the 2026-10-21 state):
1. Read the `config.yml` compaction block + `models.yml` entries for source
and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
dankclaw — the comment/config disagreed. The comments now name the real
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
truth; fix comments in the same change.** A fresh instance of the same trap is
open 2026-10-18: every `maxTokens` comment still says "compaction fires at
85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
disagreement is not academic: it is why long dragoon sessions kept drawing
400s at the old 200,704 ceiling — the trigger fires late AND the est
under-counts, and a window parked near the ceiling has no room for either.
`maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
input. Boot-shape OOM laddering is a different problem
([[Llama Cpp Serving Sizing]]).
@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
- **Hard-fail keys** — the whole unit silently fails to generate
(`systemctl --user cat <unit>` shows "could not be found"): any
dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
in group 'Container'`); `Command=` is unsupported, use `Exec=`.
- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
generator log `unsupported key … in group 'Container'` plus
`processing encountered some errors` and generate NOTHING for that
manager. The running containers keep going as orphans, so the failure
first appears on the next deploy as `Unit <x>.service not found` for a
service that is visibly up. Other offenders: any dash-prefixed
`[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
The seam `tests/quadlet_image_pins.bats` fails the build on any
generator-rejected key in any host's units.
Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
`Exec=` — repeating the binary makes it a positional arg
(`unexpected /bin/blackbox_exporter`, live 2026-10-21).
- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
spaces into the healthcheck argv; a quoted probe
`/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
"unsupported key"`, `systemctl --user cat probe.service` (shows the
generated ExecStart verbatim), then remove + reload.
## 4a. Image pins (fleet law, 2026-10-21)
Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
locally built images (`localhost/*`, `franksdank/*`) are exempt — their
build is the pin. A rolling tag lets a rebuild move under a running
service with nobody approving it, and it makes a CVE audit ungradeable.
Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
version the config label carries in the comment beside it.
## 5. Rootless Quadlet log retrieval
Step 0 — establish ownership: `systemctl --user status <name>.service`
@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
docs, unit docstrings, and the repo's rule file (the most expensive place to
be stale).
## Three rules that outlive the incident that taught them
* **A target can read down beside a healthy service.** Read `lastError` from
`/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
host volume: silences + notification log, [[Alertmanager]]).
* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
[[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
the process shape can change between two censuses ten minutes apart.
## Order and verification
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
## Recommended additions
Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
Quantized preferred. The iGPU's GTT aperture is not free RAM beside
frank-dragoon — size every pick against *What actually fits beside
frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
never on the command line.
Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
them at GTT capacity, and they double the disk bill).
the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
is not the real budget.
## What actually fits beside frank-dragoon
The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
an APU the aperture and `MemAvailable` are the same physical RAM, and
frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
~10 MB, so the budget for a generation run is roughly what `free` says.
Weights must land inside that budget with slack for compute buffers and
activations (the mmproj-class lesson: price a model at ~1.5× its file set):
| Pipeline (weights) | Fits beside frank-dragoon? |
|---|---|
| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
| Any BF16 combination | No |
| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
**The law: one process owns the memory — same rule as the 3090 bench law.**
For a Qwen-Image render: stop and mask frank-dragoon
(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
`Restart=always` from firing mid-render), generate, unmask, verify
`curl 192.168.0.143:7770/health`. The proven workflows above then run with
the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.
## See Also
@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
- Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
- The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
- The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
memory.** On the APU it is system RAM, and frank-dragoon pins most of it
(`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
stop-mask-render-verify procedure and the fit table are in
[[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.
## See Also
+26 -10
View File
@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
The picker is built from the `model` table in `webui.db` (one row per model)
plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
connection discovery — so the picker shows exactly what is seeded there.
As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
`shared/homepage/services.json`)*.
Since 2026-10-21 the picker exposes **all three fleet models to every
account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
`frank-archon` to everyone; the other two were already public). Proven with
a `role=user` token: `/api/models` lists all three ids and a chat turn to
`frank-archon` answers end-to-end.
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
|---|---|---|---|---|
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
that day's payload swap (see [[frank-zealot]] for what was deployed and when).
The `model` table carries exactly two decoration rows, `frank-zealot` and
`frank-dragoon`. Both the context length and the description live **inside
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
The `model` table carries three decoration rows — `frank-zealot`,
`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
Both the context length and the description live **inside JSON columns** —
`params` holds `{"context": {"length": …}}` and `meta` holds
`{"description": …}`; there is no `context_length` or `description` column, so
`sqlite3` one-liners against those names fail. `params.context.length` is a copy
of the endpoint's per-slot window, so the authoritative number lives on the
@@ -380,6 +382,20 @@ neither.
name: /selected model/i })`) and the welcome text "How can I help you
today?"; that snapshot is also the cheapest proof the picker is populated.
- **2026-10-21 — picker opened to all three models, for everyone.** Per
Frank: `frank-archon` accessible to everyone. Three-leg repair on
`webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
`enable: true`, a `model` row `frank-archon` (params
`{"context":{"length":200000}}`, meta description from the endpoint card),
and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
role row. Verified as a plain `role=user` token: `/api/models` lists all
three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
come from **PID 1's environ** (`podman exec` sessions do not inherit the
app process env; the `.webui_secret_key` file happens to equal it — use it
explicitly, `podman exec -e`).
## See Also
- [[Dankclaw Stack]]
+2 -2
View File
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
tags:
- dankserver
- plex
timestamp: 2026-09-15
timestamp: 2026-10-21
---
# Plex
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
| Item | Value |
|------|-------|
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
@@ -3,7 +3,7 @@ type: reference
title: Observability Provisioning
description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
tags: [observability, prometheus, grafana, provisioning, deployment]
timestamp: 2026-10-05T00:00:00-07:00
timestamp: 2026-10-21T00:00:00-07:00
---
# Observability Provisioning
@@ -20,6 +20,8 @@ committed.
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
Targets and datasources reference **container names** (`grafana:3000`
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
[[Stale Signal Audit]]).
HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
deployed 2026-10-21; see [[Stale Signal Audit]]).
## The deploy path
@@ -43,8 +45,9 @@ from `deploy_observability`:
- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
the backend (`http://prometheus:9090`), not loopback.
**Add a dashboard** — drop the JSON into
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
unified storage, not (only) the SQLite table — grep the API, not the DB, when
verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
2026-10-18: the storage-backend claim needs an admin password to confirm
against `grafana.db` and the API side by side)*
`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
the next `--service grafana` deploy provisions it. This is a repaired law, not
a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
`containers` board — merged, tested, documented as "working" — had **never
been provisioned**; the other boards existed on disk only because someone had
hand-copied them once. A repo JSON is not a live dashboard until the copier
carries it, and `deploy.sh` cannot see the gap because it only reports the
copy it performed.
## Verify through the APIs, not the filesystem
@@ -95,90 +101,32 @@ against `grafana.db` and the API side by side)*
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
# grafana: datasource + dashboard provisioned
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
# account is the Authentik RP login). The proven surface is a real browser
# session: warm the Anubis challenge on the vhost root, then hit
# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
```
The two Prometheus queries were run on 2026-10-18 and are what every number on
this page comes from: 4 alerting + 2 recording rules, and `up` for
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
listed in [[Prometheus]]. The Grafana query needs the local admin password and
was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
proof of installation only — see the next paragraph.
`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
only — see *Add a dashboard* for why a repo JSON is not a live board.
Health answers from a stale process and provisioned resources can hide in
unified storage — a file existing on disk is not proof the running server
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
table (refused = outage, 404/HTML = config defect).
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
via `host.containers.internal`) → signal-cli, addressed by number
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
`curl -s http://192.168.0.143:18093/health` → `ok`, and
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
the blackbox exporter. See [[Prometheus]] → Targets.
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
is started with the flag, so the scrape was refused at the protocol layer (the
flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
stayed firing:
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
metrics opt-in landed that day; the target
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
How many payload generations it spanned is not readable from any host
artifact. *(unverified 2026-10-18)*
Two laws fall out of it. A target can read down beside a healthy service, so
read `lastError` from `/api/v1/targets` before believing an outage. And rule
state lives in Prometheus' process, not on disk — a restart re-stamps
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
`--storage.path=/alertmanager` on a host volume keeps silences and the
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
`/opt/alertmanager/data`.)
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
([[frank-dragoon]]), which is why
`frank-dragoon` has always been green. The active-alert count is a
moment-in-time value and should never be documented as a constant:
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
minutes later, and was empty again by 11:56Z; the relay logged the matching
firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
production `frank-zealot` root unit was stopped **on purpose** for a benchmark
window on the 3090 — the repo Benchmarking Policy says stop the production
service before benching — and `:7771` answered from a manually booted candidate
arm (`systemctl is-active frank-zealot` → `failed` after
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
`TimeoutStopSec`, the known shutdown shape for these engines). So
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
and it is the repo contract.
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
reinstalls and starts it, restoring boot persistence and self-healing. Never
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
changed between two censuses ten minutes apart.
## Related
- [[Observability]] — stack topology and the alert delivery chain
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
down-beside-healthy / alert-state-persistence / candidate-arm rules
- [[Quadlet Lifecycle]] — the units that host these services
- [[Deploy State Triage]] — how the planner selects these services
@@ -67,10 +67,26 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
the wildcard — see [[Node-Exporter]].
The relay's bind is the one address a bridge container can reach on the host:
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
and the podman bridge subnet `10.89.0.0/16` before the DROP
(`scripts/deploy-remote.sh:255-262`).
**Same-host reachability from a bridge container is measured, not inferred**
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
(the podman gateway) reaches host services whether they bind host-net or are
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
every same-host Prometheus target uses the gateway and only cross-host targets
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
relay at `192.168.0.143:18093` is reached from alertmanager's network only
while `install_signal_relay`'s INPUT rules are in force: loopback, the host
itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
trailing blanket DROP. The law the poisoned host taught (2026-10-21):
`iptables -I` prepends, so inserting the DROP last lands it first — the old
install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
deployer's health gate failed beside a LISTENing socket, and the drain
aborted twice before `nft list ruleset` named the cause. The install now
rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
DROP appended); verify any change to it with the sandboxed bats case, not
with `-C` greps — `-C` answers existence, never order. Never point a scrape
target at a host LAN IP from its own host on the strength of one working
example; dial the gateway and probe.
## Deployment
@@ -87,13 +103,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
[[Observability Provisioning]] for what is provisioned as code and how to
verify it through the APIs.
## Current state (2026-10-20)
## Current state (2026-10-21)
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
and both node exporters. The four downs are structural (the `comfyui` and
`services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
every job up — the three LLM endpoints (frank-archon on `vllm:*`,
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
downs are retired — see [[Prometheus]] → Targets.
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -101,12 +118,15 @@ verify it through the APIs.
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
relay, and `journalctl --user -u signal-relay` records successful
`sent to +12062476336:` lines for every rule in the set.
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
`containers`, `observability` and `logs-overview` dashboards, all
installed from `shared/grafana/provisioning/`. The Loki datasource is how
you read logs: Loki has no web UI of its own, so Grafana Explore is the
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
`llm-fleet`, `host-system`, `containers`, `observability` and
`logs-overview` boards. Until 2026-10-21 the copier carried only
`llm-fleet.json` by name and the `containers` board had never been
provisioned; the deployer now globs the directory. The Loki datasource is
how you read logs: Loki has no web UI of its own, so Grafana Explore is the
human surface (the guarded `loki.franksdank.cloud` vhost serves only the
API). Verify method and the copier law: [[Observability Provisioning]].
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
per-container CPU/memory here, and on this rootless-Podman host it works on one
@@ -131,26 +151,26 @@ verify it through the APIs.
`power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
a rootless Alloy cannot read the `0640` journal files, and the container-log
globs point at a root-storage layout that does not exist here. The fix is built
and tested in-repo (2026-10-20): Alloy is now a **root system service on
`Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
lines and pushed 26,966 with zero drops. The **live** instance is still the old
rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
* **Not configured:** no blackbox exporter (the structural downs have no real
uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
not yet deployed (see *Logs* above).
* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
the container-log globs pointed at a root-storage layout that does not exist
here) — fixed 2026-10-20 by making Alloy a **root system service on
`Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
but the stream carried only `job`/`service_name`, so every per-container
view was structurally empty: Loki metrics queries group only by INDEXED
stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
label (plus a `host` label) at ingestion — `loki.process` in
`shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
purpose-test law in [[Observability Provisioning]].
## Citations
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
@@ -42,29 +42,29 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
**No job reads down structurally any more** (the three permanent-reds were
retired 2026-10-21; census re-run after each deploy):
| Job | Target | `health` | `lastError`, verbatim |
|-----|--------|----------|------------------------|
| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
| Job | Target | how |
|---|---|---|
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
Scraping a web root is not a health check: an HTML page fails the scrape
protocol (`unsupported Content-Type`), which is what kept those targets red
beside healthy services. The classification rule and the honesty law live in
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
that gateway lands on the host's loopback listener. The gateway address is
not itself bindable from the host netns.
Plain targets stay `host:port`, never URLs — the probe job's URL targets are
the documented exception, relabeled onto the prober.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
The remedy for a probe-shaped question is the **blackbox exporter**; the
classification rule and the honesty law for it live in [[Stale Signal Audit]].
Until the exporter is deployed, these three stay red for a structural reason:
deleting them would
hide a real service, and fixing them by publishing databases or enabling metrics
would trade a wrong signal for a real exposure. Targets are `host:port`, never
URLs.
## Volumes
| Container path | Host path |
@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
reserves the registry price, not the actual pool. Raising the pool consumes
real free VRAM without growing the arena; the registry over-prices the pool
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
block-alignment loss when converting a token target to bytes). The ask lands
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
real free VRAM without growing the arena; the registry OVER-PRICES the pool
(era G measured ~14 % under the planned B/token — the gap is the lesson, not
the number; **never reuse a B/token ratio across geometries**), and there is a
~4 MB block-alignment loss to subtract when converting a token target to
bytes. The ask lands block-aligned: verify against the engine's own
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
**Driver-reported free VRAM after hours of traffic is NOT allocatable
arena.** It is the estimator gap: the registry over-prices KV and charges
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
**KV accounting — pin the pool BYTES, floor-sized:** set
`--kv-cache-memory-bytes` PER RANK to the contract floor
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
come out of the arena/headroom and buy nothing. Measured B/token on this
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
planner's KV reserve is for ONE max-model-len request.
come out of the arena/headroom and buy nothing. The B/token rate is a property
of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
Read your live rate from the boot ledger every time; the decomposition (a
dtype-independent mamba/side stream rides on top of the attention KV) is why
the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
is for ONE max-model-len request.
**Parallelism ceilings you cannot flag past:**
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
+ engine flags in the log.
**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
| shape | util | graphs | arena | result |
|---|---|---|---|---|
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
~207k per context max.
## 7. Arena audit procedure (when asked for "more experts")
+34 -21
View File
@@ -10,14 +10,15 @@ tags:
- gfx1151
- halogen
- endpoint
timestamp: 2026-10-19
timestamp: 2026-10-21
---
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
## Current shape
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
unauthenticated token port 8730 stays on container loopback. Engine stdout →
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
`HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
`HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
(`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
- Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
(Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
Caddy/observability).
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
(= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
`reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
rides [[frank-zealot]].
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
— sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
`tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
`high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
- Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
@@ -124,14 +128,18 @@ created it — earlier topologies in .
prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
re-measure the warm ratio after any pool change.
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
`tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
- **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
(`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
return as a crash-loop.
2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
`/usr/lib/x86_64-linux-gnu`, where all three exist.
2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
`libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**
+31 -1
View File
@@ -17,6 +17,9 @@ logging {
// Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
// host network, so it pushes by loopback, never by container name — a root
// container is not on the rootless dankclaw_default network.
// GRAFANA_HOST_LABEL is replaced with this host's name by
// copy_alloy_config at deploy time; it must not appear verbatim in the
// installed config.
loki.write "default" {
endpoint {
url = "http://127.0.0.1:3100/loki/api/v1/push"
@@ -32,7 +35,34 @@ loki.write "default" {
loki.source.journal "journal" {
format_as_json = true
labels = {
job = "systemd-journal",
job = "systemd-journal",
host = "GRAFANA_HOST_LABEL",
}
forward_to = [loki.process.relabel.receiver]
}
// Promote the fields that ARE the point of the log stream to real Loki labels.
// Without this the journal arrives as one opaque JSON line per entry and every
// per-container view is structurally impossible: Loki metrics queries can only
// group by INDEXED stream labels, so `sum by (container_name)` returns a single
// anonymous series and the logs-overview board read empty beside 6.5k
// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
// podman's journald logging writes (uppercase); host units lack it and stay on
// the host-only stream. `format_as_json = true` above guarantees every line
// reaching this stage is a JSON object, so there is no non-JSON branch to
// protect — alloy 1.21's json stage has no `on_error` attribute (validated
// against the pinned image); a malformed line would drop, which is the
// correct fate for a line Loki could not index anyway.
loki.process "relabel" {
stage.json {
expressions = {
container = "CONTAINER_NAME",
}
}
stage.labels {
values = {
container = "container",
}
}
forward_to = [loki.write.default.receiver]
}
+17
View File
@@ -0,0 +1,17 @@
# Blackbox exporter modules — the honest answer to "is this HTTP service
# answering?". A Prometheus scrape cannot express it (an HTML front page is a
# 200 that fails the scrape protocol), which is why the old `services` and
# `comfyui` jobs sat permanently red on `unsupported Content-Type`.
# See serverAppConfigs-docs/Architecture/Stale Signal Audit.md §1.
# The listen address is NOT set here: the quadlet unit's Exec passes
# --listen-address (LAN bind — see the unit header for why loopback is
# unreachable from the bridge-networked Prometheus). Keep this file to
# modules only, so the two sources cannot disagree.
modules:
http_2xx:
prober: http
timeout: 10s
http:
preferred_ip_protocol: ip4
valid_status_codes: [] # default: 2xx + 3xx
@@ -104,14 +104,14 @@
{
"id": 6,
"type": "timeseries",
"title": "Memory vs limit (%)",
"description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.",
"title": "Memory vs host total (%)",
"description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
"targets": [
{
"expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)",
"expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
"legendFormat": "{{host}} / {{container}}",
"refId": "A"
}
@@ -54,7 +54,7 @@
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
"targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
"targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
},
{
"id": 4, "type": "stat", "title": "Open file handles",
@@ -194,7 +194,7 @@
"id": 6,
"type": "timeseries",
"title": "Firing fleet alerts",
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
"gridPos": {
"h": 8,
"w": 12,
@@ -2,74 +2,242 @@
"uid": "logs-overview",
"title": "Logs \u2014 Loki (systemd journal)",
"description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
"tags": ["logs", "loki"],
"tags": [
"logs",
"loki"
],
"timezone": "browser",
"schemaVersion": 39,
"refresh": "30s",
"time": { "from": "now-1h", "to": "now" },
"time": {
"from": "now-1h",
"to": "now"
},
"templating": {
"list": [
{
"name": "container",
"label": "container",
"type": "query",
"datasource": { "type": "loki", "uid": "loki" },
"query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
"datasource": {
"type": "loki",
"uid": "loki"
},
"query": {
"query": "label_values({job=\"systemd-journal\"}, container)",
"refId": "LokiVariableQueryEditor-VariableQuery"
},
"includeAll": true,
"multi": true,
"refresh": 2,
"current": { "text": "All", "value": "$__all" }
"current": {
"text": "All",
"value": "$__all"
}
}
]
},
"panels": [
{
"id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
"description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
"gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
"datasource": { "type": "loki", "uid": "loki" },
"fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
"targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
"id": 1,
"type": "timeseries",
"title": "Log volume by container (lines/s)",
"description": "Per-container journal line rate. The `container` label is promoted at ingestion by Alloy (loki.process) from podman's CONTAINER_NAME journald field - Loki metrics can only group by indexed stream labels.",
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 0
},
"datasource": {
"type": "loki",
"uid": "loki"
},
"fieldConfig": {
"defaults": {
"unit": "reqps",
"custom": {
"fillOpacity": 20,
"stacking": {
"mode": "normal"
}
}
},
"overrides": []
},
"targets": [
{
"expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} [$__auto]))",
"legendFormat": "{{container_name}}",
"refId": "A"
}
]
},
{
"id": 2, "type": "timeseries", "title": "Total log rate",
"id": 2,
"type": "timeseries",
"title": "Total log rate",
"description": "All journal lines per second.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
"datasource": { "type": "loki", "uid": "loki" },
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
"targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 9
},
"datasource": {
"type": "loki",
"uid": "loki"
},
"fieldConfig": {
"defaults": {
"unit": "reqps"
},
"overrides": []
},
"targets": [
{
"expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))",
"legendFormat": "all lines/s",
"refId": "A"
}
]
},
{
"id": 3, "type": "timeseries", "title": "Error-level line rate",
"id": 3,
"type": "timeseries",
"title": "Error-ish line rate (text match)",
"description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
"datasource": { "type": "loki", "uid": "loki" },
"fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
"targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 9
},
"datasource": {
"type": "loki",
"uid": "loki"
},
"fieldConfig": {
"defaults": {
"unit": "reqps",
"color": {
"mode": "fixed",
"fixedColor": "red"
}
},
"overrides": []
},
"targets": [
{
"expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))",
"legendFormat": "error lines/s",
"refId": "A"
}
]
},
{
"id": 4, "type": "timeseries", "title": "Errors by container",
"id": 4,
"type": "timeseries",
"title": "Errors by container",
"description": "Which unit is throwing. Rank by the top offenders.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
"datasource": { "type": "loki", "uid": "loki" },
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
"targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 17
},
"datasource": {
"type": "loki",
"uid": "loki"
},
"fieldConfig": {
"defaults": {
"unit": "reqps"
},
"overrides": []
},
"targets": [
{
"expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} |~ \"(?i)(error|panic|fatal)\" [$__auto]))",
"legendFormat": "{{container_name}}",
"refId": "A"
}
]
},
{
"id": 5, "type": "stat", "title": "Error lines (last 15m)",
"id": 5,
"type": "stat",
"title": "Error lines (last 15m)",
"description": "Absolute count over the window, not a rate.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
"datasource": { "type": "loki", "uid": "loki" },
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
"targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 17
},
"datasource": {
"type": "loki",
"uid": "loki"
},
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 50
},
{
"color": "red",
"value": 500
}
]
}
},
"overrides": []
},
"targets": [
{
"expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))",
"instant": true,
"legendFormat": "errors/15m",
"refId": "A"
}
]
},
{
"id": 6, "type": "logs", "title": "Journal (filtered)",
"id": 6,
"type": "logs",
"title": "Journal (filtered)",
"description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
"gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
"datasource": { "type": "loki", "uid": "loki" },
"options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
"targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
"gridPos": {
"h": 12,
"w": 24,
"x": 0,
"y": 25
},
"datasource": {
"type": "loki",
"uid": "loki"
},
"options": {
"showTime": true,
"wrapLogMessage": true,
"sortOrder": "Descending",
"enableLogDetails": true
},
"targets": [
{
"expr": "{job=\"systemd-journal\", container=~\"$container\"}",
"refId": "A"
}
]
}
]
}
}
@@ -2,95 +2,384 @@
"uid": "observability",
"title": "Observability \u2014 self-monitoring",
"description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
"tags": ["observability", "meta"],
"tags": [
"observability",
"meta"
],
"timezone": "browser",
"schemaVersion": 39,
"refresh": "30s",
"time": { "from": "now-6h", "to": "now" },
"time": {
"from": "now-6h",
"to": "now"
},
"panels": [
{
"id": 1, "type": "stat", "title": "Targets up",
"gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
"targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
"id": 1,
"type": "stat",
"title": "Targets up",
"gridPos": {
"h": 4,
"w": 4,
"x": 0,
"y": 0
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"targets": [
{
"expr": "count(up == 1)",
"instant": true,
"refId": "A"
}
]
},
{
"id": 2, "type": "stat", "title": "Targets down",
"description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
"gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
"targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
"id": 2,
"type": "stat",
"title": "Targets down",
"description": "Every scrape target that is down. ZERO IS THE HEALTHY STATE: the HTML-probe jobs and the disabled Gitea metrics were retired 2026-10-21 (HTTP liveness now rides the blackbox exporter, job \"services\"). A non-zero count here is a real regression - read /api/v1/targets lastError before triaging.",
"gridPos": {
"h": 4,
"w": 4,
"x": 4,
"y": 0
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
}
},
"overrides": []
},
"targets": [
{
"expr": "count(up == 0)",
"instant": true,
"refId": "A"
}
]
},
{
"id": 3, "type": "stat", "title": "TSDB head series",
"id": 3,
"type": "stat",
"title": "TSDB head series",
"description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
"gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
"targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
"gridPos": {
"h": 4,
"w": 5,
"x": 8,
"y": 0
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"targets": [
{
"expr": "prometheus_tsdb_head_series",
"instant": true,
"refId": "A"
}
]
},
{
"id": 4, "type": "stat", "title": "Discovered targets",
"gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
"targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
"id": 4,
"type": "stat",
"title": "Discovered targets",
"gridPos": {
"h": 4,
"w": 5,
"x": 13,
"y": 0
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"targets": [
{
"expr": "sum(prometheus_sd_discovered_targets)",
"instant": true,
"refId": "A"
}
]
},
{
"id": 5, "type": "stat", "title": "Prometheus mem",
"gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
"targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
"id": 5,
"type": "stat",
"title": "Prometheus mem",
"gridPos": {
"h": 4,
"w": 3,
"x": 18,
"y": 0
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "bytes"
},
"overrides": []
},
"targets": [
{
"expr": "process_resident_memory_bytes{job=\"prometheus\"}",
"instant": true,
"refId": "A"
}
]
},
{
"id": 6, "type": "stat", "title": "Grafana up",
"gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
"targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
"id": 6,
"type": "stat",
"title": "Grafana up",
"gridPos": {
"h": 4,
"w": 3,
"x": 21,
"y": 0
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"mappings": [
{
"type": "value",
"options": {
"1": {
"text": "UP",
"color": "green"
},
"0": {
"text": "DOWN",
"color": "red"
}
}
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": null
},
{
"color": "green",
"value": 1
}
]
}
},
"overrides": []
},
"targets": [
{
"expr": "up{job=\"grafana\"}",
"instant": true,
"refId": "A"
}
]
},
{
"id": 7, "type": "table", "title": "Target health",
"id": 7,
"type": "table",
"title": "Target health",
"description": "Every scrape target and its current up. Sort by Value to surface regressions.",
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
"targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 4
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"options": {
"sortBy": [
{
"displayName": "Value",
"sortOrder": "Asc"
}
]
},
"targets": [
{
"expr": "up",
"format": "table",
"instant": true,
"refId": "A"
}
]
},
{
"id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
"id": 8,
"type": "timeseries",
"title": "Scrape interval drift (p99 actual)",
"description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 4
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
},
"targets": [
{
"expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}",
"legendFormat": "{{instance}}",
"refId": "A"
}
]
},
{
"id": 9, "type": "timeseries", "title": "Scrape samples/s",
"id": 9,
"type": "timeseries",
"title": "Scrape samples/s",
"description": "Total samples ingested across all jobs.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
"targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 13
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"targets": [
{
"expr": "sum(rate(scrape_samples_scraped[5m]))",
"legendFormat": "samples/s",
"refId": "A"
}
]
},
{
"id": 10, "type": "timeseries", "title": "Loki request rate",
"id": 10,
"type": "timeseries",
"title": "Loki request rate",
"description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
"targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 13
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "reqps"
},
"overrides": []
},
"targets": [
{
"expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))",
"legendFormat": "{{route}}",
"refId": "A"
}
]
},
{
"id": 11, "type": "timeseries", "title": "Exporter process CPU",
"id": 11,
"type": "timeseries",
"title": "Exporter process CPU",
"description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
"targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 21
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
},
"targets": [
{
"expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))",
"legendFormat": "{{job}}",
"refId": "A"
}
]
}
]
}
}
+31 -15
View File
@@ -60,20 +60,33 @@ scrape_configs:
- targets: ["loki:3100"]
metrics_path: "/metrics"
# ComfyUI (no native /metrics; probe the UI)
- job_name: "comfyui"
metrics_path: "/"
static_configs:
- targets: ["comfyui:8188"]
# Blackbox-style probes for service up/down (published LAN ports;
# gitea and comfyui are covered by their own jobs above)
# HTTP liveness, probed honestly through the blackbox exporter. Scraping a
# web root is NOT a health check — an HTML front page fails the scrape
# protocol (`unsupported Content-Type`), which is exactly why the old
# `services` and `comfyui` jobs were red forever beside healthy services
# (Stale Signal Audit §1). The prober is a host-netns container listening on
# the podman gateway (the scrape address below - the netavark hairpin gap
# bars this container from its own host's LAN IP, cadvisor job note) and on
# the LAN; from that netns it reaches the host LAN IP, and published ports
# via 127.0.0.1 - bridge container names do NOT resolve in the host netns,
# which is why comfyui's probe is a loopback URL. comfyui's own
# metrics-scrape job is retired, one owner per service.
- job_name: "services"
metrics_path: "/"
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
- "192.168.0.143:3005" # OpenWebUI
- "http://192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
- "http://192.168.0.143:3005" # OpenWebUI
- "http://127.0.0.1:8188" # ComfyUI UI (published 0.0.0.0:8188; probed from the prober's host netns)
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: "host.containers.internal:9115"
# LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
# this container, not the guarded vhosts: these are machine clients and
@@ -140,12 +153,15 @@ scrape_configs:
# system.slice daemons and machine.slice scopes carry no such name and are
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
# the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
# is why node-dankclaw scrapes via host.containers.internal), a published port
# IS reachable from this bridge container at the host LAN address.
# the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
# DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
# connection whether the port is a host-net bind or a published port (the
# original note here claimed published ports were exempt; measured 2026-10-21
# from inside this container: LAN refused, gateway answers). Cross-host
# targets (dankserver) have no hairpin problem and stay plain LAN.
- job_name: "cadvisor-dankclaw"
static_configs:
- targets: ["192.168.0.143:8080"]
- targets: ["host.containers.internal:8080"]
labels:
host: dankclaw
metric_relabel_configs:
+12 -1
View File
@@ -193,7 +193,18 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
}
@test "prometheus targets are host:port (no URL schemes)" {
! grep -nE '^\s*-\s*"?http' "$REPO_ROOT/shared/prometheus/prometheus.yml"
# Plain scrape targets are host:port; a URL target is only meaningful for
# the blackbox prober job, whose relabel_configs copy it to __param_target
# and point __address__ at the prober. Anywhere else a URL target is the
# old HTML-probe defect wearing a new hat.
awk '
/job_name:/ { job=$0; sub(/.*"/, "", job); sub(/".*/, "", job); probe = 0 }
/metrics_path: \/probe/ { probe = 1 }
/^[[:space:]]*-[[:space:]]*"http/ && !probe {
print FILENAME ":" FNR ": URL target outside a prober job: " $0; bad = 1
}
END { exit bad }
' "$REPO_ROOT/shared/prometheus/prometheus.yml"
}
@test "observability containers run as container root (rootless = host user)" {
+45
View File
@@ -125,3 +125,48 @@ load helpers
[ ! -e "$sandbox/frank-default-sightsound" ]
[ -e "$BATS_TEST_TMPDIR/.AGENTTRASH/$(date +%Y%m%d)-retired-deploy-state/frank-default-sightsound" ]
}
@test "the client and remote allowlists agree per target" {
# Two gates guard `deploy.sh --service X`: the client's valid_service and
# the remote runner's service_is_allowed. When only one side gained a
# service the other answered "is not deployed on <target>" and the only
# repair path was a fleet-wide drain — cadvisor and gpu-exporter both fell
# through this crack within a day of each other. The lists MUST agree;
# searxng-network is the one sanctioned exception (client-only network
# unit, no remote dispatch handler).
run env OUT="$BATS_TEST_TMPDIR" bash -c '
grep -oE "dank(claw|server):[a-z0-9-]+" \
<(sed -n "/^valid_service()/,/^}/p" "$1/scripts/deploy.sh") \
| grep -v "dankclaw:searxng-network" | sort -u > "$OUT/cli"
grep -oE "dank(claw|server):[a-z0-9-]+" \
<(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
| sort -u > "$OUT/rem"
diff "$OUT/cli" "$OUT/rem"
' _ "$REPO_ROOT"
[ "$status" -eq 0 ] || { echo "deploy.sh vs deploy-remote.sh allowlist drift:"; echo "$output"; return 1; }
# Drift aside, the gate must actually know the host-process exporters —
# their first-ever deploy has to pass the client gate.
for pair in dankclaw:cadvisor dankclaw:gpu-exporter dankserver:cadvisor dankserver:gpu-exporter; do
grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
done
}
@test "every gated dankclaw service is dispatched" {
# A service can clear both allowlists and still die inside the deployer:
# dankclaw has no compose file, and deploy_compose_service (the case
# catch-all) hard-fails with "no quadlet handler" since 2026-09-16. So
# every name the remote gate allows for dankclaw MUST be named by a
# deploy_service pattern — blackbox-exporter (2026-10-21) shipped with
# both gates and the health gate wired but no dispatch case, and only
# the deploy attempt found it.
run env OUT="$BATS_TEST_TMPDIR" bash -c '
grep -oE "dankclaw:[a-z0-9-]+" \
<(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
| cut -d: -f2 | sort -u > "$OUT/gated"
sed -n "/^deploy_service()/,/^}/p" "$1/scripts/deploy-remote.sh" \
| grep -oE "^ +[a-z][a-z0-9|-]*\)" | tr -d " )" | tr "|" "\n" \
| sort -u > "$OUT/dispatched"
comm -23 "$OUT/gated" "$OUT/dispatched"
' _ "$REPO_ROOT"
[ -z "$output" ] || { echo "gated dankclaw services with no deploy_service case:"; echo "$output"; return 1; }
}
+50 -35
View File
@@ -3,7 +3,7 @@
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE
# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
# shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
#
@@ -85,7 +85,7 @@ good_env() {
"FRANK_DRAGOON_VISION_BYTES=0"
"FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
"HALOGEN_MODEL_ID=frank-dragoon"
"HALOGEN_CTX=200704"
"HALOGEN_CTX=262144"
"HALOGEN_KV_POOL_POSITIONS=401408"
"HALOGEN_KV_SLOTS=2"
"HALOGEN_SPEC_ADAPT=0"
@@ -159,9 +159,11 @@ good_env() {
reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
# The pool is ONE budget behind the slots (boot ledger form: "2 slots over
# ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
# a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
# (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
# resident at once — frank-zealot parity on total in-flight context. What
# a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
# native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
# MAXIMUM harness-length (200,000-token) conversations are resident at once;
# two literal ceiling-length requests would queue — widening that is a pool
# raise, not a window move.
# still binds: the pool must back ONE full-length request (below that the
# advertised per-request ceiling would be unreachable, hence pool >= ctx),
# it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
[[ "$output" == *"--name frank-dragoon-halogen"* ]]
[[ "$output" == *"--publish 7770:8731"* ]]
[[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
[[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
[[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
[[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
[[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
[[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"
@test "the shipped env keeps the harness fit law inside the ceiling" {
# HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
# naming the limit, never a silent truncation). The law is INVARIANT to
# the value: the harness window must sit ~45k UNDER it, because the
# harness clamps max_tokens = min(45056, window − est − 4096), compacts
# at 85 % of its declared window, and its chars/4 est UNDER-counts the
# rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
# est 152,938 on 2026-10-07 — with window == ceiling, every request over
# ~65 % of the window was a guaranteed 400 before compaction could fire).
# The shipped pair is ctx 200704 / window 155648 (lowered from
# 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
# monotone maxima of est + budget (est ≤ window − 4096):
# full budget at the compaction threshold: 1.03*132300 + 45056
# clamped budget at the clamp's zero point: 1.03*151552 + 0
# and the compaction request that rides on frank-zealot (its
# 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
ctx=200704
hw=155648; mt=45056
thr=$(( hw * 85 / 100 ))
# naming the limit, never a silent truncation). The law: the harness
# window must sit ~45k UNDER the ceiling, because the harness clamps
# max_tokens = min(45056, window − est − 4096), compacts at the trigger
# percent of its declared window, and its chars/4 est UNDER-counts the
# rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
# at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
# request is a guaranteed 400 before compaction can fire). The 2026-10-21
# pair: ceiling 262144 (native) under a harness window of 200,000.
# The declared window is read from the endpoint page — the one owner —
# so env, models.yml and page that disagree FAIL HERE, not as a 400
# mid-session (the recurring failure this seam exists for).
grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
grep -qE 'contextWindow: 200000' "$page"
hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
mt=45056
# Invariant checked at the live 95 % trigger (config.yml
# thresholdPercent) — strictly stronger than the 85 % design point.
# Both monotone maxima of est + budget (est ≤ window − 4096):
thr=$(( hw * 95 / 100 ))
[ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
zero=$(( hw - 4096 ))
[ $(( zero * 103 / 100 )) -le "$ctx" ]
# The compaction request rides on frank-zealot's 200,704-token slot:
# input ≈ threshold − keepRecent(20k).
[ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
}
@@ -423,9 +430,12 @@ npu_fixture() {
# the stock root:render 0660 mode, and the serve gate checks exactly that.
touch "$TEST_TMP/accel0"
chmod 666 "$TEST_TMP/accel0"
mkdir -p "$TEST_TMP/xrt"
# basename carries a multiarch shape: the third mount prefix is derived
# from the XRT dir's basename (the NPU child searches
# /opt/xilinx/lib/<multiarch>/, measured 2026-10-21).
mkdir -p "$TEST_TMP/x86_64-linux-gnu"
for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
touch "$TEST_TMP/xrt/$l"
touch "$TEST_TMP/x86_64-linux-gnu/$l"
done
mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
@@ -435,7 +445,7 @@ npu_fixture() {
"HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
"FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
"FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
"FRANK_DRAGOON_XRT_DIR=$TEST_TMP/xrt"
"FRANK_DRAGOON_XRT_DIR=$TEST_TMP/x86_64-linux-gnu"
"FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
)
}
@@ -465,11 +475,15 @@ $(cat "$TEST_TMP/serve.log")"
! grep -q 'accel0' "$TEST_TMP/serve.log"
}
@test "the lane arms the accel node, the staged model dir and each XRT library twice" {
# Each library mounts at the image's XRT path AND at its own path: the
# distro libraries find each other where they were installed, and an
# /opt/xilinx/xrt of symlinks into the system dir dangles inside the
# container (the vendor calls that out by name).
@test "the lane arms the accel node, the staged model dir and each XRT library three times" {
# Each library mounts at the image's XRT path, at its own path (the distro
# libraries find each other where they were installed), and at
# /opt/xilinx/lib/<multiarch>/ — the prefix the NPU child searches and
# names when it is missing (`No such library
# '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'`, measured
# 2026-10-21: the lane died and the supervisor took the whole container
# with it, 88-restart loop). An /opt/xilinx/xrt of symlinks into the
# system dir dangles inside the container (the vendor calls that out).
podman_stub
good_env
npu_fixture
@@ -484,8 +498,9 @@ $(cat "$TEST_TMP/serve.log")"
[[ "$output" != *":/models/npu"* ]]
[[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
[[ "$output" == *"-v $TEST_TMP/xrt/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
[[ "$output" == *"-v $TEST_TMP/xrt/$l:$TEST_TMP/xrt/$l:ro"* ]]
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:$TEST_TMP/x86_64-linux-gnu/$l:ro"* ]]
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/lib/x86_64-linux-gnu/$l:ro"* ]]
done
# The Flash payload still rides the same container: the lane is additive.
[[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
@@ -560,7 +575,7 @@ $(cat "$TEST_TMP/serve.log")"
podman_stub
good_env
npu_fixture
rm "$TEST_TMP/xrt/libxrt_driver_xdna.so.2"
rm "$TEST_TMP/x86_64-linux-gnu/libxrt_driver_xdna.so.2"
run_with_log
[ "$status" -ne 0 ]
[[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]
+2 -1
View File
@@ -77,6 +77,7 @@ FRANKBOOK_UNITS=(
}
@test "readarr keeps its pinned image" {
grep -q '^Image=lscr.io/linuxserver/readarr:amd64-1.0.980$' \
# 2026-10-21 fleet law: version tag AND digest (tests/quadlet_image_pins.bats)
grep -qE '^Image=lscr\.io/linuxserver/readarr:amd64-1\.0\.980@sha256:[0-9a-f]{64}$' \
"$REPO_ROOT/frankbook-pro/readarr/quadlet/readarr.container" || return 1
}
+30
View File
@@ -195,3 +195,33 @@ for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
print('gpu dashboard panels ok')
"
}
@test "--serve answers on EVERY declared bind, not just the first" {
# The deployed unit serves loopback + the host LAN address, and Prometheus
# reaches the host through the gateway — which dials the LAN-class bind.
# A server that only runs serve_forever() on the first socket leaves the
# second one listen()ing-but-never-accepting: the deploy health gate on
# loopback passes, every scrape on the other bind dies on the SYN backlog,
# and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
# left both gpu-* targets down beside a green deploy.)
mk_amd 0 42 1073741824 34359738368 "" 71000
port=$((19400 + RANDOM % 500))
python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
--bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
> "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
local pid=$!
trap 'kill "$pid" 2>/dev/null' EXIT
local ok=0
for _ in $(seq 1 20); do
curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
sleep 0.25
done
[ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
# Both binds must ANSWER within 2 s each — the second one is the bug.
for b in 127.0.0.1 127.0.0.2; do
curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
|| { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
done
kill "$pid" 2>/dev/null
trap - EXIT
}
+195 -9
View File
@@ -21,7 +21,7 @@ load helpers
UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager)
UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager blackbox-exporter)
# container_section <unit> — the [Container] key/value block only
container_section() {
@@ -33,7 +33,7 @@ service_section() {
awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
}
@test "all seven observability units exist under dankclaw/observability/quadlet/" {
@test "all eight observability units exist under dankclaw/observability/quadlet/" {
local svc
for svc in "${UNITS[@]}"; do
[ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
@@ -559,6 +559,64 @@ assert any('ALERTS' in e for e in exprs), exprs
grep -q -- '--dport 18093 -j DROP' "$handler"
}
@test "the deployer installs the relay firewall with the ACCEPTs deciding before the DROP" {
# Behavioral, run for real against a sandbox that plays the kernel: the
# rule set is a LIST and iptables -I prepends, so the shape this function
# had until 2026-10-21 (-C/-I ACCEPT x3, then -C/-I the blanket DROP)
# landed the DROP at position 1 and every packet died there — the drain
# failed its health gate beside a LISTENing socket twice. Seeding the
# sandbox with that poisoned order also pins the convergent repair: a
# re-deploy must rewrite the tail, not skip because -C finds every rule.
local sandbox="$BATS_TEST_TMPDIR/fw"
mkdir -p "$sandbox/bin"
cat > "$sandbox/bin/iptables" <<'EOF'
#!/usr/bin/env bash
# ordered rule list in $FW_STATE: -C lookup, -I at head, -A at tail, -D delete.
state="${FW_STATE:?}"
touch "$state"
op="$1"; shift
case "$op" in
-C) grep -Fxq -- "$*" "$state" ;;
-I) grep -Fxq -- "$*" "$state" || { { echo "$*"; cat "$state"; } > "$state.new"; mv "$state.new" "$state"; } ;;
-A) grep -Fxq -- "$*" "$state" || echo "$*" >> "$state" ;;
-D) if grep -Fxq -- "$*" "$state"; then
grep -Fxv -- "$*" "$state" > "$state.new"; mv "$state.new" "$state"
else exit 1; fi ;;
*) exit 0 ;;
esac
EOF
chmod +x "$sandbox/bin/iptables"
# Poisoned state exactly as the pre-fix deployer left Dankclaw.
printf '%s\n' \
'INPUT -p tcp --dport 18093 -j DROP' \
'INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT' \
'INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT' \
'INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT' \
> "$sandbox/rules"
(
cd "$REPO_ROOT"
export PATH="$sandbox/bin:$PATH" FW_STATE="$sandbox/rules" HOME="$sandbox/home"
mkdir -p "$HOME"
eval "$(sed -n '/^install_signal_relay()/,/^}/p' scripts/deploy-remote.sh)"
run_privileged() { "$@"; }
systemctl() { :; }
curl() { return 0; }
sleep() { :; }
install_signal_relay
)
# Exactly one blanket DROP, and it sits after every ACCEPT.
[ "$(grep -cF -- 'INPUT -p tcp --dport 18093 -j DROP' "$sandbox/rules")" -eq 1 ]
local drop_at total
drop_at="$(grep -nF -- 'INPUT -p tcp --dport 18093 -j DROP' "$sandbox/rules" | cut -d: -f1)"
total="$(wc -l < "$sandbox/rules")"
[ "$drop_at" -eq "$total" ]
local s seen
for s in 127.0.0.1 192.168.0.143 10.89.0.0/16; do
seen="$(grep -nF -- "-s $s -j ACCEPT" "$sandbox/rules" | cut -d: -f1)"
[ -n "$seen" ] && [ "$seen" -lt "$drop_at" ]
done
}
@test "the planner selects alertmanager for its config and relay files" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
--paths shared/prometheus/alertmanager.yml
@@ -595,16 +653,22 @@ assert any('ALERTS' in e for e in exprs), exprs
}
@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
# Cross-file contract: cadvisor publishes the LAN port; a cadvisor scrape
# job pointing anywhere else reads nothing (unlike node-exporter's host-net
# bind, a published port IS reachable from this bridge container at the LAN
# address). And the whole dashboard depends on the id -> container relabel:
# cadvisor names nothing itself, the quadlet unit name is only in the cgroup
# path. If either half changes alone, the Containers dashboard is blank.
# Cross-file contract, measured 2026-10-21 from inside the Prometheus
# container: a bridge container cannot dial its OWN host's LAN IP — the
# hairpin refuses published ports exactly like host-net binds, so
# cadvisor-dankclaw must scrape via the podman gateway (the original pin
# here demanded the LAN address and shipped a permanently-down target).
# dankserver is a different machine: plain LAN there. And the whole
# dashboard depends on the id -> container relabel: cadvisor names nothing
# itself, the quadlet unit name is only in the cgroup path. If either half
# changes alone, the Containers dashboard is blank.
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
grep -q '"192.168.0.143:8080"' "$cfg" || { echo "cadvisor-dankclaw does not scrape the published LAN port"; return 1; }
grep -q '"host.containers.internal:8080"' "$cfg" \
|| { echo "cadvisor-dankclaw does not scrape via the podman gateway"; return 1; }
! grep -q '"192.168.0.143:8080"' "$cfg" \
|| { echo "cadvisor-dankclaw dials its own host LAN IP — hairpin refuses it"; return 1; }
grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
# The relabel that turns …/app.slice/<name>.service… into a `container`.
grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
@@ -638,3 +702,125 @@ assert not any('container_network' in e or 'container_fs_' in e for e in names),
[ "$status" -eq 0 ]
[ "$output" = "cadvisor" ]
}
@test "the grafana copier provisions EVERY dashboard json, not a hand-listed subset" {
# Behavioral, not a text grep: the copier is extracted and executed with a
# stubbed run_privileged against a sandbox. The fleet provider turns each
# file in its directory into a dashboard, so a file the copier skips is a
# dashboard that silently never exists (the containers board lived in the
# repo, never reached /opt, and its fixes were undeployable).
# BATS_TEST_TMPDIR: per-test, removed by bats itself; a mktemp dir with a
# RETURN trap races bats' own trap machinery and vanishes mid-function.
local sandbox="$BATS_TEST_TMPDIR/sandbox"
(
cd "$REPO_ROOT"
eval "$(sed -n '/^copy_grafana_provisioning()/,/^}/p' scripts/deploy-remote.sh)"
run_privileged() { "$@"; }
export GRAFANA_PROV_ROOT="$sandbox/opt/grafana"
copy_grafana_provisioning
)
local f
for f in "$REPO_ROOT"/shared/grafana/provisioning/dashboards/*.json; do
[ -f "$sandbox/opt/grafana/provisioning/dashboards/$(basename "$f")" ]
done
[ -f "$sandbox/opt/grafana/provisioning/dashboards/provider.yml" ]
[ -f "$sandbox/opt/grafana/provisioning/datasources/prometheus.yml" ]
}
@test "every observability unit declares an Image" {
# Quadlet refuses to GENERATE a service for a .container without Image=
# - the deployer then dies at `systemctl restart` with "Unit not found"
# and no diagnosis (blackbox-exporter, live 2026-10-21: merged with the
# copier, gates and health check wired, but no Image line).
local svc
for svc in "${UNITS[@]}"; do
grep -q '^Image=' "$UNIT_DIR/$svc.container" \
|| { echo "$svc.container has no Image= - quadlet will not generate it"; return 1; }
done
}
@test "changes under shared/blackbox/ plan the prober" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
--paths shared/blackbox/blackbox.yml
[ "$status" -eq 0 ]
[ "$output" = "blackbox-exporter" ]
}
@test "the copier runs the same code path the deployer runs" {
# Textually pinned so a refactor cannot quietly route the deployer around
# the copy_* functions the seams execute.
grep -q ' prometheus)' "$REMOTE_DEPLOY"
grep -q 'copy_prometheus_config' "$REMOTE_DEPLOY"
grep -q 'copy_alertmanager_config' "$REMOTE_DEPLOY"
grep -q 'copy_loki_config' "$REMOTE_DEPLOY"
grep -q ' blackbox-exporter)' "$REMOTE_DEPLOY"
grep -q 'copy_blackbox_config' "$REMOTE_DEPLOY"
}
@test "the alloy copier installs the host-labeled config, token-free" {
# Executes copy_alloy_config for real against a sandbox (stubbed
# run_privileged). A surviving GRAFANA_HOST_LABEL token would ship a
# config whose host label is a literal placeholder — the exact class of
# silent-label defect this tranche retired the board for.
local sandbox="$BATS_TEST_TMPDIR/opt"
(
cd "$REPO_ROOT"
eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
run_privileged() { "$@"; }
# NOTE: VAR=x func does not bind for shell functions — export.
export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
copy_alloy_config
)
[ -f "$sandbox/config/config.alloy" ]
grep -q 'host = "dankclaw",' "$sandbox/config/config.alloy"
! grep -q "GRAFANA_HOST_LABEL" "$sandbox/config/config.alloy"
}
@test "the blackbox copier installs the prober config" {
local sandbox="$BATS_TEST_TMPDIR/blackbox"
(
cd "$REPO_ROOT"
eval "$(sed -n '/^copy_blackbox_config()/,/^}/p' scripts/deploy-remote.sh)"
run_privileged() { "$@"; }
export BLACKBOX_CONF_ROOT="$sandbox"
copy_blackbox_config
)
[ -f "$sandbox/blackbox.yml" ]
grep -q '^ http_2xx:' "$sandbox/blackbox.yml"
}
@test "the alloy copier refuses to install a config the engine rejects" {
# The syntax gate: alloy attributes are position-sensitive and a
# misplaced one only explodes at run time (crash-loop behind a green
# deploy). Prove the copier refuses a config the pinned image won't
# validate, and installs nothing. Needs the alloy image; skipped
# where it is absent (the copier's gate is likewise image-gated).
if ! podman image exists localhost/franksdank/alloy:latest; then
skip "franksdank/alloy image not present on this box"
fi
local sandbox="$BATS_TEST_TMPDIR/opt" cfg="$BATS_TEST_TMPDIR/broken.alloy"
sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
printf 'bogus_attr = "x"\n' >> "$cfg"
! (
cd "$REPO_ROOT"
eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
run_privileged() { "$@"; }
cp() { if [ "$1" = "$cfg" ]; then return 1; fi; command cp "$@"; }
export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
copy_alloy_config
)
[ ! -f "$sandbox/config/config.alloy" ]
}
@test "the installed alloy config is engine-valid" {
# Runs the pinned image's own validator over the shipped config. The
# 2026-10-21 defect (`on_error` on the labels stage, which alloy 1.21
# does not define) passed every textual seam and only failed at boot.
if ! podman image exists localhost/franksdank/alloy:latest; then
skip "franksdank/alloy image not present on this box"
fi
local cfg="$BATS_TEST_TMPDIR/rendered.alloy"
sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
podman run --rm -v "$cfg:/c.alloy:ro" --entrypoint /usr/bin/alloy \
localhost/franksdank/alloy:latest validate /c.alloy
}
+3 -2
View File
@@ -15,8 +15,9 @@ QDIR="$REPO_ROOT/dankclaw/open-webui/quadlet"
}
@test "open-webui uses the CPU image, not :cuda (dankclaw is CPU-only)" {
grep -qE '^Image=ghcr\.io/open-webui/open-webui:main$' "$QDIR/open-webui.container" \
|| { echo "open-webui.container must use the :main (CPU) image"; return 1; }
# 2026-10-21 fleet law: tag AND digest (tests/quadlet_image_pins.bats)
grep -qE '^Image=ghcr\.io/open-webui/open-webui:main@sha256:[0-9a-f]{64}$' "$QDIR/open-webui.container" \
|| { echo "open-webui.container must use the :main (CPU) image, digest-pinned"; return 1; }
if grep -qE '^Image=.*:cuda' "$QDIR/open-webui.container"; then
echo "open-webui.container must not use the :cuda image on a CPU-only host"
return 1
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bats
# Seam: fleet-wide quadlet laws learned the hard way.
# 1. Every registry image is digest-pinned. A rolling tag (":latest",
# ":9-alpine", ":nightly") lets a rebuild move under a running
# service at the next restart/reboot with nobody approving it, and it
# makes a CVE audit ungradeable (you cannot score a tag). Locally
# built images (localhost/*, franksdank/*) are exempt: their build
# IS the pin, and they never pull.
# 2. No unit carries a key the podman 5.7.0 quadlet generator rejects.
# The generator aborts the WHOLE batch on an unsupported key, so one
# bad file silently un-generates every other unit in that manager -
# live 2026-10-21: HostAddr= in prometheus.container orphaned
# prometheus.service while its container kept running, and the next
# deploy died on "Unit prometheus.service not found".
load helpers
@test "every registry Image= is digest-pinned across all three hosts" {
bad=""
while IFS= read -r unit; do
while IFS= read -r img; do
case "$img" in
localhost/*|franksdank/*) continue ;;
esac
case "$img" in
*@sha256:*) ;;
*) bad="$bad$unit: $img"$'\n' ;;
esac
done < <(sed -n 's/^Image=//p' "$unit")
done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
-path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
[ -z "$bad" ] || { printf 'unpinned images:\n%s' "$bad" >&2; return 1; }
}
@test "no unit uses a key the 5.7.0 quadlet generator rejects (batch killer)" {
# HostAddr rejected by podman 5.7.0 (live 2026-10-21); Device= and
# Command= are the older v1 findings, guarded host-locally too.
# Keep this list current with every generator finding.
bad=""
while IFS= read -r unit; do
for key in HostAddr Device Command; do
grep -q "^${key}=" "$unit" && bad="$bad$unit: ${key}="$'\n'
done
done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
-path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
[ -z "$bad" ] || { printf 'generator-rejected keys:\n%s' "$bad" >&2; return 1; }
}
+527
View File
@@ -0,0 +1,527 @@
#!/usr/bin/env bats
# Seam: the Qwen3.8-Flash-Next five-bit campaign (feat/qwen38-w5nl64).
#
# The campaign converts the vendor BF16 checkpoint into a w5nl64a8h .rad on
# dankserver while frank-archon keeps serving. Every gate that protects the box
# or the result lives in a script under scripts/, and each gate is tested here
# through that script's own exit status and printed output — no engine, no GPU,
# no network. Fixtures are synthetic; the expected numbers are worked examples,
# not values recomputed the way the script computes them.
#
# Seams under test (the campaign's public boundaries):
# scripts/download-qwen38-bf16.sh fetch/verify of the BF16 checkpoint
# scripts/grams-to-imatrix.py calibration recording -> imatrix GGUF
# scripts/derive-protected-experts.py --target-count sweep + the parity law
# scripts/build-qwen38-bf16-rad.sh the reference container's convert argv
# scripts/run-qwen38-bf16.sh the drained-serve argv, mode selection
# scripts/convert-qwen38-w5nl64.sh the arm driver's refusal gates
# scripts/kld-gate-qwen38.sh the three-anchor interval law
# scripts/deploy-plan.py campaign files must deploy nothing
load helpers
DOWNLOAD="$REPO_ROOT/scripts/download-qwen38-bf16.sh"
PLANNER="$REPO_ROOT/scripts/deploy-plan.py"
# make_shard <dir> <name> <total-bytes> — a REAL safetensors file whose size on
# disk is exactly <total-bytes>: 8-byte header length, JSON header, tensor data.
# Real, because the verifier reads each shard's own header to learn how many
# bytes it claims to be, which is the only per-file truth available offline.
make_shard() {
mkdir -p "$1"
python3 - "$1/$2" "$3" <<'PY'
import json, struct, sys
path, want = sys.argv[1], int(sys.argv[2])
payload = max(0, want - 96)
header = b""
for _ in range(16):
# The header is part of the file, so its own length moves the target: solve
# payload = want - 8 - len(header(payload)) by fixed-point iteration.
header = json.dumps({"w": {"dtype": "F32", "shape": [payload // 4],
"data_offsets": [0, payload]}},
separators=(",", ":")).encode()
want_payload = want - 8 - len(header)
if want_payload == payload:
break
if want_payload < 0:
raise SystemExit("fixture too small to carry a header")
payload = want_payload
else:
raise SystemExit("fixture header never converged")
with open(path, "wb") as f:
f.write(struct.pack("<Q", len(header)) + header + b"\x00" * payload)
PY
}
# make_index <dir> <shard>... — a safetensors index that references them
make_index() {
local dir="$1"; shift
python3 - "$dir/model.safetensors.index.json" "$@" <<'PY'
import json, sys
out, names = sys.argv[1], sys.argv[2:]
json.dump({"metadata": {"total_size": 0},
"weight_map": {f"model.layers.{i}.attn.q.weight": n
for i, n in enumerate(names)}},
open(out, "w"))
PY
}
# three_fixture <dir> — three valid 1000-byte shards plus the index naming them
three_fixture() {
make_shard "$1" model-00001-of-00003.safetensors 1000
make_shard "$1" model-00002-of-00003.safetensors 1000
make_shard "$1" model-00003-of-00003.safetensors 1000
make_index "$1" model-00001-of-00003.safetensors \
model-00002-of-00003.safetensors \
model-00003-of-00003.safetensors
}
# --- download / verify -------------------------------------------------------
@test "the checkpoint verifier accepts a complete checkpoint and reports its footprint" {
C="$BATS_TEST_TMPDIR/complete"
three_fixture "$C"
# 3 x 1000 bytes of shards. The index file itself is deliberately not
# counted, so a Hub-side edit to a config file cannot fail this gate.
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
[ "$status" -eq 0 ]
[[ "$output" == *"VERIFY_OK"*"3 shards"*"3000 bytes"* ]]
}
@test "the verifier names the shard whose own header disagrees with its size on disk" {
C="$BATS_TEST_TMPDIR/truncated"
three_fixture "$C"
# Cut the tail, keep the header: that is a torn transfer, which is what the
# box actually gets from an interrupted fetch. Zero-filling instead would
# destroy the header and be caught as "not a safetensors file" — a different
# gate, and it would prove nothing about the size check.
python3 -c 'import sys
p = sys.argv[1]
keep = open(p, "rb").read(400)
open(p, "wb").write(keep)' "$C/model-00002-of-00003.safetensors"
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
[ "$status" -eq 1 ]
[[ "$output" == *"TRUNCATED"*"model-00002-of-00003.safetensors"* ]]
}
@test "the verifier refuses a missing shard and names it" {
C="$BATS_TEST_TMPDIR/missing"
three_fixture "$C"
mv "$C/model-00003-of-00003.safetensors" "$C/hidden.bin"
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
[ "$status" -eq 1 ]
[[ "$output" == *"MISSING"*"model-00003-of-00003.safetensors"* ]]
}
@test "the verifier refuses when the shard census disagrees with the expected count" {
# The fragment law this repo learned the hard way: large quants ship as
# .partNofM fragments and a naive glob reports success on a partial tree.
C="$BATS_TEST_TMPDIR/count"
three_fixture "$C"
make_shard "$C" model-00004-of-00004.safetensors 1000
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
[ "$status" -eq 1 ]
[[ "$output" == *"SHARD_COUNT"*"4"*"3"* ]]
}
@test "the verifier refuses a shard the index names that never landed" {
C="$BATS_TEST_TMPDIR/orphan-index"
three_fixture "$C"
rm "$C/model-00003-of-00003.safetensors"
make_index "$C" model-00001-of-00003.safetensors \
model-00002-of-00003.safetensors \
model-00003-of-00003.safetensors
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 2 --bytes 2000
[ "$status" -eq 1 ]
[[ "$output" == *"MISSING"*"model-00003-of-00003.safetensors"* ]]
}
@test "the verifier treats an HTTP error body written as a shard as a failure, not a download" {
# curl -o FILE exits 0 on 401/403 and writes the error body; three shards
# of identical tiny size must never look like a checkpoint.
C="$BATS_TEST_TMPDIR/errorbody"
mkdir -p "$C"
for n in 1 2 3; do
printf '{"error":"You are not logged in"}' \
> "$C/model-0000${n}-of-00003.safetensors"
done
make_index "$C" model-00001-of-00003.safetensors \
model-00002-of-00003.safetensors \
model-00003-of-00003.safetensors
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
[ "$status" -eq 1 ]
[[ "$output" == *"NOT_SAFETENSORS"* ]]
}
@test "the verifier refuses a shard census whose bytes do not total what the Hub declared" {
# Every shard header can be self-consistent and the tree still be the wrong
# payload: the total is the cross-check against the Hub's declared size.
C="$BATS_TEST_TMPDIR/wrongtotal"
three_fixture "$C"
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 2500
[ "$status" -eq 1 ]
[[ "$output" == *"TOTAL_BYTES"*"3000"*"2500"* ]]
}
@test "the fetch path refuses when the destination cannot hold the checkpoint" {
# The campaign's peak-disk law starts at the fetch: the BF16 checkpoint and
# the reference container built from it are both ~335 GiB, and the tree that
# does not fit is discovered hours into a transfer.
C="$BATS_TEST_TMPDIR/tiny"
three_fixture "$C"
run env FRANK_QWEN38_MIN_FREE_MIB=999999999 \
bash "$DOWNLOAD" --fetch --no-hub --dir "$C" --shards 3 --bytes 3000
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"*"free"* ]]
}
# --- deployment scoping ------------------------------------------------------
@test "the campaign tooling deploys nothing: merging it cannot drain frank-archon" {
# Existence is asserted first: a planner that plans nothing because the
# files vanished would otherwise pass.
[ -f "$REPO_ROOT/scripts/download-qwen38-bf16.sh" ]
run python3 "$PLANNER" --target dankserver --paths \
scripts/download-qwen38-bf16.sh \
scripts/build-qwen38-bf16-rad.sh \
scripts/run-qwen38-bf16.sh \
scripts/convert-qwen38-w5nl64.sh \
scripts/kld-gate-qwen38.sh \
scripts/grams-to-imatrix.py \
scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe \
tests/qwen38_campaign.bats
[ "$status" -eq 0 ]
[ -z "$output" ]
}
# --- protected experts -------------------------------------------------------
DERIVE="$REPO_ROOT/scripts/derive-protected-experts.py"
APPLY="$REPO_ROOT/scripts/apply-protected-cast.sh"
CONVERT="$REPO_ROOT/scripts/convert-qwen38-w5nl64.sh"
GATE="$REPO_ROOT/scripts/kld-gate-qwen38.sh"
TEMPLATE="$REPO_ROOT/scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe"
# make_gram <dir> <layer> <expert> <energy> <rows> — a RADGRAM2 down-Gram whose
# trace (its down-input energy) is <energy> and whose routing count is <rows>.
make_gram() {
python3 - "$@" <<'PY'
import os, struct, sys
d, layer, e, energy, rows = sys.argv[1:6]
os.makedirs(d, exist_ok=True)
hdr = struct.pack("<8sIIQI4x", b"RADGRAM2", 2, 0, int(rows), 0)
mat = struct.pack("<4f", float(energy), 0.0, 0.0, 0.0)
with open(os.path.join(d, f"gram.r0.blk.{layer}.ffn_down_exps.{e}.bin"), "wb") as f:
f.write(hdr + mat)
PY
}
# eight_grams <dir> — one layer, eight experts inside rank half 0, energies
# 60/15/10/8/4/2/1/0.5 against equal routing. Under the default bars (3% of the
# layer's energy AND 3x its routing share) exactly ONE expert is disproportionate,
# so the parity law has to hand it a partner.
eight_grams() {
local d="$1"
make_gram "$d" 0 0 60 100
make_gram "$d" 0 1 15 100
make_gram "$d" 0 2 10 100
make_gram "$d" 0 3 8 100
make_gram "$d" 0 4 4 100
make_gram "$d" 0 5 2 100
make_gram "$d" 0 6 1 100
make_gram "$d" 0 7 0.5 100
}
cast_count() { grep -c 'cast dtype=bf16' <<< "$output"; }
@test "the bar-only pass keeps the disproportionate expert and repairs the odd rank half" {
C="$BATS_TEST_TMPDIR/calib"
eight_grams "$C"
run python3 "$DERIVE" "$C"
[ "$status" -eq 0 ]
# e0 holds 59.7% of the layer's energy on 12.5% of the routing: the only expert
# that clears three times its routing share. e1 joins it because half 0 was odd.
[ "$(cast_count)" -eq 2 ]
[[ "$output" == *"blk.0.ffn_*_exps.0.weight"* ]]
[[ "$output" == *"blk.0.ffn_*_exps.1.weight"* ]]
[[ "$output" != *"ffn_*_exps.2.weight"* ]]
[[ "$output" == *"half 0 odd -> added e1"* ]]
}
@test "--target-count lands the byte bill that was asked for, and never buys more" {
# The count is a disk and VRAM decision (each protected expert costs bf16 twice,
# once per rank half's copy), so landing it matters as much as not missing it.
C="$BATS_TEST_TMPDIR/calib"
eight_grams "$C"
run python3 "$DERIVE" "$C" --target-count 6
[ "$status" -eq 0 ]
[ "$(cast_count)" -eq 6 ]
[[ "$output" == *"count fill -> added e2"* ]]
[[ "$output" != *"ffn_*_exps.6.weight"* ]]
# the energy share the set bottoms out at is the bar the campaign paid for
[[ "$output" == *"total protected: 6 (target 6, energy cutoff"* ]]
}
@test "--target-count cannot invent experts: the pool's ceiling is reported, not padded" {
C="$BATS_TEST_TMPDIR/calib"
eight_grams "$C"
run python3 "$DERIVE" "$C" --target-count 20
[ "$status" -eq 0 ]
[ "$(cast_count)" -eq 8 ]
[[ "$output" == *"total protected: 8 (target 20"* ]]
}
@test "a protected set that cannot be made even is refused instead of emitted" {
# One disproportionate expert in half 0, its only companion in half 1: no
# candidate could make half 0 even, and an odd half hands the four-bit GEMM two
# tables that are not equal. Refusing is the only honest answer.
C="$BATS_TEST_TMPDIR/calib"
make_gram "$C" 0 0 100 100
make_gram "$C" 0 300 1 1000
run python3 "$DERIVE" "$C"
[ "$status" -eq 2 ]
[[ "$output" == *"PARITY LAW BROKEN"* ]]
[[ "$output" == *"blk.0 half 0 holds 1"* ]]
[ "$(cast_count)" -eq 0 ]
}
@test "the splice rewrites the run's copy above the expert rule and refuses a second run" {
C="$BATS_TEST_TMPDIR/calib"
eight_grams "$C"
W="$BATS_TEST_TMPDIR/recipe-work.recipe"
cp "$TEMPLATE" "$W"
run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 0
[ "$status" -eq 0 ]
grep -q "derive-protected-experts.py rewrote" "$W"
# The derived lines must precede the expert rule: the first rule whose glob
# matches a weight decides it, so a cast line below the expert line never fires.
cast_line=$(grep -n 'cast dtype=bf16' "$W" | head -1 | cut -d: -f1)
expert_line=$(grep -n 'ffn_\*_exps\.\*\.weight' "$W" | head -1 | cut -d: -f1)
[ "$cast_line" -lt "$expert_line" ]
# The committed template stays the clean, reviewable baseline.
! grep -q "derive-protected-experts.py rewrote" "$TEMPLATE"
# A second run on the same copy would derive from a recording it already spent.
run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 0
[ "$status" -ne 0 ]
[[ "$output" == *"refusing to re-derive"* ]]
}
@test "the splice refuses before touching the recipe when the count is not reached" {
C="$BATS_TEST_TMPDIR/calib"
eight_grams "$C"
W="$BATS_TEST_TMPDIR/recipe-work.recipe"
cp "$TEMPLATE" "$W"
run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 20
[ "$status" -eq 1 ]
[[ "$output" == *"target was 20"* ]]
! grep -q "derive-protected-experts.py rewrote" "$W"
}
# --- the arm driver ----------------------------------------------------------
# drv_fixture <dir> — the smallest tree the driver's gates accept: a checkpoint
# with its index and tokenizer, three grams, an imatrix inside the campaign dir
# the container mounts, and an empty output dir.
drv_fixture() {
local d="$1"
mkdir -p "$d/ck/Qwen3.8-Flash-Next" "$d/campaign" "$d/calib" "$d/out"
: > "$d/ck/Qwen3.8-Flash-Next/model.safetensors.index.json"
: > "$d/ck/Qwen3.8-Flash-Next/tokenizer.json"
make_gram "$d/calib" 0 0 60 100
make_gram "$d/calib" 0 1 15 100
make_gram "$d/calib" 0 2 10 100
# four experts, not three: a thin pool that cannot be made even trips the parity
# audit, and this fixture is meant to reach the later gates
make_gram "$d/calib" 0 3 8 100
: > "$d/campaign/imatrix.gguf"
}
# run_drv <driver args and K=V overrides, any order> — the driver with every path
# pointed at the fixture. Override words are separated out because `env` only
# accepts assignments ahead of the command; later assignments win, so a test can
# break exactly one input.
run_drv() {
local d="$BATS_TEST_TMPDIR/drv" assigns=() flags=() w
for w in "$@"; do
if [[ "$w" =~ ^[A-Za-z_][A-Za-z0-9_]*= ]]; then assigns+=("$w"); else flags+=("$w"); fi
done
run env CKPT="$d/ck/Qwen3.8-Flash-Next" CAMPAIGN="$d/campaign" STATE="$d/campaign" \
CALIB="$d/calib" OUTDIR="$d/out" IMATRIX="$d/campaign/imatrix.gguf" \
MIN_GRAMS=2 TARGET_COUNT=0 ${assigns[@]+"${assigns[@]}"} \
bash "$CONVERT" ${flags[@]+"${flags[@]}"}
}
@test "the dry run prints the plan and then the convert for every arm" {
drv_fixture "$BATS_TEST_TMPDIR/drv"
run_drv --dry-run
[ "$status" -eq 0 ]
[[ "$output" == *"== arm cd3 =="* ]]
[[ "$output" == *"== arm cd6 =="* ]]
[[ "$output" == *"== arm ao1 =="* ]]
# one --plan-only bill per arm, and one convert line per arm beside it
[ "$(grep -c -- '--plan-only' <<< "$output")" -eq 3 ]
[ "$(grep -c 'rad-convert' <<< "$output")" -eq 6 ]
awk '/arm cd3/ {next}
/qwen38-w5nl64-cd3\.rad/ { if ($0 ~ /--plan-only/) p = NR; else c = NR }
END { exit !(p && c && p < c) }' <<< "$output"
}
@test "the cd6 arm re-sends the whole expert option set with cd=6; the control sends none" {
# --quant replaces the recipe line wholesale, so an override that forgot an
# option would quietly convert with the default for it.
drv_fixture "$BATS_TEST_TMPDIR/drv"
run_drv --arm cd3 --arm cd6 --dry-run
[ "$status" -eq 0 ]
cd6line=$(grep -- '-cd6\.rad --quant' <<< "$output" | head -1)
[[ "$cd6line" == *"table=w5nl"* ]]
[[ "$cd6line" == *"group=64"* ]]
[[ "$cd6line" == *"scale=fp8_e4m3"* ]]
[[ "$cd6line" == *"transform=fwht128"* ]]
[[ "$cd6line" == *"calib=\$CALIB"* ]]
[[ "$cd6line" == *",cd=6"* ]]
cd3line=$(grep -- '-cd3\.rad' <<< "$output" | grep -v -- '--plan-only' | head -1)
[[ "$cd3line" != *"--quant"* ]]
}
@test "the ao1 arm carries act_order=1 and no cd=, because libquant refuses the pair" {
drv_fixture "$BATS_TEST_TMPDIR/drv"
run_drv --arm ao1 --dry-run
[ "$status" -eq 0 ]
line=$(grep -- '-ao1\.rad --quant' <<< "$output" | head -1)
[[ "$line" == *"act_order=1"* ]]
[[ "$line" != *",cd="* ]]
}
@test "the driver splices the protected plane into its own copy, never the template" {
drv_fixture "$BATS_TEST_TMPDIR/drv"
run_drv --dry-run
[[ "$output" == *"apply-protected-cast.sh $BATS_TEST_TMPDIR/drv/calib "* ]]
[[ "$output" == *"/campaign/recipe-"* ]]
[[ "$output" != *"$TEMPLATE"* ]]
}
@test "the driver refuses while the BF16 reference container still holds the volume" {
drv_fixture "$BATS_TEST_TMPDIR/drv"
: > "$BATS_TEST_TMPDIR/drv/out/qwen38-bf16.rad"
run_drv --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"peak-disk"* ]]
}
@test "a thin recording is refused: the driver never converts uncalibrated" {
# rad-convert converts WITHOUT a calibration recording with one stderr note,
# which is silently worse than a GPTQ run that failed.
drv_fixture "$BATS_TEST_TMPDIR/drv"
run_drv MIN_GRAMS=9 --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"run-qwen38-bf16.sh calib"* ]]
}
@test "an imatrix outside the mounted campaign dir is refused before any convert" {
drv_fixture "$BATS_TEST_TMPDIR/drv"
mv "$BATS_TEST_TMPDIR/drv/campaign/imatrix.gguf" "$BATS_TEST_TMPDIR/imatrix.gguf"
run_drv IMATRIX="$BATS_TEST_TMPDIR/imatrix.gguf" --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"not in the campaign dir"* ]]
}
@test "the driver never overwrites an arm that already exists" {
drv_fixture "$BATS_TEST_TMPDIR/drv"
: > "$BATS_TEST_TMPDIR/drv/out/qwen38-w5nl64-cd6.rad"
run_drv --arm cd6 --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"move it to trash first"* ]]
}
@test "an unknown arm name is refused before the gates run" {
# No fixture built: the name check must not need a checkpoint to exist.
run_drv --arm cd9 --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"unknown arm: cd9"* ]]
}
@test "the driver stops when the protected derivation cannot land the target" {
drv_fixture "$BATS_TEST_TMPDIR/drv"
run_drv PY=python3 TARGET_COUNT=20 --arm cd3
[ "$status" -eq 1 ]
[[ "$output" == *"target was 20"* ]]
[[ "$output" == *"would convert unprotected"* ]]
[[ "$output" != *"--entrypoint"* ]]
}
# --- the acceptance gate -----------------------------------------------------
# kld_report <path> <model> <kl mean> <top1 fraction> — one score report, in the
# shape `run-qwen38-bf16.sh score` writes: an `all` summary plus a per-source one.
kld_report() {
python3 - "$@" <<'PY'
import json, sys
path, model, kl, top1 = sys.argv[1:5]
kl, top1 = float(kl), float(top1)
group = {"kld": {"mean": kl, "p99_9": kl * 4}, "top1_agreement": top1, "positions": 50432}
json.dump({"model": model, "all": group,
"by_source": {"ops": dict(group, positions=20000)}}, open(path, "w"))
PY
}
gate_case() { # gate_case <kl fp8> <kl iq4r> <kl cand> <top1 cand> [extra flags...]
D="$BATS_TEST_TMPDIR/kl"
mkdir -p "$D"
kld_report "$D/fp8.json" fp8-native "$1" 0.981
kld_report "$D/iq4r.json" iq4r-moe "$2" 0.962
kld_report "$D/cand.json" w5nl64 "$3" "$4"
run bash "$GATE" "$D/fp8.json" "$D/iq4r.json" "$D/cand.json" "${@:5}"
}
@test "an arm strictly inside the interval that closes half the gap passes" {
gate_case 0.0021 0.0190 0.0090 0.971
[ "$status" -eq 0 ]
[[ "$output" == *"closes 59.2%"* ]]
[[ "$output" == *"KLD GATE PASS"* ]]
}
@test "an arm worse than the served payload fails the interval and the recovery" {
gate_case 0.0021 0.0190 0.0210 0.971
[ "$status" -eq 1 ]
[[ "$output" == *"[FAIL] 2."* ]]
[[ "$output" == *"[FAIL] 3."* ]]
[[ "$output" == *"KLD GATE FAIL"* ]]
}
@test "anchors that are not ordered stop the gate: the instrument is wrong, not the arm" {
# If the near-lossless FP8 release does not sit CLOSER to the bf16 reference
# than the 4-bit payload does, the corpus, tokenizer or boot shape is broken and
# no arm number can mean anything.
gate_case 0.0300 0.0190 0.0120 0.971
[ "$status" -eq 1 ]
[[ "$output" == *"[FAIL] 1."* ]]
[[ "$output" == *"gap is not positive"* ]]
}
@test "an arm measured outside the interval fails even when it looks better" {
# Closer to the reference than the achievable anchor is a measurement fault.
gate_case 0.0021 0.0190 0.0010 0.971
[ "$status" -eq 1 ]
[[ "$output" == *"[FAIL] 2."* ]]
}
@test "an arm that buys KL with top-1 agreement fails" {
gate_case 0.0021 0.0190 0.0090 0.940
[ "$status" -eq 1 ]
[[ "$output" == *"[PASS] 3."* ]]
[[ "$output" == *"[FAIL] 4."* ]]
}
@test "--recover tightens the same numbers until they fail" {
gate_case 0.0021 0.0190 0.0090 0.971 --recover 0.8
[ "$status" -eq 1 ]
[[ "$output" == *"needs 80%"* ]]
[[ "$output" == *"[FAIL] 3."* ]]
}
@test "a slice the report does not carry is named, not guessed at" {
gate_case 0.0021 0.0190 0.0090 0.971 --slice code
[ "$status" -eq 2 ]
[[ "$output" == *"has no slice 'code'"* ]]
[[ "$output" == *"it has: ops"* ]]
}
+445
View File
@@ -0,0 +1,445 @@
#!/usr/bin/env bats
# Seam: scripts/grams-to-imatrix.py — the calibration recording (RADGRAM2 grams)
# folded into the GGUF v3 imatrix `rad-convert --imatrix F` reads.
#
# The only contract that matters is the reader's: core/format/imatrix.cpp joins
# `<base>.in_sum2` with `<base>.counts`, takes cols from in_sum2 dims[0] and
# n_expert from dims[1], demands ggml f32 tensors and GGUF version 3, and
# silently IGNORES an entry whose width does not match the weight it is asked
# about. So every test here parses the artifact back with its own tiny GGUF
# reader and asserts bytes — a file that merely exists and exits 0 proves
# nothing about a quantiser that will read it hours later on another machine.
#
# Fixtures are synthetic: k=4 grams with real RADGRAM2 headers (rows is a u64
# at offset 16 — pack it as u32 and 100 rows read back as 4294967296000) and
# real f32 payloads whose OFF-DIAGONAL is filled with 99.0, so an entry built
# from anything but the diagonal is instantly visible. Expected numbers are
# worked examples from the fixture tables below, never recomputed the way the
# tool computes them.
load helpers
TOOL="$REPO_ROOT/scripts/grams-to-imatrix.py"
# make_gram <path> <k> <rank> <rows> <domain> <diag,csv>
# A RADGRAM2 file: 32-byte header <8sIIQI (magic, k, rank, rows, domain, pad)
# then k*k f32 row-major. Off-diagonals are 99.0 — a sentinel, not data.
make_gram() {
python3 - "$@" <<'PY'
import struct, sys
path, k, rank, rows, domain = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), \
int(sys.argv[4]), int(sys.argv[5])
diag = [float(x) for x in sys.argv[6].split(",")]
assert len(diag) == k, "diagonal must hold k values"
vals = [99.0] * (k * k)
for i, d in enumerate(diag):
vals[i * k + i] = d
hdr = struct.pack("<8sIIQI", b"RADGRAM2", k, rank, rows, domain) + struct.pack("<I", 0)
with open(path, "wb") as f:
f.write(hdr + struct.pack("<%df" % (k * k), *vals))
PY
}
# full_recording <dir> — the fixture every happy-path test reads.
#
# layer 0 (k=4, own-gram floor 4*k = 16 rows):
# pooled r0 rows 40 diag 1,2,3,4 + r1 rows 8 diag .5,.5,.5,.5
# -> summed gram (the engine's calib_sum law): diag 1.5,2.5,3.5,4.5,
# pooled rows 48
# expert 0 r0 rows 100 diag 10,20,30,40 (>= 16: own gram kept)
# expert 1 r0 rows 3 diag 7,7,7,7 (< 16: inherits pooled)
# expert 2 r1 rows 64 diag 1,1,1,2 (>= 16: own gram kept)
# expert 3 no file at all (inherits pooled, count 0)
# expert 4 r1 rows 100 diag 5,6,7,8 (>= 16: own gram kept)
# -> n_expert 5, counts 100 3 64 0 100, inherited 2, routing 267
# layer 1: pooled r0 rows 48 diag 2,2,2,2 ; expert 0 on r1 rows 48 diag 1,1,1,1
# -> n_expert 1, counts 48, inherited 0, routing 48
# Corpus rows = the largest single-layer pooled rows = 48.
full_recording() {
local d="$1"
mkdir -p "$d"
make_gram "$d/gram.r0.blk.0.ffn_down_exps.bin" 4 0 40 0 "1,2,3,4"
make_gram "$d/gram.r1.blk.0.ffn_down_exps.bin" 4 1 8 0 "0.5,0.5,0.5,0.5"
make_gram "$d/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 100 0 "10,20,30,40"
make_gram "$d/gram.r0.blk.0.ffn_down_exps.1.bin" 4 0 3 0 "7,7,7,7"
make_gram "$d/gram.r1.blk.0.ffn_down_exps.2.bin" 4 1 64 0 "1,1,1,2"
make_gram "$d/gram.r1.blk.0.ffn_down_exps.4.bin" 4 1 100 0 "5,6,7,8"
make_gram "$d/gram.r0.blk.1.ffn_down_exps.bin" 4 0 48 0 "2,2,2,2"
make_gram "$d/gram.r1.blk.1.ffn_down_exps.0.bin" 4 1 48 0 "1,1,1,1"
}
# gguf_read <file> [vals <tensor> <float-index> <count>] — an independent GGUF
# v3 reader: KV block, tensor infos, and (on request) the f32 values behind a
# tensor's own declared offset. Prints KV/HDR/TENSOR lines. Nothing here shares
# code with the writer, which is the only reason the byte assertions mean
# anything: imatrix.cpp walks the same three structures.
gguf_read() {
python3 - "$@" <<'PY'
import struct, sys
d = open(sys.argv[1], "rb").read()
def u32(o): return struct.unpack_from("<I", d, o)[0]
def u64(o): return struct.unpack_from("<Q", d, o)[0]
def st(o):
n = u64(o)
return d[o + 8:o + 8 + n].decode(), o + 8 + n
assert d[:4] == b"GGUF", "magic is %r" % d[:4]
ver, n_t, n_kv = u32(4), u64(8), u64(16)
p, align, out = 24, 32, []
for _ in range(n_kv):
key, p = st(p)
ty = u32(p); p += 4
if ty == 4: # uint32
v = u32(p); p += 4
if key == "general.alignment":
align = v
elif ty == 8: # string
v, p = st(p)
elif ty == 9: # array
et = u32(p); n = u64(p + 4); p += 12
got = []
for _i in range(n):
if et != 8:
raise SystemExit("kv array element type %d" % et)
s, p = st(p)
got.append(s)
v = ",".join(got)
else:
raise SystemExit("kv type %d is not steppable" % ty)
out.append("KV %s %s" % (key, v))
tens = []
for _ in range(n_t):
name, p = st(p)
nd = u32(p); p += 4
dims = [u64(p + 8 * i) for i in range(nd)]; p += 8 * nd
ty = u32(p); off = u64(p + 4); p += 12
tens.append((name, dims, ty, off))
ds = (p + align - 1) // align * align
out.append("HDR version=%d n_tensors=%d n_kv=%d alignment=%d data_start=%d"
% (ver, n_t, n_kv, align, ds))
for name, dims, ty, off in tens:
out.append("TENSOR %s %s %d %d" % (name, "x".join(str(x) for x in dims), ty, off))
if len(sys.argv) > 2 and sys.argv[2] == "vals":
name, i0, n = sys.argv[3], int(sys.argv[4]), int(sys.argv[5])
for nm, dims, ty, off in tens:
if nm == name:
base = ds + off
vals = [("%d" % v) if float(v).is_integer() else repr(v)
for v in (struct.unpack_from("<f", d, base + 4 * (i0 + i))[0]
for i in range(n))]
out.append("VALS %s %s" % (name, " ".join(vals)))
print("\n".join(out))
PY
}
# aligned_offsets <file> — exit 0 iff the data section starts aligned AND every
# tensor's absolute data offset (data_start + t.off) is a multiple of
# general.alignment. This is the writer bug that no value assertion can see: an
# unaligned offset makes the reader's extent arithmetic land mid-float in the
# previous tensor, and it reads the result without complaining.
aligned_offsets() {
python3 - "$1" <<'PY'
import struct, sys
d = open(sys.argv[1], "rb").read()
def u32(o): return struct.unpack_from("<I", d, o)[0]
def u64(o): return struct.unpack_from("<Q", d, o)[0]
def st(o):
n = u64(o)
return d[o + 8:o + 8 + n].decode(), o + 8 + n
ver, n_t, n_kv = u32(4), u64(8), u64(16)
p, align = 24, 32
for _ in range(n_kv):
key, p = st(p)
ty = u32(p); p += 4
if ty == 4:
v = u32(p); p += 4
if key == "general.alignment":
align = v
if ty == 8:
_v, p = st(p)
elif ty == 9:
et = u32(p); n = u64(p + 4); p += 12
for _i in range(n):
if et == 8:
_v, p = st(p)
elif et == 4:
p += 4
else:
raise SystemExit("kv array element type %d" % et)
elif ty in (0, 1):
p += 1
elif ty in (2, 3):
p += 2
elif ty in (5, 6):
p += 4
elif ty in (10, 11, 12):
p += 8
tens = []
for _ in range(n_t):
name, p = st(p)
nd = u32(p); p += 4
p += 8 * nd
ty = u32(p); off = u64(p + 4); p += 12
tens.append((name, ty, off))
ds = (p + align - 1) // align * align
bad = [] if ds % align == 0 else ["data_start=%d" % ds]
bad += ["%s=%d" % (nm, ds + off) for nm, _ty, off in tens if (ds + off) % align]
if bad:
print("UNALIGNED", ";".join(bad))
sys.exit(1)
print("ALIGNED %d tensor(s), data_start=%d, alignment=%d" % (len(tens), ds, align))
PY
}
# --- the artifact -----------------------------------------------------------
@test "a complete recording becomes a GGUF v3 file and the run reports itself" {
C="$BATS_TEST_TMPDIR/full"
full_recording "$C"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf" \
--datasets calib,chat --chunk-size 16
[ "$status" -eq 0 ]
[[ "$output" == *"IMATRIX_OK"* ]]
head -c 4 "$BATS_TEST_TMPDIR/im.gguf" | grep -q "GGUF"
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
[[ "$output" == *"HDR version=3"* ]]
# The reader refuses anything but version 3, so this is the whole gate.
[[ "$output" != *"version=2"* ]]
# Everything the reader looks for by key, and the honesty of the chunk math:
# 48 corpus rows over 16-row chunks is 3 chunks.
[[ "$output" == *"KV general.alignment 32"* ]]
[[ "$output" == *"KV imatrix.chunk_size 16"* ]]
[[ "$output" == *"KV imatrix.chunk_count 3"* ]]
[[ "$output" == *"KV imatrix.datasets calib,chat"* ]]
[[ "$output" == *"n_tensors=12"* ]]
}
@test "the reader finds the folded MoE entry: one per layer, expert-major, f32" {
C="$BATS_TEST_TMPDIR/full"
full_recording "$C"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
[ "$status" -eq 0 ]
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
# dims (k, n_expert) with k fastest, ggml type 0 (f32) or the entry is
# warned away, and 5 experts because the fixture's largest global id is 4.
[[ "$output" == *"TENSOR blk.0.ffn_down_exps.in_sum2 4x5 0 "* ]]
[[ "$output" == *"TENSOR blk.0.ffn_down_exps.counts 1x5 0 "* ]]
# counts are the routing rows, not the energy trace: 100 3 64 0 100.
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 0 5
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
# Expert 0 (floats 0..3) and expert 4 (floats 16..19) carry their own
# matrices' diagonals; the 99.0 off-diagonal fill appears nowhere.
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 10 20 30 40"* ]]
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 16 4
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 5 6 7 8"* ]]
[[ "$output" != *"99"* ]]
}
@test "an expert with no own gram carries the pooled diagonal, and the count of them is reported" {
C="$BATS_TEST_TMPDIR/full"
full_recording "$C"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
[ "$status" -eq 0 ]
[[ "$output" == *"inherited=2"* ]]
# Expert 1 has a gram (3 rows, under the 4*k floor) and expert 3 has none at
# all; both slices must be the POOLED diag 1+0.5, 2+0.5, 3+0.5, 4+0.5.
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 4 4
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 12 4
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
# Their counts stay their own routing: 3 for the under-threshold expert, 0
# for the unrecorded one — popularity is count's job, never the gram's.
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 1 3
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 3 64 0"* ]]
# The census is a report for the operator, so it belongs on stderr. `run`
# folds both streams into $output, so a second run into a different -o (an
# existing output is refused by design) captures stderr alone: without the
# stderr write there is no file to grep and this assertion goes red.
python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im2.gguf" 2>"$BATS_TEST_TMPDIR/err"
[ -s "$BATS_TEST_TMPDIR/err" ]
grep -q "inherited=2" "$BATS_TEST_TMPDIR/err"
}
@test "the hidden-width entries are the same bytes, and nothing else is emitted" {
C="$BATS_TEST_TMPDIR/full"
full_recording "$C"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
[ "$status" -eq 0 ]
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
# gate_up shares the down matrices (same hidden-state rows), so its slice 4
# must read 5 6 7 8 at its own tensor info's offset.
[[ "$output" == *"TENSOR blk.0.ffn_gate_up_exps.in_sum2 4x5 0 "* ]]
[[ "$output" == *"TENSOR blk.0.ssm_inz.in_sum2 4 0 "* ]]
[[ "$output" == *"TENSOR blk.0.ssm_inz.counts 1 0 "* ]]
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_gate_up_exps.in_sum2 16 4
[[ "$output" == *"VALS blk.0.ffn_gate_up_exps.in_sum2 5 6 7 8"* ]]
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.in_sum2 0 4
[[ "$output" == *"VALS blk.0.ssm_inz.in_sum2 1.5 2.5 3.5 4.5"* ]]
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.counts 0 1
[[ "$output" == *"VALS blk.0.ssm_inz.counts 48"* ]]
# The stop rule: importance_for() false-on-a-width-mismatch means an entry
# for a projection whose input is NOT the gram's width is a silent lie.
[[ "$output" != *"ssm_out"* ]]
[[ "$output" != *"attn"* ]]
[[ "$output" != *"mtp."* ]]
[[ "$output" != *"ple_ngram"* ]]
[[ "$output" != *"output.weight"* ]]
}
@test "rows is a u64: a corpus too big for u32 survives the header" {
# The trap this fixture law exists for: pack rows as u32 at offset 16 and a
# 100-row file reads back as 4294967296000. 2**32 is exactly representable
# in f32, so the count round-trips only if the tool read 8 bytes.
C="$BATS_TEST_TMPDIR/wide"
mkdir -p "$C"
make_gram "$C/gram.r0.blk.0.ffn_down_exps.bin" 4 0 4294967296 0 "1,1,1,1"
make_gram "$C/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 4294967296 0 "3,3,3,3"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide.gguf" --chunk-size 4096
[ "$status" -eq 0 ]
run gguf_read "$BATS_TEST_TMPDIR/wide.gguf" vals blk.0.ffn_down_exps.counts 0 1
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 4294967296"* ]]
[[ "$output" != *"4294967296000"* ]]
run gguf_read "$BATS_TEST_TMPDIR/wide.gguf"
[[ "$output" == *"KV imatrix.chunk_count 1048576"* ]]
# imatrix.chunk_count is a u32 field, so 2**32 rows in 1-row chunks has no
# legal encoding: wrapping it would put a number downstream believes.
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide2.gguf" --chunk-size 1
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"*"CHUNK_COUNT"* ]]
[ ! -e "$BATS_TEST_TMPDIR/wide2.gguf" ]
}
@test "every tensor's data offset is a multiple of general.alignment" {
C="$BATS_TEST_TMPDIR/full"
full_recording "$C"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
[ "$status" -eq 0 ]
run aligned_offsets "$BATS_TEST_TMPDIR/im.gguf"
[ "$status" -eq 0 ]
[[ "$output" == "ALIGNED 12 tensor(s),"*"alignment=32" ]]
# Red proof, in the same test so the assertion cannot rot: the same file
# with one tensor's declared offset nudged by 4. A counts tensor is 4 bytes,
# so a writer that forgets to pad between tensors produces exactly this
# shape of file — and every value assertion above still passes on it.
python3 - "$BATS_TEST_TMPDIR/im.gguf" "$BATS_TEST_TMPDIR/unaligned.gguf" <<'PY'
import struct, sys
src, dst = sys.argv[1], sys.argv[2]
d = bytearray(open(src, "rb").read())
def u32(o): return struct.unpack_from("<I", d, o)[0]
def u64(o): return struct.unpack_from("<Q", d, o)[0]
def st(o):
n = u64(o)
return d[o + 8:o + 8 + n].decode(), o + 8 + n
p = 24
for _ in range(u64(16)):
key, p = st(p)
ty = u32(p); p += 4
if ty == 4:
p += 4
elif ty == 8:
_v, p = st(p)
elif ty == 9:
et = u32(p); n = u64(p + 4); p += 12
for _i in range(n):
_v, p = st(p)
for _ in range(u64(8)):
name, p = st(p)
nd = u32(p); p += 4
p += 8 * nd
off_at = p + 4 # the u64 offset field of this tensor
if name == "blk.0.ssm_inz.counts":
struct.pack_into("<Q", d, off_at, u64(off_at) + 4)
p += 12
open(dst, "wb").write(bytes(d))
PY
run aligned_offsets "$BATS_TEST_TMPDIR/unaligned.gguf"
[ "$status" -eq 1 ]
[[ "$output" == *"UNALIGNED"*"blk.0.ssm_inz.counts"* ]]
}
# --- the refusals -----------------------------------------------------------
@test "a gram short by one float is refused and the file is named" {
C="$BATS_TEST_TMPDIR/short"
full_recording "$C"
# Cut the last 4 bytes of one expert gram: a torn write, which is what a
# killed tap actually leaves behind.
p="$C/gram.r0.blk.0.ffn_down_exps.0.bin"
python3 -c 'import sys
p = sys.argv[1]
open(p, "wb").write(open(p, "rb").read()[:-4])' "$p"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"*"SHORT"* ]]
[[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
[ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
}
@test "an empty recording directory is refused" {
mkdir -p "$BATS_TEST_TMPDIR/empty"
run python3 "$TOOL" "$BATS_TEST_TMPDIR/empty" -o "$BATS_TEST_TMPDIR/no.gguf"
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"*"NO_GRAMS"* ]]
[[ "$output" == *"$BATS_TEST_TMPDIR/empty"* ]]
[ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
}
@test "expert grams with no pooled gram for the layer are refused" {
# Without the pooled gram there is nothing for an under-threshold expert to
# inherit, and the tool would quietly invent a zero diagonal for it.
C="$BATS_TEST_TMPDIR/nopooled"
mkdir -p "$C"
make_gram "$C/gram.r0.blk.3.ffn_down_exps.7.bin" 4 0 500 0 "1,2,3,4"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"*"NO_POOLED"*"layer 3"* ]]
[[ "$output" == *"gram.r<rank>.blk.3.ffn_down_exps.bin"* ]]
}
@test "the same global expert recorded twice with different numbers is refused" {
# Rank files carry GLOBAL expert numbering and so describe disjoint experts;
# a collision means the recording is corrupt, and whichever file sorted last
# would win.
C="$BATS_TEST_TMPDIR/conflict"
full_recording "$C"
make_gram "$C/gram.r1.blk.0.ffn_down_exps.0.bin" 4 1 77 0 "10,20,30,40"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"*"EXPERT_CONFLICT"*"expert 0"* ]]
[[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
[[ "$output" == *"gram.r1.blk.0.ffn_down_exps.0.bin"* ]]
}
@test "an existing output is refused by name rather than overwritten" {
# The trash law: a re-run must not destroy the imatrix a convert is reading.
C="$BATS_TEST_TMPDIR/full"
full_recording "$C"
: > "$BATS_TEST_TMPDIR/keep.gguf"
printf 'do not clobber me\n' > "$BATS_TEST_TMPDIR/keep.gguf"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/keep.gguf"
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"*"EXISTS"*"keep.gguf"* ]]
run cat "$BATS_TEST_TMPDIR/keep.gguf"
[[ "$output" == *"do not clobber me"* ]]
}
@test "--min-rows moves the threshold and the bytes follow it" {
# The default threshold is the engine's own law (4*k rows, lq_hess.h), but
# the operator has to be able to raise it when a corpus is thin. Asserted on
# the slices, not on a report line: a flag that only changes prose is a
# flag that changes nothing.
C="$BATS_TEST_TMPDIR/floor"
full_recording "$C"
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/far.gguf" --min-rows 200
[ "$status" -eq 0 ]
# 200 is above every expert's routing (100, 3, 64, 0, 100 and layer 1's 48),
# so all six slices now inherit.
[[ "$output" == *"inherited=6"* ]]
run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
# The counts do not move with the threshold: routing is a measurement, not a
# judgement about how much data is enough.
run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.counts 0 5
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
}
+337
View File
@@ -0,0 +1,337 @@
#!/usr/bin/env bats
# Seam: the BF16 REFERENCE container of the Qwen3.8-Flash-Next five-bit
# campaign (feat/qwen38-w5nl64) — scripts/build-qwen38-bf16-rad.sh and
# scripts/run-qwen38-bf16.sh.
#
# The reference is what every later stage is judged or derived from (the KLD
# record scores the arms, the Grams derive the protected experts), so the
# deliverable here is the set of refusals: a half checkpoint, a second convert,
# a volume that cannot hold the convert, an engine-refused plan, and any run —
# including a dry one — that would take the GPUs from frank-archon. Each test
# drives the script's own exit status and printed argv. Offline: no engine, no
# GPU, no container, no systemctl, no network. Path knobs aim at
# $BATS_TEST_TMPDIR.
#
# The third mode, score, carries the campaign's comparability law: its engine
# argv must equal kldref's except for --model and the --kld-* pair, or the KL
# numbers of the FP8-native anchor, the served payload and the arms are not one
# measurement series. The seam proves that by diffing the two dry runs.
# The 1.8.1 law asserted in two places below: --vram-kv-mib was REMOVED at
# 1.3.0, so naming it is a boot refusal — the reference serve must carry the
# budget-resolver knobs (--host-pool-mib / --gpu-headroom-mib) instead.
load helpers
BUILD="$REPO_ROOT/scripts/build-qwen38-bf16-rad.sh"
RUNNER="$REPO_ROOT/scripts/run-qwen38-bf16.sh"
# The reference shape, spelled exactly as run-qwen38-bf16.sh must pass it: this
# is the contract the recorded reference is measured at, so a silent edit here
# is a silent edit to every KLD number in the campaign.
SHAPE="--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 49152 --gpu-headroom-mib 96 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --ngram-placement ram --override-chat-template /models/tpl.jinja"
# oneline — both dry runs print one shell-quoted argument a line; joining them
# lets an option and its value be asserted as the pair they are.
oneline() { local s="${output//$'\n'/ }"; printf '%s' "$s"; }
# engine_argv — the arguments AFTER the image line of a dry run, with --model
# and every --kld-* flag and its value removed: what is left is the measuring
# shape, and it must be identical between kldref and score.
engine_argv() {
local seen=0 skip=0 a
while IFS= read -r a; do
if [ "$seen" = 0 ]; then
[[ "$a" == *"radiance@sha256:"* ]] && seen=1
continue
fi
case "$a" in
--model|--kld-record|--kld-corpus|--kld-ref|--kld-out) skip=1; continue ;;
esac
if [ "$skip" = 1 ]; then skip=0; continue; fi
printf '%s\n' "$a"
done <<< "$1"
}
# flag_value FLAG — the value a dry run printed for FLAG (the argv is one
# argument a line, so the value is the line after it). Lets a test ask what
# number the script chose instead of re-declaring the literal it guards.
flag_value() {
local want="$1" prev="" a
while IFS= read -r a; do
if [ "$prev" = "$want" ]; then printf '%s' "$a"; return 0; fi
prev="$a"
done
return 1
}
@test "the build dry run prints the rad-convert argv for the checkpoint and the bf16 output" {
M="$BATS_TEST_TMPDIR/models"
run env FRANK_QWEN38_MODELS="$M" bash "$BUILD" --dry-run
[ "$status" -eq 0 ]
[[ "$output" == *"rad-convert"* ]]
# the argv is the container's view: the models dir rides the /models mount
[[ "$output" == *"$M:/models"* ]]
flat="$(oneline)"
[[ "$flat" == *"/models/Qwen3.8-Flash-Next -o /models/rad/qwen38-bf16.rad"* ]]
# a reference is the checkpoint re-headed, not re-encoded, and it declares a
# window the long KLD corpus can be read inside of
[[ "$output" != *"--recipe"* ]]
[[ "$output" != *"--reuse"* ]]
[[ "$flat" == *"--max-ctx 32768"* ]]
# and a dry run looked at nothing: the fixture dirs were never created
[ ! -e "$M" ]
}
@test "the build refuses a checkpoint that is not on disk and names the fetch command" {
M="$BATS_TEST_TMPDIR/models"
run env FRANK_QWEN38_MODELS="$M" bash "$BUILD"
[ "$status" -eq 2 ]
[[ "$output" == *"$M/Qwen3.8-Flash-Next"* ]]
[[ "$output" == *"scripts/download-qwen38-bf16.sh --fetch"* ]]
[ ! -e "$M/rad/qwen38-bf16.rad" ]
}
@test "the build refuses an existing reference container and names --force as the way past it" {
M="$BATS_TEST_TMPDIR/models"
mkdir -p "$M/Qwen3.8-Flash-Next" "$M/rad"
echo "not a container" > "$M/rad/qwen38-bf16.rad"
run env FRANK_QWEN38_MODELS="$M" QWEN38_MIN_FREE_MIB=1 \
QWEN38_TRASH="$BATS_TEST_TMPDIR/trash" bash "$BUILD"
[ "$status" -eq 2 ]
[[ "$output" == *"$M/rad/qwen38-bf16.rad"* ]]
[[ "$output" == *"--force"* ]]
# refusing retires nothing: the trash law means the file survives a refusal
[ -f "$M/rad/qwen38-bf16.rad" ]
[ ! -e "$BATS_TEST_TMPDIR/trash" ]
}
@test "the build refuses when the output filesystem cannot hold a second 335 GiB" {
# The campaign's peak-disk law: 343,323 MiB of container written next to
# 343,323 MiB of checkpoint. The wanted figure is set above whatever this
# box really has, which is what the gate reads.
M="$BATS_TEST_TMPDIR/models"
mkdir -p "$M/Qwen3.8-Flash-Next" "$M/rad"
run env FRANK_QWEN38_MODELS="$M" QWEN38_MIN_FREE_MIB=999999999 bash "$BUILD"
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"* ]]
[[ "$output" =~ [0-9]+" MiB free" ]]
[[ "$output" == *"999999999 MiB wanted"* ]]
# the disk gate is the cheap one: it refuses long before the Hub sweep runs
[[ "$output" != *"VERIFY_OK"* ]]
[ ! -e "$M/rad/qwen38-bf16.rad" ]
}
@test "the reference serve rejects an unknown mode and names every valid mode" {
# `probe` is the swift15 era's third mode; it is retired and must not boot
# a container here either.
run bash "$RUNNER" probe
[ "$status" -eq 2 ]
[[ "$output" == *"probe"* ]]
[[ "$output" == *"kldref"* ]]
[[ "$output" == *"calib"* ]]
[[ "$output" == *"score"* ]]
[[ "$output" != *"podman"* ]]
run bash "$RUNNER"
[ "$status" -eq 2 ]
[[ "$output" == *"kldref"* ]]
[[ "$output" == *"calib"* ]]
[[ "$output" == *"score"* ]]
# score without its two positionals is the same usage refusal
run bash "$RUNNER" score
[ "$status" -eq 2 ]
[[ "$output" == *"score RAD OUT"* ]]
}
@test "the kldref dry run carries the record flags at the reference shape and no --vram-kv-mib" {
run bash "$RUNNER" kldref --dry-run
[ "$status" -eq 0 ]
flat="$(oneline)"
[[ "$flat" == *"--kld-record /data/kld-ref"* ]]
[[ "$flat" == *"--kld-corpus /data/kld.jsonl"* ]]
# the corpus arrives read-only, the record dir writable
[[ "$flat" == *":/data/kld.jsonl:ro"* ]]
[[ "$flat" == *":/data/kld-ref "* ]]
[[ "$flat" == *"$SHAPE"* ]]
# 1.8.1: the flag does not exist — naming it is a boot refusal
[[ "$output" != *"--vram-kv-mib"* ]]
}
@test "the calib dry run opens the tap at 12M tokens and serves on 7770" {
run bash "$RUNNER" calib --dry-run
[ "$status" -eq 0 ]
flat="$(oneline)"
[[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]]
[[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]]
[[ "$flat" == *"--port 7770"* ]]
[[ "$flat" == *":/data/calib "* ]]
# calib serves; it does not record
[[ "$output" != *"--kld-record"* ]]
}
@test "a dry run prints argv only: no drain, no container, nothing written" {
L="$BATS_TEST_TMPDIR/logs"
run env FRANK_QWEN38_LOG_DIR="$L" \
bash "$RUNNER" kldref --dry-run --image example.invalid/radiance@sha256:0000000000000000000000000000000000000000000000000000000000000000
[ "$status" -eq 0 ]
[[ "$output" != *"systemctl"* ]]
[[ "$output" != *"frank-archon stopped"* ]]
[[ "$output" != *"podman rm"* ]]
# not vacuous: the podman argv WAS printed, and --image replaced the pin
[[ "$output" == *"podman"* ]]
[[ "$output" == *"example.invalid/radiance@sha256:000"* ]]
[[ "$output" != *"stilldeadcode/radiance"* ]]
[ ! -d "$L" ]
}
@test "the score dry run is the kldref argv with only --model and the --kld-* pair changed" {
M="$BATS_TEST_TMPDIR/models"
C="$BATS_TEST_TMPDIR/campaign"
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
bash "$RUNNER" kldref --dry-run
[ "$status" -eq 0 ]
ref_argv="$(engine_argv "$output")"
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
[ "$status" -eq 0 ]
score_argv="$(engine_argv "$output")"
# The whole value of the campaign's three KL numbers is that they rode this
# one shape: same pin, --tp 2, fp8 KV, step size, --deterministic, no spec.
if [[ "$score_argv" != "$ref_argv" ]]; then
echo "--- kldref engine argv ---"; echo "$ref_argv"
echo "--- score engine argv ---"; echo "$score_argv"
return 1
fi
flat="$(oneline)"
[[ "$flat" == *"--model /models/rad/arm-w5nl64.rad"* ]]
[[ "$flat" == *"--kld-ref /data/kld-ref --kld-out /data/out/arm1.json"* ]]
# the zero point mounts read-only; the report needs a writable bind because
# --kld-out F also writes F.rows beside it
[[ "$flat" == *":/data/kld-ref:ro "* ]]
[[ "$flat" == *":/data/out "* ]]
[[ "$output" != *"--kld-record"* ]]
[[ "$output" != *"--kld-corpus"* ]]
[[ "$output" != *"--vram-kv-mib"* ]]
}
@test "score refuses a missing reference, a missing container and an existing report" {
M="$BATS_TEST_TMPDIR/models"
C="$BATS_TEST_TMPDIR/campaign"
L="$BATS_TEST_TMPDIR/logs"
mkdir -p "$M/rad"
: > "$M/rad/arm-w5nl64.rad"
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json"
[ "$status" -eq 2 ]
[[ "$output" == *"$C/kld-ref"* ]]
[[ "$output" == *"run-qwen38-bf16.sh kldref"* ]]
# refused before the log dir, so before anything that could reach systemctl
[ ! -d "$L" ]
mkdir -p "$C/kld-ref"
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
bash "$RUNNER" score "$M/rad/gone.rad" "$C/scores/arm1.json"
[ "$status" -eq 2 ]
[[ "$output" == *"$M/rad/gone.rad"* ]]
[ ! -d "$L" ]
mkdir -p "$C/scores"
: > "$C/scores/arm1.json"
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json"
[ "$status" -eq 2 ]
[[ "$output" == *"$C/scores/arm1.json"* ]]
[[ "$output" == *"trash"* ]]
# the trash law: a refusal keeps the old report, it does not overwrite it
[ -f "$C/scores/arm1.json" ]
[ ! -d "$L" ]
}
@test "the reference and a score pass ride ONE step size, legal on the file tier" {
# The gate's claim is one shape; that shape's ceiling is set by the tightest
# payload it boots, and the served iq4r and every arm are file-tier
# containers, which stream only at <=2048. A reference recorded above that is
# a KL between two shapes, not two weights.
M="$BATS_TEST_TMPDIR/models"
C="$BATS_TEST_TMPDIR/campaign"
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" bash "$RUNNER" kldref --dry-run
[ "$status" -eq 0 ]
ref_step="$(flag_value --max-num-batched-tokens <<< "$output")"
[ -n "$ref_step" ]
[ "$ref_step" -le 2048 ]
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
[ "$status" -eq 0 ]
score_step="$(flag_value --max-num-batched-tokens <<< "$output")"
[ "$ref_step" = "$score_step" ]
# one knob moves both, so the operator can lower it at 2 a.m. without an edit
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
bash "$RUNNER" kldref --dry-run
[ "$(flag_value --max-num-batched-tokens <<< "$output")" = "1024" ]
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
[ "$(flag_value --max-num-batched-tokens <<< "$output")" = "1024" ]
# calibration is exempt: grams are not logits, so the tap keeps the wide step
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
bash "$RUNNER" calib --dry-run
[ "$(flag_value --max-num-batched-tokens <<< "$output")" = "65536" ]
# a step size that is not a token count is refused, never handed to the engine
run env FRANK_QWEN38_MNBT=broad bash "$RUNNER" kldref --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"FRANK_QWEN38_MNBT"* ]]
[[ "$output" == *"must be a token count"* ]]
}
@test "kldref refuses a corpus its window cannot hold, before the service is touched" {
M="$BATS_TEST_TMPDIR/models" R="$M/rad"
mkdir -p "$M/tpl" "$R" "$BATS_TEST_TMPDIR/out"
: > "$M/tpl/qwen38-froggeric-v22.5.jinja"
: > "$R/qwen38-bf16.rad"
CORPUS="$BATS_TEST_TMPDIR/long.jsonl"
{
printf '{"prompt":"short one"}\n'
printf '{"prompt":"%s"}\n' "$(head -c 120000 /dev/zero | tr '\0' 'x')"
printf '{"prompt":"short three"}\n'
} > "$CORPUS"
# 32,768 tokens x 3 chars is the pessimistic limit; the 120,002-char line
# that opens record 2 is well past it, so the campaign says no up front.
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
bash "$RUNNER" kldref --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"REFUSED"* ]]
[[ "$output" == *"record 2"* ]]
[[ "$output" == *"120013 chars"* ]]
[[ "$output" == *"past the 98304 a 32768-token window"* ]]
[[ "$output" == *"FRANK_QWEN38_MAX_MODEL_LEN"* ]]
# the same corpus is fine once the window is raised past it, and the raised
# window is what the boot is asked for
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
FRANK_QWEN38_MAX_MODEL_LEN=65536 FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" \
bash "$RUNNER" kldref --dry-run
[ "$status" -eq 0 ]
[[ "$output" == *"corpus fits"* ]]
flat="$(oneline)"
[[ "$flat" == *"--max-model-len 65536"* ]]
# the shipped corpus of three short records fits the shipped window
SHORT="$BATS_TEST_TMPDIR/short.jsonl"
printf '{"prompt":"tiny"}\n{"prompt":"also tiny"}\n' > "$SHORT"
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$SHORT" \
FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" bash "$RUNNER" kldref --dry-run
[ "$status" -eq 0 ]
[[ "$output" == *"corpus fits: longest record 22 chars (number 2)"* ]]
# a real kldref run refuses at the same place: no log dir, no drain, and the
# service would still be up when the script gave up
L="$BATS_TEST_TMPDIR/logs-real"
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
FRANK_QWEN38_LOG_DIR="$L" bash "$RUNNER" kldref
[ "$status" -eq 2 ]
[ ! -d "$L" ]
# and it is kldref that reads the corpus: score and calib never consult it
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
FRANK_QWEN38_KLD_REF_DIR="$BATS_TEST_TMPDIR/ref" \
FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" \
bash "$RUNNER" score "$R/qwen38-bf16.rad" "$BATS_TEST_TMPDIR/out/arm.json" --dry-run
[ "$status" -eq 0 ]
[[ "$output" != *"corpus fits"* ]]
[[ "$output" != *"REFUSED"* ]]
}
+3 -2
View File
@@ -20,8 +20,9 @@ SEERR_UNIT="$REPO_ROOT/dankclaw/seerr/quadlet/seerr.container"
grep -q '^Restart=always$' "$SEERR_UNIT"
}
@test "seerr image is pinned to an exact version tag" {
grep -Eq '^Image=ghcr\.io/seerr-team/seerr:v[0-9]+\.[0-9]+\.[0-9]+$' "$SEERR_UNIT"
@test "seerr image is pinned to an exact version tag and digest" {
# 2026-10-21 fleet law: tag AND digest (tests/quadlet_image_pins.bats)
grep -Eq '^Image=ghcr\.io/seerr-team/seerr:v[0-9]+\.[0-9]+\.[0-9]+@sha256:[0-9a-f]{64}$' "$SEERR_UNIT"
}
@test "seerr publishes 5055 on loopback only (guard chain is the user path)" {