Compare commits
60
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
96c5a0625c | ||
|
|
a7329d190f | ||
|
|
5fe04e569f | ||
|
|
bb027265db | ||
|
|
4e551fafd7 | ||
|
|
3d8d18a45c | ||
|
|
c13a2568d4 | ||
|
|
e7a835e41f | ||
|
|
0ecfe830f7 | ||
|
|
ce370a5b28 | ||
|
|
bfe4d185bd | ||
|
|
94a9f3aa31 | ||
|
|
d2810ab610 | ||
|
|
7a8cc83385 | ||
|
|
c4623f1629 | ||
|
|
065a6911f9 | ||
|
|
745d66d60d | ||
|
|
4009f118c1 | ||
|
|
7de343c171 | ||
|
|
139a3cd4fa | ||
|
|
5ffbf19fb2 | ||
|
|
1fd98cab3f | ||
|
|
d61c19d41f | ||
|
|
e6fcf217b4 | ||
|
|
cb86ad8fd7 | ||
|
|
078b550bd8 | ||
|
|
765a8cb7de | ||
|
|
999c4fe440 | ||
|
|
2af2e6ac56 | ||
|
|
fbe54ca1ad | ||
|
|
f073b91033 | ||
|
|
425994515a | ||
|
|
c41384e03d | ||
|
|
127621bdd7 | ||
|
|
c96d0228a4 | ||
|
|
a61308d72e | ||
|
|
ccb805cf0b | ||
|
|
c6e2887f94 | ||
|
|
9eb53ef937 | ||
|
|
688928d489 | ||
|
|
56503dbad1 | ||
|
|
fbeb9701e4 | ||
|
|
650b6e0281 | ||
|
|
b62a05837f | ||
|
|
0367244d2a | ||
|
|
656d9567d5 | ||
|
|
0b4bc60a38 | ||
|
|
a2011f4eb6 | ||
|
|
b02647b992 | ||
|
|
b35269bfa5 | ||
|
|
1aeb7af6df | ||
|
|
cd0be97b39 | ||
|
|
7993a336bf | ||
|
|
1bb6399524 | ||
|
|
31b71a16d6 | ||
|
|
a4edaa7d9d | ||
|
|
aa75b45543 | ||
|
|
b1fc1113c6 | ||
|
|
9f7457e39e | ||
|
|
687dbfbabe |
No files matched your search
@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
|
||||
(`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
|
||||
IP, the one bind a container can reach as `host.containers.internal`;
|
||||
iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
|
||||
Prometheus targets read down for structural reasons (HTML-scraped "probes",
|
||||
Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
|
||||
a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
|
||||
without `--metrics`. See
|
||||
`serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||||
The former structural target downs (HTML-scraped "probes", Gitea metrics off)
|
||||
are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
|
||||
(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
|
||||
every Prometheus job reports up. `frank-zealot` was down for a different reason
|
||||
until 2026-10-18: llama.cpp will not serve `/metrics` at all without
|
||||
`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -419,16 +420,17 @@ disagrees with it is the bug.
|
||||
digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
|
||||
Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
|
||||
`/home/dank/logs/frank_dragoon.log`, NOT the journal.
|
||||
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
|
||||
is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
|
||||
`HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
|
||||
the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
|
||||
drafter always armed (the adaptive default disarms it and halves decode);
|
||||
server sampler 1.0/0.95/20.
|
||||
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
|
||||
The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
|
||||
under-counts, so a harness window equal to the ceiling makes every long
|
||||
request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
|
||||
- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
|
||||
**per-request ceiling** (a longer prompt is a 400, never a truncation);
|
||||
`HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
|
||||
pool positions buy concurrency, the per-request window never moves;
|
||||
`HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
|
||||
default disarms it and halves decode); server sampler 1.0/0.95/20.
|
||||
- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
|
||||
ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
|
||||
estimate under-counts (trigger is 95 %), so a harness window at the ceiling
|
||||
makes every long request a guaranteed 400.
|
||||
Reasoning arrives in `reasoning_content`; the
|
||||
template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
|
||||
OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
|
||||
`/completion`, `/ping`, `/props`, `/tokenize` are 404.
|
||||
|
||||
@@ -20,7 +20,7 @@ Wants=authentik-server.service
|
||||
|
||||
[Container]
|
||||
ContainerName=authentik-outpost
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
|
||||
Exec=worker
|
||||
Network=dankclaw_default
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
|
||||
@@ -14,7 +14,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=authentik-postgres
|
||||
Image=docker.io/library/postgres:16-alpine
|
||||
Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
Network=dankclaw_default
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
Environment=POSTGRES_DB=authentik
|
||||
|
||||
@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
|
||||
|
||||
[Container]
|
||||
ContainerName=authentik-server
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
|
||||
Exec=server
|
||||
Network=dankclaw_default
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
|
||||
@@ -16,7 +16,7 @@ Wants=authentik-server.service
|
||||
|
||||
[Container]
|
||||
ContainerName=authentik-worker
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
|
||||
Exec=worker
|
||||
Network=dankclaw_default
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
|
||||
@@ -15,7 +15,7 @@ After=network.target
|
||||
[Container]
|
||||
ContainerName=gitea-postgres
|
||||
Network=gitea.network
|
||||
Image=docker.io/library/postgres:16-alpine
|
||||
Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
EnvironmentFile=/home/dank/Containers/gitea/env
|
||||
Environment=POSTGRES_USER=gitea
|
||||
Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}
|
||||
|
||||
@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
|
||||
[Container]
|
||||
ContainerName=gitea-server
|
||||
Network=gitea.network
|
||||
Image=docker.io/gitea/gitea:latest
|
||||
Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
|
||||
EnvironmentFile=/home/dank/Containers/gitea/env
|
||||
Environment=DATABASE_TYPE=postgres
|
||||
Environment=DATABASE_HOST=gitea-postgres
|
||||
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
|
||||
Environment=DATABASE_PORT=5432
|
||||
Environment=USER_UID=1000
|
||||
Environment=USER_GID=1000
|
||||
# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
|
||||
# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
|
||||
# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
|
||||
# the container; the only route in is this host's published 3000, and the
|
||||
# exposed surface is counters only — same posture as the LLM /metrics.
|
||||
Environment=GITEA__metrics__ENABLED=true
|
||||
Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
|
||||
PublishPort=0.0.0.0:3000:3000
|
||||
PublishPort=2222:22
|
||||
Volume=/home/dank/Containers/gitea/data:/data
|
||||
|
||||
@@ -23,7 +23,7 @@ After=gluetun.service network.target
|
||||
[Container]
|
||||
ContainerName=marreta_browser
|
||||
Network=gluetun.container
|
||||
Image=docker.io/lightpanda/browser:nightly
|
||||
Image=docker.io/lightpanda/browser:nightly@sha256:979c36ed869f1016b8dcf22939a0633c262f2c30e892a03a9f720eba2ff1931e
|
||||
|
||||
[Service]
|
||||
TimeoutStartSec=300
|
||||
|
||||
@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
|
||||
[Container]
|
||||
ContainerName=marreta
|
||||
Network=gluetun.container
|
||||
Image=ghcr.io/manualdousuario/marreta:latest
|
||||
Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
|
||||
# Relocate off SearXNG's 8080 inside the shared netns.
|
||||
Environment=NGINX_HTTP_PORT=8085
|
||||
Environment=APP_NAME=Marreta
|
||||
|
||||
@@ -33,7 +33,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=alertmanager
|
||||
Image=docker.io/prom/alertmanager:latest
|
||||
Image=docker.io/prom/alertmanager:latest@sha256:e9733bafb1bdef9b00e25a21f8f99dc26a22224bf16641ad754d1649f4c3357a
|
||||
Network=dankclaw_default
|
||||
User=0:0
|
||||
Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw:
|
||||
# ~/.config/containers/systemd/blackbox-exporter.container
|
||||
#
|
||||
# WHY Network=host. The probes it runs are (a) the host's own LAN IP
|
||||
# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
|
||||
# bridge container cannot reach; netavark DNATs published ports only, see
|
||||
# Container Host Reachability in the vault) and (b) ComfyUI's published port
|
||||
# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
|
||||
# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
|
||||
# reach neither class reliably.
|
||||
#
|
||||
# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
|
||||
# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
|
||||
# bridge container dialing the podman gateway 169.254.1.2 lands on the host
|
||||
# LOOPBACK — that is the scrape path, and it needs this exporter to be
|
||||
# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
|
||||
# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
|
||||
# address itself is NOT bindable from the host netns — it exists only
|
||||
# inside each rootless netns; binding it dies "bind: cannot assign
|
||||
# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
|
||||
# No public DNS name, no guard chain: internal net only, per the exposure
|
||||
# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
|
||||
# flag per bind; plain --listen-address does not exist and the exporter
|
||||
# exits in 100 ms with "unknown long flag" (live 2026-10-21).
|
||||
|
||||
[Unit]
|
||||
Description=Blackbox exporter (HTTP probe prober)
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
|
||||
ContainerName=blackbox-exporter
|
||||
Network=host
|
||||
# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
|
||||
# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
|
||||
# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
|
||||
# exits "unexpected /bin/blackbox_exporter".
|
||||
Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
|
||||
Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
|
||||
# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
|
||||
# is passed as one quoted token and quadlet then folds the following
|
||||
# --health-interval/--health-retries (and the image name) into the command
|
||||
# string — unit died twice that way, live 2026-10-21. wget only; this
|
||||
# busybox variant has no curl, and host-network wget answers on loopback.
|
||||
HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
|
||||
HealthInterval=30s
|
||||
HealthRetries=3
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -35,7 +35,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=cadvisor
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
|
||||
PodmanArgs=--pid=host --cgroupns=host
|
||||
PublishPort=127.0.0.1:8080:8080
|
||||
PublishPort=192.168.0.143:8080:8080
|
||||
|
||||
@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
|
||||
|
||||
[Container]
|
||||
ContainerName=grafana
|
||||
Image=docker.io/grafana/grafana:latest
|
||||
Image=docker.io/grafana/grafana:latest@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
|
||||
Network=dankclaw_default
|
||||
User=0:0
|
||||
PublishPort=127.0.0.1:3002:3000
|
||||
|
||||
@@ -33,7 +33,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=node-exporter
|
||||
Image=docker.io/prom/node-exporter:latest
|
||||
Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
|
||||
Network=host
|
||||
PodmanArgs=--pid=host
|
||||
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100
|
||||
|
||||
@@ -43,8 +43,16 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=prometheus
|
||||
Image=docker.io/prom/prometheus:latest
|
||||
Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
|
||||
Network=dankclaw_default
|
||||
# Same-host scrape targets are dialed at host.containers.internal (the podman
|
||||
# gateway): a bridge container cannot dial its own host's LAN IP (netavark
|
||||
# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
|
||||
# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
|
||||
# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
|
||||
# it unsupported and the WHOLE user quadlet batch then fails to generate,
|
||||
# silently orphaning every other unit (live 2026-10-21: prometheus.service
|
||||
# vanished while its container kept running as an orphan).
|
||||
User=0:0
|
||||
Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
|
||||
PublishPort=127.0.0.1:9090:9090
|
||||
|
||||
@@ -27,7 +27,7 @@ After=network.target
|
||||
[Container]
|
||||
ContainerName=open-webui
|
||||
Network=open-webui.network
|
||||
Image=ghcr.io/open-webui/open-webui:main
|
||||
Image=ghcr.io/open-webui/open-webui:main@sha256:4860ba005ae8f19f08d1e12b0868855b7b339ea0dad9927f196820fc0de04575
|
||||
PublishPort=3005:8080
|
||||
Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
|
||||
Environment=ANONYMIZED_TELEMETRY=False
|
||||
|
||||
@@ -24,7 +24,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=pihole
|
||||
Image=docker.io/pihole/pihole:latest
|
||||
Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
|
||||
Network=slirp4netns
|
||||
PublishPort=0.0.0.0:53:53/udp
|
||||
PublishPort=0.0.0.0:53:53/tcp
|
||||
|
||||
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
|
||||
[Container]
|
||||
ContainerName=gluetun
|
||||
Network=searxng.network
|
||||
Image=docker.io/qmcgaw/gluetun:latest
|
||||
Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
|
||||
AddCapability=NET_ADMIN
|
||||
AddDevice=/dev/net/tun:/dev/net/tun
|
||||
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
|
||||
|
||||
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
|
||||
ContainerName=searxng-valkey
|
||||
Network=searxng.network
|
||||
NetworkAlias=valkey
|
||||
Image=docker.io/valkey/valkey:9-alpine
|
||||
Image=docker.io/valkey/valkey:9-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
|
||||
Volume=/opt/searxng/valkey:/data
|
||||
|
||||
[Service]
|
||||
|
||||
@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
|
||||
[Container]
|
||||
ContainerName=searxng
|
||||
Network=gluetun.container
|
||||
Image=ghcr.io/searxng/searxng:latest
|
||||
Image=ghcr.io/searxng/searxng:latest@sha256:5ed0323b5f5f8701264ef65e239a2bd83cb57098f2b971fa30bf79c06eeae271
|
||||
Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
|
||||
Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
|
||||
Volume=/opt/searxng/config:/etc/searxng
|
||||
|
||||
@@ -29,7 +29,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=seerr
|
||||
Image=ghcr.io/seerr-team/seerr:v3.4.1
|
||||
Image=ghcr.io/seerr-team/seerr:v3.4.1@sha256:f4768de5f616248d723e05891f3345a1402123775d03bf0890dbfedc0831bda1
|
||||
Network=dankclaw_default
|
||||
PublishPort=127.0.0.1:5055:5055
|
||||
Volume=/home/dank/Containers/seerr/data:/app/config
|
||||
|
||||
@@ -43,7 +43,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=calibre-server
|
||||
Image=lscr.io/linuxserver/calibre:9.15.0
|
||||
Image=lscr.io/linuxserver/calibre:9.15.0@sha256:59e4b0e036e62f228f8bcfae1ade42f47047719885fe454ddbad38e4597b987b
|
||||
Network=pasta
|
||||
EnvironmentFile=/home/dank/Containers/calibre-server/env
|
||||
Environment=PUID=1000
|
||||
|
||||
@@ -16,7 +16,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=calibre-web
|
||||
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
|
||||
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402@sha256:802729966faed5e32d671cd4f92ba6196e9e754a6c0f740ce1ee9acd86b10e55
|
||||
Network=pasta
|
||||
Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
|
||||
Environment=LANG=en_US.UTF-8
|
||||
|
||||
@@ -21,7 +21,13 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=copyparty
|
||||
Image=ghcr.io/9001/copyparty-ac:latest
|
||||
# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
|
||||
# the pin rides :latest@digest (the tag in the reference is cosmetic; the
|
||||
# digest is the pull). Pinned 2026-10-21 at the build whose label is
|
||||
# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
|
||||
# (1.20.26); the previously running build was the rolling :latest at
|
||||
# 1.20.24 (issue #73 on command-center).
|
||||
Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
|
||||
Network=dankserver_appnet
|
||||
EnvironmentFile=/home/dank/Containers/copyparty/env
|
||||
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank
|
||||
|
||||
@@ -16,7 +16,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=jellyfin
|
||||
Image=ghcr.io/jellyfin/jellyfin:latest
|
||||
Image=ghcr.io/jellyfin/jellyfin:latest@sha256:c3efa69a22f2691fc88d6bb035e87a8560e395a6047f062fb381c61c3185e8f5
|
||||
Network=dankserver_appnet
|
||||
Environment=DEBIAN_FRONTEND=noninteractive
|
||||
Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
|
||||
|
||||
@@ -18,7 +18,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=cadvisor
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
|
||||
PodmanArgs=--pid=host --cgroupns=host
|
||||
PublishPort=127.0.0.1:8080:8080
|
||||
PublishPort=192.168.0.235:8080:8080
|
||||
|
||||
@@ -37,7 +37,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=node-exporter
|
||||
Image=docker.io/prom/node-exporter:latest
|
||||
Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
|
||||
Network=host
|
||||
PodmanArgs=--pid=host
|
||||
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100
|
||||
|
||||
@@ -4,7 +4,7 @@ After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
|
||||
Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329@sha256:8f655bad7dcbc5398a78f70a5af12600b4ed3424f901566ed1eaa6710e9e2334
|
||||
ContainerName=plex
|
||||
Network=dankserver_appnet:alias=plex
|
||||
Environment=PUID=0
|
||||
|
||||
@@ -14,7 +14,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=romm-mariadb
|
||||
Image=docker.io/library/mariadb:11
|
||||
Image=docker.io/library/mariadb:11@sha256:f16d7a344413e519dff518f27bdb221388f84b0d274c97c4fb5f2cb0dc1b34fa
|
||||
Network=dankserver_appnet:alias=mariadb
|
||||
EnvironmentFile=/home/dank/Containers/romm/env
|
||||
Environment=MARIADB_DATABASE=romm
|
||||
|
||||
@@ -12,7 +12,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=romm-redis
|
||||
Image=docker.io/library/redis:7-alpine
|
||||
Image=docker.io/library/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
Network=dankserver_appnet:alias=redis
|
||||
Volume=/home/dank/Containers/romm/redis:/data
|
||||
HealthCmd=redis-cli ping
|
||||
|
||||
@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=romm
|
||||
Image=ghcr.io/rommapp/romm:latest
|
||||
Image=ghcr.io/rommapp/romm:latest@sha256:03295f95a6290e752641f9553ac4198b0e5b71b879d57f7b2371b98f608feb54
|
||||
Network=dankserver_appnet
|
||||
EnvironmentFile=/home/dank/Containers/romm/env
|
||||
Environment=DB_HOST=mariadb
|
||||
|
||||
@@ -4,7 +4,7 @@ After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
|
||||
Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245@sha256:c66d3afa8f7a701a48dd1b1f57b22eca0483d895d670fe11ecff783f6a9bbd73
|
||||
ContainerName=tautulli
|
||||
Network=dankserver_appnet:alias=tautulli
|
||||
Environment=PUID=0
|
||||
|
||||
@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
|
||||
|
||||
[Container]
|
||||
ContainerName=gluetun
|
||||
Image=docker.io/qmcgaw/gluetun:latest
|
||||
Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
|
||||
AddCapability=NET_ADMIN
|
||||
AddDevice=/dev/net/tun:/dev/net/tun
|
||||
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
|
||||
|
||||
@@ -15,7 +15,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=lidarr
|
||||
Image=lscr.io/linuxserver/lidarr:latest
|
||||
Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -24,7 +24,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=prowlarr
|
||||
Image=lscr.io/linuxserver/prowlarr:latest
|
||||
Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -29,7 +29,7 @@ Network=gluetun.container
|
||||
# Pinned to the last 4.x line: Readarr's retired qB client requires the
|
||||
# qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
|
||||
# empty body, which fails Readarr's auth check. All other clients accept 4.x.
|
||||
Image=lscr.io/linuxserver/qbittorrent:4.6.7
|
||||
Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -15,7 +15,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=radarr
|
||||
Image=lscr.io/linuxserver/radarr:latest
|
||||
Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -19,7 +19,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=readarr
|
||||
Image=lscr.io/linuxserver/readarr:amd64-1.0.980
|
||||
Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -16,7 +16,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=sonarr
|
||||
Image=lscr.io/linuxserver/sonarr:latest
|
||||
Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -1,27 +1,45 @@
|
||||
#!/usr/bin/env bash
|
||||
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
|
||||
# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
|
||||
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
|
||||
# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
|
||||
# derive-protected-experts.py over the calib dir, splice its cast lines into
|
||||
# the marked section of the recipe in place, keep the pre-edit recipe in the
|
||||
# campaign trash. Refuses to run twice (marker text says derived).
|
||||
# campaign trash. Refuses to run twice (marker text says derived), and refuses
|
||||
# when a requested target count was not reached. Run it on a COPY of the
|
||||
# template so the committed recipe stays the clean, reviewable baseline.
|
||||
set -euo pipefail
|
||||
CALIB=${1:-/home/dank/corpora/calib}
|
||||
RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
|
||||
PY=/home/dank/venvs/radcorpus/bin/python
|
||||
DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
|
||||
RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
|
||||
TARGET_COUNT=${3:-0}
|
||||
PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
|
||||
DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
|
||||
TS=$(date +%Y%m%d-%H%M%S)
|
||||
# OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
|
||||
# scratch directory; on the box they are the campaign's usual places.
|
||||
OUT=${OUT:-/tmp/protected-cast-$TS.txt}
|
||||
TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
|
||||
|
||||
grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
|
||||
echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
|
||||
grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
|
||||
|
||||
"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
|
||||
N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
|
||||
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
|
||||
echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
|
||||
ARGS=("$DERIVE" "$CALIB")
|
||||
if [ "$TARGET_COUNT" -gt 0 ]; then
|
||||
ARGS+=(--target-count "$TARGET_COUNT")
|
||||
fi
|
||||
"$PY" "${ARGS[@]}" > "$OUT"
|
||||
N=$(grep -c 'cast dtype=bf16' "$OUT")
|
||||
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
|
||||
if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
|
||||
echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
|
||||
echo " (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "derived $N protected experts -> $OUT"
|
||||
|
||||
cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
|
||||
"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
|
||||
mkdir -p "$TRASH_DIR"
|
||||
cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
|
||||
"$PY" - "$RECIPE" "$OUT" <<'PYEOF'
|
||||
import sys
|
||||
recipe, castfile = sys.argv[1], sys.argv[2]
|
||||
lines = open(recipe).read().splitlines()
|
||||
@@ -35,7 +53,7 @@ for l in lines:
|
||||
elif 'PROTECTED-END' in l:
|
||||
insec = False
|
||||
out.append(l)
|
||||
out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
|
||||
out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
|
||||
elif not insec:
|
||||
out.append(l)
|
||||
open(recipe, 'w').write('\n'.join(out) + '\n')
|
||||
|
||||
Executable
+220
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env bash
|
||||
# build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
|
||||
# checkpoint /home/dank/models/Qwen3.8-Flash-Next into
|
||||
# /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
|
||||
# no rule matches keeps the checkpoint's encoding, so every tensor stays the
|
||||
# bf16 it is in the safetensors and the container is the checkpoint's own
|
||||
# numbers behind a .rad header.
|
||||
#
|
||||
# This file is the campaign's measuring stick, twice over: scripts/
|
||||
# run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
|
||||
# arm's --kld-ref is scored against that record) and the same container serves
|
||||
# with the calibration tap to write the Grams that derive the protected-expert
|
||||
# list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
|
||||
# byte in bf16, silently moves every gate in the campaign — hence no recipe, no
|
||||
# --reuse, and the four refusal gates below.
|
||||
#
|
||||
# Why a container at all: a checkpoint dir has no mapped/file tiers, and a
|
||||
# weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
|
||||
# served from one; rad-convert also bakes the tokenizer. The convert runs on
|
||||
# the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
|
||||
# kernels on CPU and production serving is untouched for the whole run (the
|
||||
# precedent that proved it: scripts/convert-swift15-w4dn8.sh).
|
||||
#
|
||||
# Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
|
||||
# written, cheapest first — the verifier runs last of them because its Hub gate
|
||||
# hashes 343 GiB and must not be paid for by a run that was going to refuse on
|
||||
# the output file anyway.
|
||||
# checkpoint the tree must be on disk; an absent dir names the command that
|
||||
# makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
|
||||
# free space the convert writes ~343,323 MiB next to the ~343,323 MiB
|
||||
# checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
|
||||
# plus 6.5 GiB of slack) is the floor. The volume that cannot
|
||||
# hold both is discovered at minute 2, not hour 5.
|
||||
# output the campaign converts the reference ONCE. An existing
|
||||
# qwen38-bf16.rad is refused; --force retires it to
|
||||
# ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
|
||||
# deleted, and the move is a rename on the same volume).
|
||||
# verified scripts/download-qwen38-bf16.sh's verify mode must pass. A
|
||||
# failed verify exits 1, not 2: the tree being the wrong payload
|
||||
# is a result, a missing environment is a refusal.
|
||||
# plan rad-convert --plan-only -v first — the oracle for "does this
|
||||
# shape boot". A refused plan costs seconds and saves a convert.
|
||||
#
|
||||
# usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
|
||||
# --dry-run print the argv it would exec (podman wrapping rad-convert),
|
||||
# one shell-quoted argument a line, and exit 0. No gate runs, no
|
||||
# container starts, nothing is written — the offline test seam.
|
||||
# It does resolve the checkpoint and output paths through the
|
||||
# /models mount, because an argv that carries host paths would
|
||||
# not run either.
|
||||
# --plan-only run the plan and every gate except the disk and output ones —
|
||||
# those two protect bytes the plan never writes, and this is the
|
||||
# probe to run when the disk gate itself is what refuses
|
||||
# --force retire an existing output container to the trash dir first
|
||||
#
|
||||
# knobs: FRANK_QWEN38_MODELS the host dir mounted at /models (default
|
||||
# /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
|
||||
# Qwen3.8-Flash-Next, the same knob the verifier reads);
|
||||
# FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
|
||||
# QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
|
||||
# verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
|
||||
# reach it through the environment untouched.
|
||||
# exit: 0 converted (or planned), 1 the checkpoint failed verification,
|
||||
# 2 refused before anything was written
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# ONE owner of the engine pin: the digest frank-archon serves on. The reference
|
||||
# must be built by the SAME engine build that scores it, or a KLD delta is
|
||||
# partly a version delta.
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/frank-archon.env"
|
||||
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
||||
|
||||
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
|
||||
CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
|
||||
OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
|
||||
TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
|
||||
MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
|
||||
LOG="/home/dank/logs/qwen38_bf16_convert.log"
|
||||
|
||||
DRY=0
|
||||
PLAN_ONLY=0
|
||||
FORCE=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY=1 ;;
|
||||
--plan-only) PLAN_ONLY=1 ;;
|
||||
--force) FORCE=1 ;;
|
||||
*) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
# The convert sees the tree only through the /models mount, so both paths must
|
||||
# be under it; the container-side spelling is what goes into the argv.
|
||||
case "$CKPT" in
|
||||
"$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
|
||||
*) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
case "$OUT" in
|
||||
"$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
|
||||
*) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
|
||||
# Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
|
||||
# bound this container will actually be served at. Both flags come off the pinned
|
||||
# image's own help: --kv-cache-dtype at convert "only picks the kernels it
|
||||
# checks" and the container is identical either way, so naming fp8 is free
|
||||
# insurance — if the fp8 KV kernels the fleet serves would not resolve, the
|
||||
# convert refuses here instead of the boot failing on a drained window.
|
||||
# --max-ctx is the bound declare is run at, whose default is the checkpoint's
|
||||
# training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
|
||||
# machine, so the default is not an option). 32,768 is set by the corpus, not by
|
||||
# taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
|
||||
# pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
|
||||
# corpus — so the window the rig boots at clears every record 2.6x over and the
|
||||
# reference boot cannot die on a sequence that will not fit. The rig's window is
|
||||
# one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
|
||||
# 32,768) and that script refuses a corpus its window cannot hold, so raising one
|
||||
# without the other is caught before the drained window, not inside it. The
|
||||
# payload containers the gate scores declare at the training context because
|
||||
# production must hold the 200k shape; a declared bound is a declare-time check,
|
||||
# not a weight change, so the reference and the payloads carry the same numbers
|
||||
# whatever each declared at — which is why one can be measured against the other.
|
||||
CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
|
||||
PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
|
||||
PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
|
||||
CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
|
||||
|
||||
if [ "$DRY" = "1" ]; then
|
||||
echo "dry run: no gate checked, no container started, nothing written"
|
||||
echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
|
||||
echo "# plan argv:"
|
||||
printf '%q\n' "${PLAN_CMD[@]}"
|
||||
echo "# convert argv:"
|
||||
printf '%q\n' "${CONV_CMD[@]}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- gate: the checkpoint is on disk ---------------------------------------
|
||||
if [ ! -d "$CKPT" ]; then
|
||||
echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ---- gate: peak disk --------------------------------------------------------
|
||||
# The deepest existing ancestor is the right thing to ask: a fresh
|
||||
# /home/dank/models/rad has no free space of its own to report.
|
||||
OUT_DIR="$(dirname "$OUT")"
|
||||
PROBE="$OUT_DIR"
|
||||
while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
|
||||
FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
|
||||
if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
|
||||
echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
|
||||
echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ---- gate: one conversion ---------------------------------------------------
|
||||
if [ -e "$OUT" ]; then
|
||||
if [ "$PLAN_ONLY" = "1" ]; then
|
||||
echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
|
||||
elif [ "$FORCE" = "1" ]; then
|
||||
mkdir -p "$TRASH"
|
||||
DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
|
||||
mv "$OUT" "$DEST"
|
||||
echo "TRASHED: retired the existing reference container $OUT -> $DEST"
|
||||
else
|
||||
echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- gate: the tree really is the checkpoint (the full verify) -------------
|
||||
echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
|
||||
rc=0
|
||||
bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
if [ "$rc" -eq 2 ]; then
|
||||
echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
|
||||
exit 2
|
||||
fi
|
||||
echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ---- plan, then convert ------------------------------------------------------
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
# rad-convert opens -o during declare as well, so the output directory exists
|
||||
# before the plan; that empty directory is all this creates ahead of it.
|
||||
mkdir -p "$OUT_DIR"
|
||||
echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
|
||||
"${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
|
||||
rc=${PIPESTATUS[0]}
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$PLAN_ONLY" = "1" ]; then
|
||||
echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "== convert: $CKPT -> $OUT ($(date -Is))"
|
||||
"${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
|
||||
rc=${PIPESTATUS[0]}
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "rad-convert rc=$rc; log: $LOG" >&2
|
||||
exit "$rc"
|
||||
fi
|
||||
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
|
||||
echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
|
||||
echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"
|
||||
Executable
+221
@@ -0,0 +1,221 @@
|
||||
#!/usr/bin/env bash
|
||||
# convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
|
||||
# campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
|
||||
# `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
|
||||
#
|
||||
# Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
|
||||
# converter runs on the host backend and the fleet can keep serving through the whole script.
|
||||
# The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
|
||||
# boundary between measurement and serving.
|
||||
#
|
||||
# Three arms, because the campaign's question is which search knob pays for its hours
|
||||
# (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
|
||||
#
|
||||
# cd3 the recipe as written — coordinate descent three sweeps. The control.
|
||||
# cd6 six sweeps: the same search, twice the passes.
|
||||
# ao1 act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
|
||||
# stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
|
||||
# cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
|
||||
# act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
|
||||
# the two knobs cannot be combined into a best-of-both container.
|
||||
#
|
||||
# The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
|
||||
# recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
|
||||
# a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
|
||||
# expert line and the other 60 lines cannot drift between arms.
|
||||
#
|
||||
# Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
|
||||
# (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
|
||||
# a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
|
||||
#
|
||||
# Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
|
||||
# inputs checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
|
||||
# directory converts UNCALIBRATED with one stderr note, which is silently worse
|
||||
# than failing here.
|
||||
# reference the 335.3 GiB BF16 reference container must be GONE before the converts start.
|
||||
# It and the checkpoint together eat the volume; its job (the KLD recording and
|
||||
# the calibration grams) is finished by then, and both survive on disk without the
|
||||
# container beside it. The FP8-native anchor is FETCHED after the converts (disk
|
||||
# sequencing), so nothing here waits on it.
|
||||
# plan --plan-only runs FIRST, per arm, and prints "about X would be written". The
|
||||
# number is the bill: free space must cover it plus the 128k-prompt overshoot
|
||||
# margin, and the finished artifact is checked against it.
|
||||
# outputs no arm overwrites anything; move the old one to trash.
|
||||
#
|
||||
# Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
|
||||
# [--target-count N] [--keep-recipe]
|
||||
set -u -o pipefail
|
||||
|
||||
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed}
|
||||
CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
|
||||
CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
|
||||
RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
|
||||
CALIB=${CALIB:-$CAMPAIGN/calib}
|
||||
IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
|
||||
OUTDIR=${OUTDIR:-/home/dank/models/rad}
|
||||
BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad} # the reference container, not an arm output
|
||||
STATE=${STATE:-$CAMPAIGN} # the dir the container mounts read-only: the
|
||||
# spliced recipe and the imatrix live here
|
||||
TARGET_COUNT=${TARGET_COUNT:-48}
|
||||
MIN_GRAMS=${MIN_GRAMS:-24576} # per-expert DOWN grams: 48 recorded layers x 512
|
||||
# experts. The checkpoint has 49 MoE layers, but the
|
||||
# MTP layer is never recorded, so its 512 experts are
|
||||
# absent from the tap's output by design
|
||||
MARGIN_MIB=${MARGIN_MIB:-40960} # the long-prompt overshoot, off the plan number
|
||||
TS=$(date +%Y%m%d-%H%M%S)
|
||||
|
||||
ARMS=()
|
||||
DRY=0
|
||||
PLAN_ONLY=0
|
||||
KEEP_RECIPE=0
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--arm) ARMS+=("$2"); shift 2 ;;
|
||||
--dry-run) DRY=1; shift ;;
|
||||
--plan-only) PLAN_ONLY=1; shift ;;
|
||||
--target-count) TARGET_COUNT="$2"; shift 2 ;;
|
||||
--keep-recipe) KEEP_RECIPE=1; shift ;;
|
||||
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
|
||||
for a in "${ARMS[@]}"; do
|
||||
case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
|
||||
done
|
||||
|
||||
# The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
|
||||
# every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
|
||||
# container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
|
||||
# it for empty).
|
||||
expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
|
||||
arm_quant() {
|
||||
case "$1" in
|
||||
# The control is the template itself: no override, so the recipe's own line decides.
|
||||
cd3) return 0 ;;
|
||||
cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
|
||||
# act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
|
||||
# columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
|
||||
# this arm is the interleave knob alone, not cd=3 plus it.
|
||||
ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# --- gates ------------------------------------------------------------------------------------
|
||||
[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
|
||||
[ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
|
||||
[ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
|
||||
[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
|
||||
GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
|
||||
[ "$GRAMS" -ge "$MIN_GRAMS" ] || {
|
||||
echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
|
||||
[ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
|
||||
# The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
|
||||
# no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
|
||||
# converter fail halfway through a planned arm.
|
||||
[ "$(dirname "$IMATRIX")" = "$STATE" ] || {
|
||||
echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
|
||||
if [ -e "$BF16_RAD" ]; then
|
||||
echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
|
||||
"the calibration grams are already taken, so move it to trash — the converts need its" \
|
||||
"space (peak-disk law)" >&2
|
||||
exit 2
|
||||
fi
|
||||
for a in "${ARMS[@]}"; do
|
||||
[ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
|
||||
echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
|
||||
done
|
||||
|
||||
# --- the protected plane, spliced into a per-run copy of the template --------------------------
|
||||
mkdir -p "$STATE"
|
||||
WORK="$STATE/recipe-$TS.recipe"
|
||||
cp "$RECIPE" "$WORK"
|
||||
if [ "$DRY" = 1 ]; then
|
||||
echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
|
||||
else
|
||||
# Positional, because the tool reads its recipe and calib dir by position: passing them as
|
||||
# environment variables would be silently ignored and it would rewrite the COMMITTED
|
||||
# template in place instead of this run's copy.
|
||||
"$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
|
||||
|| { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
|
||||
fi
|
||||
|
||||
run_convert() { # run_convert ARM [extra flags...]
|
||||
local arm="$1"; shift
|
||||
local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
|
||||
local argv=(--rm --name "qwen38-convert-$arm"
|
||||
-e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
|
||||
-v "$(dirname "$CKPT"):/models:ro"
|
||||
-v "$CALIB:/data/calib:ro"
|
||||
-v "$STATE:/campaign:ro"
|
||||
-v "$OUTDIR:/out")
|
||||
local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
|
||||
--tokenizer "/models/${CKPT##*/}/tokenizer.json"
|
||||
--imatrix /campaign/"${IMATRIX##*/}"
|
||||
# No --max-ctx: the context declare then runs at the checkpoint's training context
|
||||
# (262144), which is what the shipped container did — rad-info --meta on it carries
|
||||
# no override. A container declared at a smaller bound cannot be expected to serve
|
||||
# the 200k window D2 holds, and the convert is not where to discover that.
|
||||
# --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
|
||||
# checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
|
||||
# to by the convert, not by a boot on a drained window.
|
||||
--kv-cache-dtype fp8 -o "/out/$(basename "$out")")
|
||||
local q; q=$(arm_quant "$arm")
|
||||
[ -n "$q" ] && conv+=(--quant "$q")
|
||||
if [ "$DRY" = 1 ]; then
|
||||
echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
|
||||
return 0
|
||||
fi
|
||||
podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
|
||||
2>&1 | tee "$log"
|
||||
return "${PIPESTATUS[0]}"
|
||||
}
|
||||
|
||||
# --- plan, then bill, then convert -------------------------------------------------------------
|
||||
overall=0
|
||||
for arm in "${ARMS[@]}"; do
|
||||
echo "== arm $arm =="
|
||||
if [ "$DRY" = 1 ]; then
|
||||
# Both lines, in order: the plan runs before the convert, and the convert is what the
|
||||
# operator is about to pay an afternoon for.
|
||||
run_convert "$arm" --plan-only
|
||||
run_convert "$arm"
|
||||
continue
|
||||
fi
|
||||
rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
|
||||
[ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
|
||||
echo "$plan_log"
|
||||
# "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
|
||||
# hardcoding a size, so the gate still works after any recipe edit.
|
||||
want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
|
||||
[ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
|
||||
need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
|
||||
'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
|
||||
if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
|
||||
echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
|
||||
free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
|
||||
echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
|
||||
if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
|
||||
echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
|
||||
overall=1; continue
|
||||
fi
|
||||
if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
|
||||
rc=0; run_convert "$arm" || rc=$?
|
||||
[ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
|
||||
out="$OUTDIR/qwen38-w5nl64-$arm.rad"
|
||||
got_mib=$(( $(stat -c %s "$out") / 1048576 ))
|
||||
# The receipt: the artifact is what the plan predicted, within the metadata the plan does not
|
||||
# count. An arm that came out materially smaller lost a plane somewhere.
|
||||
lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
|
||||
if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
|
||||
echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
|
||||
overall=1; continue
|
||||
fi
|
||||
echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
|
||||
done
|
||||
|
||||
if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
|
||||
mkdir -p "$CAMPAIGN/.AGENTTRASH"
|
||||
mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
|
||||
fi
|
||||
exit "$overall"
|
||||
+16
-2
@@ -34,13 +34,16 @@ SERVICE_ORDER = (
|
||||
"hermes",
|
||||
"hermes-dashboard",
|
||||
"hermes-webui",
|
||||
"prometheus",
|
||||
"grafana",
|
||||
"loki",
|
||||
"alloy",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
"gpu-exporter",
|
||||
"alertmanager",
|
||||
# The prober must answer before prometheus re-runs the services job.
|
||||
"blackbox-exporter",
|
||||
"prometheus",
|
||||
"authentik-postgres",
|
||||
"authentik-server",
|
||||
"authentik-worker",
|
||||
@@ -94,7 +97,9 @@ DANKCLAW_SERVICES = {
|
||||
"alloy",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
"gpu-exporter",
|
||||
"alertmanager",
|
||||
"blackbox-exporter",
|
||||
"authentik-postgres",
|
||||
"authentik-server",
|
||||
"authentik-worker",
|
||||
@@ -127,6 +132,7 @@ DANKSERVER_SERVICES = {
|
||||
"frank-archon",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
"gpu-exporter",
|
||||
}
|
||||
|
||||
# FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
|
||||
@@ -274,6 +280,8 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# Provisioned datasources and dashboards: git is the source of
|
||||
# truth, so a change here must restart the service that reads them.
|
||||
_add(services, "grafana")
|
||||
elif group == "blackbox":
|
||||
_add(services, "blackbox-exporter")
|
||||
elif group == "homepage":
|
||||
# Homepage is a Dankclaw-owned deployment. Shared homepage files
|
||||
# are intentionally not copied during a Dankserver-only run.
|
||||
@@ -377,7 +385,7 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# consumer (prometheus before grafana, loki before alloy).
|
||||
if len(parts) >= 4 and parts[2] == "quadlet":
|
||||
name = parts[3].removesuffix(".container")
|
||||
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
|
||||
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
|
||||
_add(services, name)
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "gitea":
|
||||
@@ -452,6 +460,12 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# change to it redeploying alertmanager is the only path that
|
||||
# reaches it on the host.
|
||||
_add(services, "alertmanager")
|
||||
if name in {"gpu-exporter.py", "gpu-exporter.service"}:
|
||||
# The host-process GPU exporter is installed directly by the
|
||||
# gpu-exporter deploy on BOTH hosts, so its script and unit select
|
||||
# that service (target_services then keeps it only on the host being
|
||||
# planned). Editing the collector restarts the running exporter.
|
||||
_add(services, "gpu-exporter")
|
||||
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
|
||||
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-zealot only. The ik_llama.cpp
|
||||
|
||||
+112
-18
@@ -86,7 +86,7 @@ service_is_allowed() {
|
||||
dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
|
||||
dankclaw:comfyui|dankclaw:hermes|\
|
||||
dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
|
||||
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:alertmanager|\
|
||||
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
|
||||
dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
|
||||
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
|
||||
@@ -97,7 +97,7 @@ service_is_allowed() {
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
||||
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|\
|
||||
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||
frankbook-pro:readarr)
|
||||
@@ -252,16 +252,24 @@ install_signal_relay() {
|
||||
# The relay binds the host's LAN IP (the one address a bridge container
|
||||
# can reach as host.containers.internal), so keep the unauthenticated
|
||||
# endpoint off the rest of the LAN: admit loopback, the host itself
|
||||
# (its own health probe) and the podman bridge subnet only. Idempotent:
|
||||
# -C before -I.
|
||||
# (its own health probe) and the podman bridge subnet only.
|
||||
# ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
|
||||
# shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
|
||||
# ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
|
||||
# (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
|
||||
# dankclaw drain aborted at the alertmanager gate twice before the cause
|
||||
# was read off `nft list ruleset`). Rebuild the tail instead of inserting
|
||||
# it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
|
||||
# at the head (-C before -I, unchanged), then APPEND the DROP so it is
|
||||
# last and the ACCEPTs decide first. Converges on poisoned hosts.
|
||||
run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
|
||||
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
|
||||
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
|
||||
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
|
||||
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
|
||||
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
|
||||
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
|
||||
run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
|
||||
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
|
||||
run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
|
||||
# The interpreter needs a moment to bind after restart; a one-shot curl
|
||||
# raced the socket on 2026-09-17 and failed a healthy deploy.
|
||||
local i
|
||||
@@ -275,14 +283,81 @@ install_signal_relay() {
|
||||
return 1
|
||||
}
|
||||
|
||||
# Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
|
||||
# it runs python3 from the checkout on the host so it can read the DRM/cgroup
|
||||
# sysfs and shell out to nvidia-smi. Same two-address bind posture as
|
||||
# node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
|
||||
deploy_gpu_exporter() {
|
||||
local unit_dir="$HOME/.config/systemd/user" lan dropin i
|
||||
case "$TARGET" in
|
||||
dankclaw) lan=192.168.0.143 ;;
|
||||
dankserver) lan=192.168.0.235 ;;
|
||||
*) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
|
||||
esac
|
||||
mkdir -p "$unit_dir"
|
||||
cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
|
||||
dropin="$unit_dir/gpu-exporter.service.d"
|
||||
mkdir -p "$dropin"
|
||||
printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
|
||||
> "$dropin/bind.conf"
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable gpu-exporter.service
|
||||
systemctl --user restart gpu-exporter.service
|
||||
# The interpreter needs a moment to bind after restart (the same race that
|
||||
# once failed a healthy signal-relay deploy).
|
||||
for i in $(seq 1 15); do
|
||||
curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
|
||||
sleep 1
|
||||
done
|
||||
echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
copy_loki_config() {
|
||||
run_privileged mkdir -p /opt/loki/config
|
||||
run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
|
||||
}
|
||||
|
||||
copy_alloy_config() {
|
||||
run_privileged mkdir -p /opt/alloy/config
|
||||
run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
|
||||
# The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
|
||||
# carries the host's name, so Loki streams say WHICH machine's journal a
|
||||
# line came from without a second config file per host.
|
||||
# *_CONF_ROOT are test seams: the bats cases execute these copiers for
|
||||
# real against a sandbox with a stubbed run_privileged.
|
||||
local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
|
||||
tmp="$(mktemp)"
|
||||
sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
|
||||
if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
|
||||
echo "alloy config: host-label substitution failed" >&2
|
||||
rm -f "$tmp"
|
||||
return 1
|
||||
fi
|
||||
# Syntax gate: alloy's config language has no schema — a misplaced
|
||||
# attribute (e.g. `on_error` outside the json stage) parses only at RUN
|
||||
# time, and the unit then crash-loops with a healthy-looking deploy.
|
||||
# Validate against the very image that will run it, before installing.
|
||||
# Skipped only in the bats sandbox (no image there).
|
||||
if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
|
||||
if ! podman run --rm -v "$tmp:/c.alloy:ro" \
|
||||
--entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
|
||||
validate /c.alloy; then
|
||||
echo "alloy config: validate FAILED against the pinned image" >&2
|
||||
rm -f "$tmp"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
run_privileged mkdir -p "$root/config"
|
||||
run_privileged cp "$tmp" "$root/config/config.alloy"
|
||||
rm -f "$tmp"
|
||||
}
|
||||
|
||||
copy_blackbox_config() {
|
||||
# Single-file mount target: the unit binds this exact path into
|
||||
# /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
|
||||
# BLACKBOX_CONF_ROOT is the same test seam.
|
||||
local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
|
||||
run_privileged mkdir -p "$root"
|
||||
run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
|
||||
}
|
||||
|
||||
copy_grafana_provisioning() {
|
||||
@@ -291,14 +366,25 @@ copy_grafana_provisioning() {
|
||||
# datasource would be invisible to review, lost with the data volume, and
|
||||
# unknown to anyone reading the repository. The unit already mounts
|
||||
# /opt/grafana/provisioning read-only, so this only has to populate it.
|
||||
run_privileged mkdir -p /opt/grafana/provisioning/datasources \
|
||||
/opt/grafana/provisioning/dashboards
|
||||
# GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
|
||||
# copy runs unprivileged (the bats case executes this function for real).
|
||||
local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
|
||||
run_privileged mkdir -p "$root/provisioning/datasources" \
|
||||
"$root/provisioning/dashboards"
|
||||
run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
|
||||
/opt/grafana/provisioning/datasources/prometheus.yml
|
||||
"$root/provisioning/datasources/prometheus.yml"
|
||||
run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
|
||||
/opt/grafana/provisioning/dashboards/provider.yml
|
||||
run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
|
||||
/opt/grafana/provisioning/dashboards/llm-fleet.json
|
||||
"$root/provisioning/dashboards/provider.yml"
|
||||
# EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
|
||||
# turns each file in the directory into a dashboard, so a copier that names
|
||||
# files one by one silently drops new boards (measured 2026-10-21: the
|
||||
# containers board existed in the repo since its commit and was NEVER
|
||||
# provisioned — the panel-census browser check could not find it while the
|
||||
# hand-placed older boards happened to be on disk). Same rule as the quadlet
|
||||
# tier: the directory is the contract, the copy is generated state.
|
||||
for dash in shared/grafana/provisioning/dashboards/*.json; do
|
||||
run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
|
||||
done
|
||||
}
|
||||
|
||||
render_caddyfile() {
|
||||
@@ -522,6 +608,9 @@ deploy_observability() {
|
||||
;;
|
||||
cadvisor)
|
||||
;;
|
||||
blackbox-exporter)
|
||||
copy_blackbox_config
|
||||
;;
|
||||
*)
|
||||
echo "unknown observability service: $service" >&2
|
||||
return 2
|
||||
@@ -536,6 +625,9 @@ deploy_observability() {
|
||||
loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
|
||||
node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
|
||||
cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
|
||||
# Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
|
||||
# is not bindable from the host netns — see the unit header).
|
||||
blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
@@ -1371,9 +1463,10 @@ deploy_anubis() {
|
||||
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
|
||||
# qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
|
||||
# + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
|
||||
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
|
||||
# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
|
||||
# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
|
||||
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
|
||||
# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
|
||||
# run a 200,000 window with its ~45k fit gap; the pool still carries two
|
||||
# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
|
||||
# ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
|
||||
# relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
|
||||
# file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
|
||||
@@ -2003,7 +2096,8 @@ deploy_service() {
|
||||
plex) deploy_plex ;;
|
||||
tautulli) deploy_tautulli ;;
|
||||
alloy) deploy_alloy ;;
|
||||
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
|
||||
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
|
||||
gpu-exporter) deploy_gpu_exporter ;;
|
||||
comfyui) deploy_comfyui ;;
|
||||
*) deploy_compose_service "$service" ;;
|
||||
esac
|
||||
|
||||
+6
-4
@@ -133,13 +133,15 @@ fi
|
||||
valid_service() {
|
||||
local service="$1"
|
||||
case "$TARGET:$service" in
|
||||
# dankclaw:searxng-network is client-only: a network quadlet with no
|
||||
# remote dispatch handler, so the remote gate does not (must not) list it.
|
||||
dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
|
||||
dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
|
||||
dankclaw:gitea-server|dankclaw:homepage|\
|
||||
dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
|
||||
dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
|
||||
dankclaw:alertmanager|\
|
||||
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:alertmanager|dankclaw:blackbox-exporter|\
|
||||
dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
|
||||
dankclaw:open-notebook|\
|
||||
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
|
||||
@@ -147,8 +149,8 @@ valid_service() {
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
|
||||
dankserver:frank-archon|\
|
||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
||||
dankserver:prowlarr|dankserver:frank-archon|\
|
||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||
frankbook-pro:readarr)
|
||||
|
||||
@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
|
||||
Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
|
||||
|
||||
Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
|
||||
[--min-route-mult 3.0]
|
||||
[--min-route-mult 3.0] [--target-count 0]
|
||||
--target-count N fills the byte bill: the bars pick the base set exactly as
|
||||
the bar-only pass does, then the energy ranking admits experts one at a time
|
||||
(each with its parity partner) until the count lands. The energy bar NEVER
|
||||
relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
|
||||
walks down only when the ranking cannot produce that many candidates.
|
||||
0 takes the bars as given.
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
@@ -32,6 +39,7 @@ import sys
|
||||
import numpy as np
|
||||
|
||||
NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
|
||||
CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
|
||||
|
||||
|
||||
def read_gram(path):
|
||||
@@ -49,27 +57,8 @@ def read_gram(path):
|
||||
return k, rank, int(rows), tr
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("calib_dir")
|
||||
ap.add_argument("--min-share", type=float, default=0.03)
|
||||
ap.add_argument("--min-route-mult", type=float, default=3.0)
|
||||
a = ap.parse_args()
|
||||
|
||||
# layer -> global expert -> (rank, rows, energy)
|
||||
data = {}
|
||||
for path in sorted(glob.glob(os.path.join(
|
||||
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
|
||||
m = NAME_RE.match(os.path.basename(path))
|
||||
if not m:
|
||||
continue
|
||||
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
|
||||
k, _, rows, tr = read_gram(path)
|
||||
data.setdefault(layer, {})[expert] = (rank, rows, tr)
|
||||
|
||||
if not data:
|
||||
raise SystemExit("no per-expert down grams found")
|
||||
|
||||
def select(data, min_share, min_route_mult):
|
||||
"""(cast lines, report lines, count) for one pair of bars."""
|
||||
lines, report = [], []
|
||||
total = 0
|
||||
for layer in sorted(data):
|
||||
@@ -83,7 +72,7 @@ def main():
|
||||
for e, (rank, rows, tr) in experts.items():
|
||||
es, rs = tr / tot_e, rows / max(tot_r, 1)
|
||||
rec = (e, rank, es, rs)
|
||||
if es >= a.min_share and es >= a.min_route_mult * rs:
|
||||
if es >= min_share and es >= min_route_mult * rs:
|
||||
qual.append(rec)
|
||||
elif es > 0:
|
||||
pool.append(rec)
|
||||
@@ -107,10 +96,148 @@ def main():
|
||||
f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
|
||||
f"ratio {es/max(rs,1e-12):6.1f}")
|
||||
total += len(chosen)
|
||||
return lines, report, total
|
||||
|
||||
|
||||
def parity_repair(chosen, by_half, report, only=None):
|
||||
"""Add the next-best candidate to every half — or to `only` — holding an odd count.
|
||||
|
||||
The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
|
||||
of bf16 experts on a half is not a style problem: the two tables would not match.
|
||||
`by_half` is energy-descending, so the first unchosen candidate is the next most
|
||||
energetic, which is the expert worth keeping. Returns how many were added.
|
||||
"""
|
||||
added = 0
|
||||
for key in ([only] if only is not None else sorted(by_half)):
|
||||
layer, half = key
|
||||
n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
|
||||
if n % 2 == 0:
|
||||
continue
|
||||
for e, es, rs in by_half.get(key, []):
|
||||
if (layer, e) not in chosen:
|
||||
chosen[(layer, e)] = (es, rs)
|
||||
added += 1
|
||||
report.append(f"blk.{layer}: half {half} odd -> added e{e} "
|
||||
f"(energy {es*100:.2f}%)")
|
||||
break
|
||||
else:
|
||||
report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
|
||||
f"left to make it even")
|
||||
return added
|
||||
|
||||
|
||||
def select_to_count(data, min_share, min_route_mult, target):
|
||||
"""(lines, report, total, cutoff) for the set that LANDS the count.
|
||||
|
||||
The bars decide the base set; the count is then filled from the energy ranking,
|
||||
one expert at a time, each addition taking its half's next-best candidate with it
|
||||
so every rank's half stays even. Only the RATIO bar walks down to find candidates
|
||||
— it is the law that says an expert is disproportionate and not merely busy, and
|
||||
the caller's byte bill has to be filled — while the ENERGY bar holds, because that
|
||||
bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
|
||||
the set bottoms out at: the bar this campaign actually converted with.
|
||||
"""
|
||||
report = []
|
||||
shares, ranked = {}, []
|
||||
for layer in sorted(data):
|
||||
experts = data[layer]
|
||||
tot_e = sum(v[2] for v in experts.values())
|
||||
tot_r = sum(v[1] for v in experts.values())
|
||||
if tot_e <= 0 or tot_r <= 0:
|
||||
report.append(f"blk.{layer}: empty recording, skipped")
|
||||
continue
|
||||
for e, (_rank, rows, tr) in experts.items():
|
||||
es, rs = tr / tot_e, rows / max(tot_r, 1)
|
||||
if es <= 0:
|
||||
continue
|
||||
shares[(layer, e)] = (es, rs)
|
||||
ranked.append((es, layer, e, rs))
|
||||
ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
|
||||
by_half = {}
|
||||
for es, layer, e, rs in ranked:
|
||||
by_half.setdefault((layer, e // 256), []).append((e, es, rs))
|
||||
|
||||
chosen = {k: v for k, v in shares.items()
|
||||
if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
|
||||
total = len(chosen)
|
||||
total += parity_repair(chosen, by_half, report)
|
||||
|
||||
mult = min_route_mult
|
||||
while True:
|
||||
pool = [(es, l, e, rs) for es, l, e, rs in ranked
|
||||
if es >= mult * rs and (l, e) not in chosen]
|
||||
if len(pool) + total >= target or mult <= 0.0:
|
||||
break
|
||||
mult -= 0.5
|
||||
report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
|
||||
f"for {target - total} slots; relaxing toward 0")
|
||||
for es, layer, e, rs in pool:
|
||||
if total >= target:
|
||||
break
|
||||
if (layer, e) in chosen:
|
||||
continue
|
||||
chosen[(layer, e)] = (es, rs)
|
||||
total += 1
|
||||
report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
|
||||
total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
|
||||
lines, cutoff = [], None
|
||||
for (layer, e) in sorted(chosen):
|
||||
es, rs = chosen[(layer, e)]
|
||||
cutoff = es if cutoff is None else min(cutoff, es)
|
||||
lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight cast dtype=bf16")
|
||||
return lines, report, total, cutoff
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("calib_dir")
|
||||
ap.add_argument("--min-share", type=float, default=0.03)
|
||||
ap.add_argument("--min-route-mult", type=float, default=3.0)
|
||||
ap.add_argument("--target-count", type=int, default=0)
|
||||
a = ap.parse_args()
|
||||
|
||||
# layer -> global expert -> (rank, rows, energy)
|
||||
data = {}
|
||||
for path in sorted(glob.glob(os.path.join(
|
||||
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
|
||||
m = NAME_RE.match(os.path.basename(path))
|
||||
if not m:
|
||||
continue
|
||||
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
|
||||
k, _, rows, tr = read_gram(path)
|
||||
data.setdefault(layer, {})[expert] = (rank, rows, tr)
|
||||
|
||||
if not data:
|
||||
raise SystemExit("no per-expert down grams found")
|
||||
|
||||
if a.target_count > 0:
|
||||
lines, report, total, cutoff = select_to_count(
|
||||
data, a.min_share, a.min_route_mult, a.target_count)
|
||||
bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
|
||||
else:
|
||||
lines, report, total = select(data, a.min_share, a.min_route_mult)
|
||||
bar = f"min-share {a.min_share:.3g}"
|
||||
|
||||
# The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
|
||||
# every layer must hold an even number of bf16 experts. Judged on the output and
|
||||
# not on the internal sets, so a slip in either pass is caught here rather than at
|
||||
# engine load time.
|
||||
halves = collections.Counter()
|
||||
for rule in lines:
|
||||
m = CAST_RE.match(rule)
|
||||
if m:
|
||||
halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
|
||||
odd = {key: n for key, n in halves.items() if n % 2}
|
||||
if odd:
|
||||
print("PARITY LAW BROKEN: " + ", ".join(
|
||||
f"blk.{lay} half {half} holds {n} protected experts"
|
||||
for (lay, half), n in sorted(odd.items())), file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
print("\n".join(lines))
|
||||
print("\n".join(report), file=sys.stderr)
|
||||
print(f"total protected: {total}", file=sys.stderr)
|
||||
print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
|
||||
f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Executable
+249
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
|
||||
# BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
|
||||
# converts, and the source the KLD reference is recorded from.
|
||||
#
|
||||
# Why the base and not a finetune: the campaign converts to compare against the
|
||||
# publisher's own iq4r container, and that container was built from this
|
||||
# checkpoint. Base-in, base-out; anything else confounds the gate. (The
|
||||
# predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
|
||||
# set is identical to this repo's but whose weights are not — none of its
|
||||
# calibration grams, KLD reference or protected list carries over.)
|
||||
#
|
||||
# Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
|
||||
# in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
|
||||
# 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
|
||||
# verified the key set against this repo, so no gate letter is needed).
|
||||
#
|
||||
# Verification is four gates deep, because the failure this repo actually lived
|
||||
# through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
|
||||
# HTTP 401/403 and writes the error body as the artifact — three shards of
|
||||
# identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
|
||||
#
|
||||
# 1. shard census: the file count must equal the expected shard count, so a
|
||||
# partially fetched fragment tree (the .partNofM trap) cannot pass.
|
||||
# 2. per-shard completeness, from the shard's OWN safetensors header: the
|
||||
# header declares where the tensor data ends, so every shard's true size is
|
||||
# known offline. A torn tail is named here, by file, instead of surfacing
|
||||
# as a load error five hours later.
|
||||
# 3. index coverage: every shard model.safetensors.index.json references
|
||||
# exists and is named.
|
||||
# 4. total bytes against the number the Hub declared, then, unless --no-hub,
|
||||
# every file's sha256 against the Hub's LFS record — including the shards
|
||||
# already on disk, so a previous partial download is never trusted.
|
||||
#
|
||||
# The token never enters a command line (the 2026-09-16 leak law): it is read
|
||||
# into HF_TOKEN in the environment, where a local process listing is the only
|
||||
# reader. This repo is ungated, so the fetch simply runs without one when no
|
||||
# token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
|
||||
# hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
|
||||
#
|
||||
# usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
|
||||
# [--shards N] [--bytes B] [--repo ID]
|
||||
# --fetch download first (resumable: re-run to continue), then verify
|
||||
# --no-hub skip the Hub-side sha256 sweep; the other gates always run
|
||||
# exit: 0 verified, 1 verification failed, 2 environment / usage error
|
||||
set -euo pipefail
|
||||
|
||||
REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
|
||||
LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
|
||||
# Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
|
||||
# 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
|
||||
# shards only, so a config file edited on the Hub cannot fail it.
|
||||
EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
|
||||
EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
|
||||
MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
|
||||
|
||||
FETCH=0
|
||||
HUB=1
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--fetch) FETCH=1 ;;
|
||||
--no-hub) HUB=0 ;;
|
||||
--dir) LOCAL_DIR="$2"; shift ;;
|
||||
--shards) EXPECTED_SHARDS="$2"; shift ;;
|
||||
--bytes) EXPECTED_BYTES="$2"; shift ;;
|
||||
--repo) REPO="$2"; shift ;;
|
||||
*) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
FAILS=0
|
||||
note() { echo "$@"; FAILS=$((FAILS + 1)); }
|
||||
|
||||
if [ "$FETCH" = "1" ]; then
|
||||
# Every fetch gate is a hard stop: skipping the download silently and then
|
||||
# verifying a half tree is how a campaign ships a stale payload.
|
||||
HFGET="$SELF_DIR/hfget.sh"
|
||||
if [ ! -f "$HFGET" ]; then
|
||||
echo "REFUSED: the fetch helper is not here: $HFGET" >&2
|
||||
exit 2
|
||||
fi
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
echo "REFUSED: hfget.sh needs jq on PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
mkdir -p "$LOCAL_DIR"
|
||||
# The campaign's peak-disk law starts at the fetch: the checkpoint and the
|
||||
# reference container built from it are ~335 GiB each, and a tree that does
|
||||
# not fit is discovered hours into a transfer, when it costs the most.
|
||||
free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
|
||||
if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
|
||||
echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
|
||||
echo " (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
|
||||
echo " nothing is left half-written)" >&2
|
||||
exit 2
|
||||
fi
|
||||
echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
|
||||
if [ -z "${HF_TOKEN:-}" ]; then
|
||||
CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
|
||||
if [ -f "$CREDSTORE" ]; then
|
||||
HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
|
||||
elif [ -f "$HOME/.cache/huggingface/token" ]; then
|
||||
# The legacy single-line token file the hub library writes.
|
||||
HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
|
||||
elif [ -f "$HOME/.hf_token" ]; then
|
||||
HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
|
||||
fi
|
||||
fi
|
||||
if [ -n "${HF_TOKEN:-}" ]; then
|
||||
export HF_TOKEN
|
||||
fi
|
||||
bash "$HFGET" "$REPO" "$LOCAL_DIR" main
|
||||
unset HF_TOKEN
|
||||
fi
|
||||
|
||||
if [ ! -d "$LOCAL_DIR" ]; then
|
||||
echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
|
||||
exit 1
|
||||
fi
|
||||
INDEX="$LOCAL_DIR/model.safetensors.index.json"
|
||||
if [ ! -f "$INDEX" ]; then
|
||||
echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "== gate 1: shard census ($LOCAL_DIR)"
|
||||
SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
|
||||
if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
|
||||
note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
|
||||
else
|
||||
echo " ok: $SHARDS shards"
|
||||
fi
|
||||
|
||||
echo "== gate 2: every shard is whole, judged by its own header"
|
||||
# A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
|
||||
# declares where the data ends, so the file's true size is computable offline:
|
||||
# 8 + header + the largest end offset. Truncation and error bodies both fall
|
||||
# here, and each is reported with the file that caused it.
|
||||
while IFS= read -r f; do
|
||||
state=$(python3 - "$f" <<'PY'
|
||||
import json, os, struct, sys
|
||||
path = sys.argv[1]
|
||||
size = os.path.getsize(path)
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
(hlen,) = struct.unpack("<Q", fh.read(8))
|
||||
if hlen == 0 or hlen > size:
|
||||
print("NOT_SAFETENSORS")
|
||||
raise SystemExit
|
||||
header = json.loads(fh.read(hlen))
|
||||
except Exception:
|
||||
print("NOT_SAFETENSORS")
|
||||
raise SystemExit
|
||||
if not header:
|
||||
print("NOT_SAFETENSORS")
|
||||
raise SystemExit
|
||||
end = max((v["data_offsets"][1] for v in header.values()
|
||||
if isinstance(v, dict) and "data_offsets" in v), default=0)
|
||||
want = 8 + hlen + end
|
||||
print("OK" if want == size else f"TRUNCATED want={want} have={size}")
|
||||
PY
|
||||
)
|
||||
case "$state" in
|
||||
OK) ;;
|
||||
NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
|
||||
TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
|
||||
esac
|
||||
done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
|
||||
if [ "$FAILS" -eq 0 ]; then
|
||||
echo " ok: every shard's data extent matches its size on disk"
|
||||
fi
|
||||
|
||||
echo "== gate 3: the index's shards are all present"
|
||||
INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
|
||||
import json, sys
|
||||
map_ = json.load(open(sys.argv[1]))["weight_map"]
|
||||
print("\n".join(sorted(set(map_.values()))))
|
||||
PY
|
||||
)
|
||||
while IFS= read -r name; do
|
||||
[ -n "$name" ] || continue
|
||||
if [ ! -f "$LOCAL_DIR/$name" ]; then
|
||||
note "FAIL MISSING: $name is named by the index and is not on disk"
|
||||
fi
|
||||
done <<< "$INDEX_SHARDS"
|
||||
if [ "$FAILS" -eq 0 ]; then
|
||||
echo " ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
|
||||
fi
|
||||
|
||||
echo "== gate 4: total bytes vs the Hub's declaration"
|
||||
TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
|
||||
if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
|
||||
note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
|
||||
else
|
||||
echo " ok: $TOTAL bytes"
|
||||
fi
|
||||
|
||||
if [ "$HUB" = "1" ]; then
|
||||
echo "== gate 5: every file's sha256 vs the Hub's LFS record"
|
||||
# size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
|
||||
TMPD="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMPD"' EXIT
|
||||
MANIFEST="$TMPD/tree.json"
|
||||
PLAN="$TMPD/plan"
|
||||
curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
|
||||
-o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
|
||||
python3 - "$MANIFEST" > "$PLAN" <<'PY'
|
||||
import json, sys
|
||||
for e in json.load(open(sys.argv[1])):
|
||||
if e.get("type") != "file":
|
||||
continue
|
||||
sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
|
||||
if sha:
|
||||
print(e["path"], e["size"], sha)
|
||||
PY
|
||||
BAD=0
|
||||
while read -r rel size sha; do
|
||||
[ -n "${rel:-}" ] || continue
|
||||
if [ ! -f "$LOCAL_DIR/$rel" ]; then
|
||||
note "FAIL MISSING: $rel is on the Hub and not on disk"
|
||||
continue
|
||||
fi
|
||||
have=$(stat -c %s "$LOCAL_DIR/$rel")
|
||||
if [ "$have" != "$size" ]; then
|
||||
note "FAIL SIZE: $rel is $have bytes, Hub says $size"
|
||||
continue
|
||||
fi
|
||||
got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
|
||||
if [ "$got" != "$sha" ]; then
|
||||
note "FAIL SHA256: $rel got $got want $sha"
|
||||
BAD=$((BAD + 1))
|
||||
fi
|
||||
done < "$PLAN"
|
||||
if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
|
||||
echo " ok: sha256 matches for every file"
|
||||
fi
|
||||
rm -rf "$TMPD"
|
||||
trap - EXIT
|
||||
fi
|
||||
|
||||
if [ "$FAILS" -gt 0 ]; then
|
||||
echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
|
||||
echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"
|
||||
+49
-31
@@ -46,31 +46,35 @@ FRANK_ARCHON_TP=2
|
||||
# stays EXACT for single-sequence decode and any step of <= 25 rows — see
|
||||
# MAX_SEQS x (1 + SPEC_TOKENS) below.
|
||||
FRANK_ARCHON_TP_WIRE=wht6
|
||||
# Decode slots. A slot costs KV, never decode; the pool is sized to this
|
||||
# contract and nothing more. At 1.8.1 the budget resolver caps the cache at
|
||||
# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height
|
||||
# conversation answers 400 while /health and every shorter chat work fine
|
||||
# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two
|
||||
# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane,
|
||||
# instead of advertising a third slot that only exists at shorter prompts.
|
||||
FRANK_ARCHON_MAX_SEQS=2
|
||||
# Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
|
||||
# exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
|
||||
# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
|
||||
# least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
|
||||
# target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
|
||||
# the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
|
||||
# keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
|
||||
# 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
|
||||
# fourth conversations are backed by the cache instead of over-committing it.
|
||||
FRANK_ARCHON_MAX_SEQS=4
|
||||
# Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
|
||||
# limit, and models.yml contextWindow MUST quote this number (over-length is a
|
||||
# refusal, not a truncation). The pool is sized for seqs x maxlen, not one
|
||||
# sequence.
|
||||
FRANK_ARCHON_MAX_MODEL_LEN=200000
|
||||
FRANK_ARCHON_PLACEMENT=expert_tiered
|
||||
# Pinned host pool for expert rows that do not fit VRAM. Under the budget
|
||||
# resolver this is ALSO the lever that decides which experts stay on the card:
|
||||
# the card's leftover goes to the expert plane first and the KV cache takes the
|
||||
# rest, so growing the pool moves experts off the card and shrinking it spills
|
||||
# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
|
||||
# refused a boot). The ledger's coverage line is the referee, not this file.
|
||||
# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left
|
||||
# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra
|
||||
# host RAM above what the expert plane's footprint needs buys no cache. Back at
|
||||
# 44,928, the size that boots clean.
|
||||
FRANK_ARCHON_HOST_POOL_MIB=44928
|
||||
# Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
|
||||
# host must pin tp x this number, and the refusal that names it is
|
||||
# core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
|
||||
# lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
|
||||
# MiB of experts off the card and into the cache. Derived from the boot ledger (a
|
||||
# 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
|
||||
# drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
|
||||
# the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
|
||||
# twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
|
||||
# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
|
||||
# x8 link when the router picks it. The ledger's coverage line is the referee, not
|
||||
# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
|
||||
FRANK_ARCHON_HOST_POOL_MIB=49152
|
||||
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
|
||||
# floor is built from it.
|
||||
FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
||||
@@ -93,16 +97,29 @@ FRANK_ARCHON_SPEC_TOKENS=2
|
||||
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
||||
# Step sharing. A prefill chunk that shares a step with decoding sequences holds
|
||||
# the device for the chunk's whole time, and every decode row in that step waits
|
||||
# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
|
||||
# `auto` leaves each decode row behind the chunk a row plus its draft window and
|
||||
# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
|
||||
# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
|
||||
# chunk took the step and admission never ran. Interleave inserts N decode-only
|
||||
# steps after a chunk step taken beside decoders: it protects their cadence and
|
||||
# holds a waiting request out of those steps, so it stays 0 until the post-migration
|
||||
# decode-vs-prefill buckets say it is wanted.
|
||||
# that long for its next token: measured at this shape, decode-only steps 15.5 ms
|
||||
# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
|
||||
# while another session prefills (law and census: [[frank-archon]]).
|
||||
# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
|
||||
# draft window per decoding sequence (3 tokens) and an eighth of the budget per
|
||||
# waiting request or waiting prefill, and the source states it will not halve for
|
||||
# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
|
||||
# steps. A number is the only value that applies whenever the chunk shares a step,
|
||||
# and it must not go under the prefill stager's smallest pass -- 1025 rows here,
|
||||
# named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
|
||||
# because below it every routed layer reads its pooled experts synchronously, so
|
||||
# 512 is a trap and the vendor measures the whole job slower there. Interleave
|
||||
# inserts N decode-only steps after a chunk step taken beside decoders: it protects
|
||||
# their cadence and holds a waiting request out of those steps. That is the decoder
|
||||
# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
|
||||
# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
|
||||
# price falls on the prefilling prompt (its chunks lose every held step) and on any
|
||||
# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
|
||||
# ALONE, so it cannot buy this back while decoders run. Meter:
|
||||
# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
|
||||
# before N rises.
|
||||
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
|
||||
FRANK_ARCHON_PREFILL_INTERLEAVE=0
|
||||
FRANK_ARCHON_PREFILL_INTERLEAVE=1
|
||||
# Prefix cache: pinned host memory only. The disk tier was removed outright
|
||||
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
|
||||
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
|
||||
@@ -118,9 +135,10 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
||||
# RESETS pages_limit to half of RAM on every boot; below the plan the load dies
|
||||
# mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
|
||||
# itself before refusing by name, so recovery is a boot, not a visit.
|
||||
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
|
||||
# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB.
|
||||
FRANK_ARCHON_TTM_MIN_MIB=150000
|
||||
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
|
||||
# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
|
||||
# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
|
||||
FRANK_ARCHON_TTM_MIN_MIB=162000
|
||||
FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
|
||||
# this unquoted on purpose, so keep multi-flag values as one string.
|
||||
|
||||
+22
-12
@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
|
||||
# stays at the image default, which is what was qualified) ------------------
|
||||
# Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
|
||||
HALOGEN_MODEL_ID=frank-dragoon
|
||||
# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
|
||||
# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
|
||||
# the page) — a window equal to the ceiling makes every long request a 400.
|
||||
HALOGEN_CTX=200704
|
||||
# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
|
||||
# at once. Pool positions buy concurrency; the per-request window never moves
|
||||
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
|
||||
# RAM the pool takes is RAM the n-gram table's page cache loses.
|
||||
# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
|
||||
# longer prompt is a 400 naming the limit, never a truncation; stretching
|
||||
# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
|
||||
# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
|
||||
# page): with the window at 200,000 the gap is 62,144, which absorbs the
|
||||
# chars/4 under-count. A window at or near the ceiling makes every long
|
||||
# request a guaranteed 400.
|
||||
HALOGEN_CTX=262144
|
||||
# ONE pool behind both slots. The harness window is 200,000, so 401,408
|
||||
# still carries TWO maximum harness-length conversations at once
|
||||
# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
|
||||
# ceiling-length requests would queue instead — buying those means raising
|
||||
# the pool, and RAM the pool takes is RAM the n-gram table's page cache
|
||||
# loses. Pool positions buy concurrency; the per-request window never moves
|
||||
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
|
||||
HALOGEN_KV_POOL_POSITIONS=401408
|
||||
# Two concurrent conversations: the MTP drafter is not slot-locked on this
|
||||
# engine, so the second slot is capacity, not a speculation tax.
|
||||
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
|
||||
FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
|
||||
# Node paths and the XRT directory are variables so the bats seam can drive
|
||||
# these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
|
||||
# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
|
||||
# once at its own path, because those libraries find each other at their install
|
||||
# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
|
||||
# system dir, which dangle inside the container.
|
||||
# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
|
||||
# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
|
||||
# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
|
||||
# searches that third prefix; without it the lane dies naming it and the
|
||||
# supervisor takes the container down — 2026-10-21 boot loop). The basename of
|
||||
# this directory IS the multiarch name. The vendor forbids mounting an
|
||||
# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
|
||||
FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
|
||||
FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
|
||||
# sysfs root the fabric-clock preflight reads (same knob the vendor's
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
|
||||
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
Executable
+212
@@ -0,0 +1,212 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
|
||||
|
||||
Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
|
||||
|
||||
gpu_utilization_percent gpu_memory_used_bytes gpu_memory_total_bytes
|
||||
gpu_temperature_celsius gpu_power_watts
|
||||
|
||||
each labelled {gpu, vendor, model}. The scrape job adds `host`.
|
||||
|
||||
Two sources, both host-native — no container, no GPU passthrough:
|
||||
|
||||
* AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
|
||||
Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
|
||||
as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
|
||||
is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
|
||||
the R9700s, so those cards legitimately have no gpu_power_watts line; a
|
||||
missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
|
||||
would plot as a real reading.
|
||||
|
||||
* NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
|
||||
Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
|
||||
fails; that is tolerated and simply yields no NVIDIA series.
|
||||
|
||||
Default mode prints one scrape to stdout and exits (the test harness, and any
|
||||
`service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
|
||||
Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
|
||||
wildcard, matching node-exporter's posture on these boxes.
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
FAMILIES = [
|
||||
("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
|
||||
("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
|
||||
("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
|
||||
("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
|
||||
("gpu_power_watts", "gauge", "Board power draw in watts."),
|
||||
]
|
||||
|
||||
DEFAULT_NVIDIA_CMD = (
|
||||
"nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
|
||||
"memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
|
||||
)
|
||||
|
||||
|
||||
def _read(path):
|
||||
try:
|
||||
with open(path) as fh:
|
||||
return fh.read().strip()
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def _pci_slot(device_dir):
|
||||
uevent = _read(os.path.join(device_dir, "uevent"))
|
||||
for line in uevent.splitlines():
|
||||
if line.startswith("PCI_SLOT_NAME="):
|
||||
return line.split("=", 1)[1]
|
||||
return ""
|
||||
|
||||
|
||||
def _pci_id(device_dir):
|
||||
uevent = _read(os.path.join(device_dir, "uevent"))
|
||||
for line in uevent.splitlines():
|
||||
if line.startswith("PCI_ID="):
|
||||
return "amd:" + line.split("=", 1)[1].lower()
|
||||
return "amd"
|
||||
|
||||
|
||||
def collect_amd(sys_dir):
|
||||
for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
|
||||
device = os.path.join(card, "device")
|
||||
busy = _read(os.path.join(device, "gpu_busy_percent"))
|
||||
if busy == "":
|
||||
continue # not an amdgpu compute node (e.g. a render node w/o metrics)
|
||||
labels = {"gpu": _pci_slot(device) or os.path.basename(card),
|
||||
"vendor": "amd", "model": _pci_id(device)}
|
||||
yield "gpu_utilization_percent", labels, busy
|
||||
for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
|
||||
("gpu_memory_total_bytes", "mem_info_vram_total")):
|
||||
v = _read(os.path.join(device, leaf))
|
||||
if v != "":
|
||||
yield name, labels, v
|
||||
temp = ""
|
||||
for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
|
||||
temp = _read(hwmon)
|
||||
if temp != "":
|
||||
break
|
||||
if temp != "":
|
||||
yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
|
||||
power = _read(os.path.join(device, "power1_average"))
|
||||
if power != "": # empty on the R9700s -> metric absent, not zero
|
||||
yield "gpu_power_watts", labels, str(int(power) / 1e6)
|
||||
|
||||
|
||||
def collect_nvidia(cmd):
|
||||
if not cmd:
|
||||
return
|
||||
try:
|
||||
out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
|
||||
timeout=10, check=True).stdout
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return # no NVIDIA GPU / no driver: tolerate, emit nothing
|
||||
for line in out.splitlines():
|
||||
parts = [p.strip() for p in line.split(",")]
|
||||
if len(parts) < 7:
|
||||
continue
|
||||
idx, name, util, mused, mtotal, temp, power = parts[:7]
|
||||
labels = {"gpu": idx, "vendor": "nvidia", "model": name}
|
||||
yield "gpu_utilization_percent", labels, util
|
||||
yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
|
||||
yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
|
||||
if temp not in ("", "[N/A]"):
|
||||
yield "gpu_temperature_celsius", labels, temp
|
||||
if power not in ("", "[N/A]"):
|
||||
yield "gpu_power_watts", labels, power
|
||||
|
||||
|
||||
def collect(sys_dir, nvidia_cmd):
|
||||
samples = {}
|
||||
for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
|
||||
for name, labels, value in source:
|
||||
try:
|
||||
value = str(float(value)) # normalise; drop trailing junk
|
||||
except ValueError:
|
||||
continue
|
||||
samples.setdefault(name, []).append((labels, value))
|
||||
return samples
|
||||
|
||||
|
||||
def _fmt(value):
|
||||
f = float(value)
|
||||
if f == int(f) and abs(f) < 1e15:
|
||||
return str(int(f))
|
||||
return repr(f)
|
||||
|
||||
|
||||
def render(samples):
|
||||
out = []
|
||||
for name, kind, help_text in FAMILIES:
|
||||
out.append("# HELP %s %s" % (name, help_text))
|
||||
out.append("# TYPE %s %s" % (name, kind))
|
||||
for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
|
||||
lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
|
||||
out.append("%s{%s} %s" % (name, lset, _fmt(value)))
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
sys_dir = "/sys/class/drm"
|
||||
nvidia_cmd = DEFAULT_NVIDIA_CMD
|
||||
|
||||
def do_GET(self): # noqa: N802 (http.server API)
|
||||
if self.path.split("?")[0] not in ("/metrics", "/"):
|
||||
self.send_error(404)
|
||||
return
|
||||
body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/plain; version=0.0.4")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
|
||||
def main(argv):
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
|
||||
ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
|
||||
ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
|
||||
ap.add_argument("--port", type=int, default=9401)
|
||||
ap.add_argument("--bind", action="append", default=None,
|
||||
help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
|
||||
"comma-list if unset. NEVER a wildcard.")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if not args.serve:
|
||||
sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
|
||||
return 0
|
||||
|
||||
binds = args.bind
|
||||
if not binds:
|
||||
env = os.environ.get("GPU_EXPORTER_BINDS", "")
|
||||
binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
|
||||
Handler.sys_dir = args.amdgpu_sysfs
|
||||
Handler.nvidia_cmd = args.nvidia_cmd
|
||||
servers = []
|
||||
for b in binds:
|
||||
if b in ("0.0.0.0", "::", "*"):
|
||||
sys.stderr.write("refusing wildcard bind %s\n" % b)
|
||||
return 2
|
||||
servers.append(ThreadingHTTPServer((b, args.port), Handler))
|
||||
sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
|
||||
# EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
|
||||
# healthy in `ss` yet times out every scrape once its SYN backlog fills —
|
||||
# and the loopback deploy gate cannot see it. One thread per extra bind.
|
||||
for s in servers[1:]:
|
||||
threading.Thread(target=s.serve_forever, daemon=True).start()
|
||||
servers[0].serve_forever()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,35 @@
|
||||
# Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
|
||||
#
|
||||
# Install path: ~/.config/systemd/user/gpu-exporter.service
|
||||
# Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
|
||||
#
|
||||
# Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
|
||||
# AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
|
||||
# user unit, not a quadlet: it runs python3 straight from the checkout (a pull
|
||||
# updates it with no copy step), a hand-written user unit can be enabled where
|
||||
# the podman v1 generator's units cannot, and with Linger=yes for dank it starts
|
||||
# at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
|
||||
# it must run on the host — a container would need GPU/DRM passthrough for
|
||||
# nothing gained.
|
||||
#
|
||||
# BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
|
||||
# drop-in adding THIS host's LAN address (the same two-address posture
|
||||
# node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
|
||||
# unauthenticated metrics on the LAN, identical to node-exporter :9100 and
|
||||
# cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
|
||||
# the Dankserver instance over the LAN from Dankclaw.
|
||||
|
||||
[Unit]
|
||||
Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
# Loopback by default; the deploy drop-in appends the host LAN address.
|
||||
Environment=GPU_EXPORTER_BINDS=127.0.0.1
|
||||
ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
Executable
+418
@@ -0,0 +1,418 @@
|
||||
#!/usr/bin/env python3
|
||||
"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
|
||||
llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
|
||||
beside the recording. Python 3 stdlib only: numpy must NOT be required, because
|
||||
this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
|
||||
|
||||
WHY. `rad-convert` takes its per-column activation second moments and its
|
||||
expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
|
||||
Only the GGUF form carries a per-expert dimension — with the legacy flat form,
|
||||
or with none, every expert in a layer is weighted by the layer's aggregate,
|
||||
which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
|
||||
already measured everything; this tool only re-voices it, so the imatrix and the
|
||||
protected-expert census (derive-protected-experts.py) are the SAME recording:
|
||||
one measurement, two consumers, neither able to disagree about which experts are
|
||||
hot.
|
||||
|
||||
INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
|
||||
(arch/common/rad_arch.h, calib_write). Header law, same parse as
|
||||
scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
|
||||
u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
|
||||
kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
|
||||
numbering (rad_block_moe_fp8.h: ge = expert_base + e):
|
||||
|
||||
gram.r<rank>.blk.L.ffn_down_exps.bin the layer's pooled gram
|
||||
gram.r<rank>.blk.L.ffn_down_exps.<e>.bin expert e's own gram
|
||||
|
||||
`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
|
||||
rows, so **the diagonal is exactly the per-column activation second moment an
|
||||
imatrix wants**, and `rows` is the count the reader hands back from `count()`.
|
||||
Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
|
||||
put a full matrix, and the quantiser asks for columns.
|
||||
|
||||
OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
|
||||
layer's experts folded into ONE entry, so expert 9 is slice 9 of
|
||||
`blk.L.ffn_down_exps`):
|
||||
|
||||
blk.L.ffn_down_exps in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
|
||||
blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
|
||||
gate_up input is the same hidden-state rows as its down
|
||||
input, so the second moment over them is identical and
|
||||
so is the routing count. Two tensor infos over one
|
||||
payload: the reader maps each half independently
|
||||
(imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
|
||||
by base name), and copying ~200 MB to state one fact
|
||||
twice buys no correctness.
|
||||
blk.L.ssm_inz [k] and [1], from the pooled gram: that row's input is
|
||||
the same hidden state, so its columns take the pooled
|
||||
diagonal and its count is the layer's pooled rows.
|
||||
|
||||
Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
|
||||
attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
|
||||
entry's cols do not equal the weight's width and the convert proceeds without
|
||||
it: an entry the reader silently ignores is a lie in the artifact, worse than
|
||||
an absent one. The corollary to check before trusting the two hidden-width
|
||||
entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
|
||||
expert width, so the gram's k must equal the checkpoint's hidden width (its
|
||||
config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
|
||||
|
||||
INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
|
||||
its count stays its own routing rows — the rule the engine itself applies when it
|
||||
picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
|
||||
once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
|
||||
is worse than round-to-nearest, and those experts carry a fraction of a percent
|
||||
of the routing). The count never inherits: popularity is `count`'s job. The count
|
||||
of inherited experts is reported on stderr.
|
||||
|
||||
RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
|
||||
so the ranks describe DISJOINT experts and the fold is a UNION into the one
|
||||
[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
|
||||
same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
|
||||
calib_rows). The same global expert in two per-expert files is legal only when
|
||||
the numbers agree exactly; otherwise the recording is corrupt and the entry would
|
||||
be whichever file sorted last, so refuse.
|
||||
|
||||
GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
|
||||
`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
|
||||
`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
|
||||
relative to the aligned data section. Every tensor offset is padded up to
|
||||
`general.alignment` and the data section itself starts aligned. That is the
|
||||
classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
|
||||
arithmetic point into the middle of the previous tensor's floats, and it reads
|
||||
them without complaint.
|
||||
|
||||
chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
|
||||
row total, not a sum over layers: every layer's tap accumulated over the whole
|
||||
corpus, so adding the layers counts each token once per layer and inflates the
|
||||
figure by the layer count. chunk_count = that row total // chunk_size, the
|
||||
"chunks of chunk_size" line imatrix.cpp prints for itself.
|
||||
|
||||
REFUSALS, each exit 2 with a message naming what is missing:
|
||||
NO_GRAMS the directory holds no RADGRAM2 down-projection gram
|
||||
NOT_RADGRAM2 a gram whose magic is not RADGRAM2
|
||||
SHORT a gram whose length is not 32 + k*k*4
|
||||
NO_POOLED a layer with expert grams and no pooled gram to inherit from
|
||||
K_MISMATCH a layer whose files disagree about k (rank pooled grams, or an
|
||||
expert gram against its layer's pooled gram)
|
||||
EXPERT_CONFLICT a global expert recorded twice with different numbers
|
||||
EXISTS the -o path already holds a file (the trash law: move it)
|
||||
CHUNK_COUNT the derived chunk count does not fit the u32 metadata field
|
||||
|
||||
USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
|
||||
[--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
|
||||
MAGIC = b"RADGRAM2"
|
||||
HDR = 32 # 8s I I Q I + pad, rad_arch.h
|
||||
KOWN_ROWS_PER_K = 4 # libquant/lq_hess.h kOwnRowsPerK
|
||||
ALIGNMENT = 32 # general.alignment, and the pad unit
|
||||
|
||||
POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
|
||||
EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
|
||||
|
||||
GT_U32, GT_STR, GT_ARR = 4, 8, 9
|
||||
GGML_F32 = 0
|
||||
|
||||
|
||||
def refuse(msg):
|
||||
sys.stderr.write("REFUSED %s\n" % msg)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def read_gram(path):
|
||||
"""(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
raw = f.read()
|
||||
except OSError as e:
|
||||
refuse("UNREADABLE %s: %s" % (path, e))
|
||||
if len(raw) < HDR:
|
||||
refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
|
||||
% (path, len(raw), HDR))
|
||||
magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
|
||||
if magic != MAGIC:
|
||||
refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
|
||||
if k <= 0:
|
||||
refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
|
||||
need = HDR + k * k * 4
|
||||
if len(raw) != need:
|
||||
refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
|
||||
% (path, len(raw), need, k))
|
||||
# The diagonal only, read by stride out of the row-major matrix.
|
||||
diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
|
||||
for i in range(k)]
|
||||
return k, rank, rows, domain, diag
|
||||
|
||||
|
||||
def load_dir(d, verbose):
|
||||
"""{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
|
||||
|
||||
Files outside the ffn_down_exps stem are not ours to fold: the tap names a
|
||||
pooled gram after every projection it taps, and those carry other widths."""
|
||||
if not os.path.isdir(d):
|
||||
refuse("NO_GRAMS %s: not a directory" % d)
|
||||
layers, rotated = {}, 0
|
||||
for name in sorted(os.listdir(d)):
|
||||
m = POOLED_RE.match(name) or EXPERT_RE.match(name)
|
||||
if not m:
|
||||
continue
|
||||
path = os.path.join(d, name)
|
||||
if not os.path.isfile(path):
|
||||
continue
|
||||
pooled = bool(POOLED_RE.match(name))
|
||||
layer = int(m.group(2))
|
||||
k, rank, rows, domain, diag = read_gram(path)
|
||||
if domain != 0:
|
||||
rotated += 1
|
||||
L = layers.setdefault(layer, {"pooled": None, "experts": {}})
|
||||
if pooled:
|
||||
P = L["pooled"]
|
||||
if P is None:
|
||||
L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
|
||||
elif P["k"] != k:
|
||||
refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
|
||||
" disagree about the width"
|
||||
% (layer, P["paths"][0], P["k"], path, k))
|
||||
else:
|
||||
# calib_sum: each rank summed the same stem over its own rows.
|
||||
P["rows"] += rows
|
||||
P["paths"].append(path)
|
||||
for i in range(k):
|
||||
P["diag"][i] += diag[i]
|
||||
if verbose:
|
||||
sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
|
||||
% (layer, name, k, rows))
|
||||
continue
|
||||
e = int(m.group(3))
|
||||
own = L["experts"].get(e)
|
||||
if own is None:
|
||||
L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
|
||||
"path": path, "ranks": [rank]}
|
||||
if verbose:
|
||||
sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
|
||||
% (layer, rank, e, rows))
|
||||
continue
|
||||
if own["rows"] != rows or own["diag"] != diag:
|
||||
refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
|
||||
" (rows=%d) record the same global expert with different"
|
||||
" numbers" % (layer, e, own["path"], own["rows"], path, rows))
|
||||
own["ranks"].append(rank)
|
||||
sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
|
||||
" and %s; folded once\n"
|
||||
% (layer, e, own["path"], name))
|
||||
if not layers:
|
||||
refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
|
||||
"[.<expert>].bin files in the directory" % d)
|
||||
return layers, rotated
|
||||
|
||||
|
||||
def build_entries(layers, min_rows, verbose):
|
||||
"""([ (name, dims, payload|None, key) ], census, layers emitted)."""
|
||||
entries = []
|
||||
census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
|
||||
for layer in sorted(layers):
|
||||
P, E = layers[layer]["pooled"], layers[layer]["experts"]
|
||||
if E and P is None:
|
||||
refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
|
||||
" expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
|
||||
" under the row threshold would have nothing to inherit"
|
||||
% (layer, len(E), layer))
|
||||
if P is None:
|
||||
continue
|
||||
k = P["k"]
|
||||
census["layers"] += 1
|
||||
census["rows"] = max(census["rows"], P["rows"])
|
||||
dn = "blk.%d.ffn_down_exps" % layer
|
||||
# A staging key names ONE payload. A gate_up entry reuses the down
|
||||
# layer's bytes by carrying the same key with no payload of its own;
|
||||
# in_sum2 and counts are different bytes and must never share a key.
|
||||
ksum, kcnt = (layer, "sum2"), (layer, "counts")
|
||||
kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
|
||||
# One entry holds one width: an expert gram recorded at another k would
|
||||
# inherit (or be inherited by) a diagonal of the wrong length, and the
|
||||
# reader would then answer `cols` from whichever file set dims[0].
|
||||
for e in sorted(E):
|
||||
if E[e]["k"] != k:
|
||||
refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
|
||||
" layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
|
||||
E[e]["k"], k))
|
||||
|
||||
if not E:
|
||||
# No per-expert recording for this layer: emit the dense form
|
||||
# ([k], [1]) rather than 512 identical slices, which would claim a
|
||||
# per-expert measurement that was never taken.
|
||||
entries.append((dn + ".in_sum2", [k],
|
||||
struct.pack("<%df" % k, *P["diag"]), ksum))
|
||||
entries.append((dn + ".counts", [1],
|
||||
struct.pack("<f", float(P["rows"])), kcnt))
|
||||
sys.stderr.write("note: layer %d has no per-expert grams; %s is"
|
||||
" dense over the pooled gram\n" % (layer, dn))
|
||||
else:
|
||||
floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
|
||||
n_expert = max(E) + 1
|
||||
sum2, counts = bytearray(), bytearray()
|
||||
for e in range(n_expert):
|
||||
own = E.get(e)
|
||||
if own is not None and own["rows"] >= floor:
|
||||
src = own["diag"]
|
||||
rows = own["rows"]
|
||||
else:
|
||||
src = P["diag"]
|
||||
rows = own["rows"] if own is not None else 0
|
||||
census["inherited"] += 1
|
||||
sum2 += struct.pack("<%df" % k, *src)
|
||||
counts += struct.pack("<f", float(rows))
|
||||
census["experts"] += 1
|
||||
census["routing"] += rows
|
||||
if min(E) != 0:
|
||||
sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
|
||||
" those slices carry the pooled gram at count 0\n"
|
||||
% (layer, min(E) - 1))
|
||||
entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
|
||||
entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
|
||||
gu = "blk.%d.ffn_gate_up_exps" % layer
|
||||
entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
|
||||
entries.append((gu + ".counts", [1, n_expert], None, kcnt))
|
||||
if verbose:
|
||||
sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
|
||||
" own-gram floor=%d\n"
|
||||
% (layer, k, n_expert, P["rows"], floor))
|
||||
|
||||
inz = "blk.%d.ssm_inz" % layer
|
||||
entries.append((inz + ".in_sum2", [k],
|
||||
struct.pack("<%df" % k, *P["diag"]), kpgm))
|
||||
entries.append((inz + ".counts", [1],
|
||||
struct.pack("<f", float(P["rows"])), kpcnt))
|
||||
return entries, census
|
||||
|
||||
|
||||
def enc_str(v):
|
||||
b = v.encode("utf-8")
|
||||
return struct.pack("<Q", len(b)) + b
|
||||
|
||||
|
||||
def write_gguf(path, entries, datasets, chunk_size, chunk_count):
|
||||
"""Write GGUF v3 and return its size in bytes.
|
||||
|
||||
One payload per key: entries that share a key share the bytes and the
|
||||
offset, so a second tensor info states the same fact without copying it."""
|
||||
kv = bytearray()
|
||||
kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
|
||||
+ struct.pack("<I", ALIGNMENT)
|
||||
kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
|
||||
+ struct.pack("<I", chunk_size)
|
||||
kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
|
||||
+ struct.pack("<I", chunk_count)
|
||||
kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
|
||||
+ struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
|
||||
for d in datasets:
|
||||
kv += enc_str(d)
|
||||
|
||||
head = bytearray()
|
||||
head += b"GGUF" + struct.pack("<I", 3)
|
||||
head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
|
||||
head += kv
|
||||
|
||||
payloads = {}
|
||||
for name, _dims, payload, key in entries:
|
||||
if payload is None:
|
||||
continue
|
||||
prev = payloads.get(key)
|
||||
if prev is not None and prev != payload:
|
||||
# Two different payloads on one key would ship whichever was
|
||||
# appended last while BOTH tensor infos pointed at it: the counts
|
||||
# payload landing over the in_sum2 payload is exactly how a
|
||||
# hand-rolled GGUF writer loses half its data without an error.
|
||||
refuse("INTERNAL %s: two different payloads share one staging key"
|
||||
% name)
|
||||
payloads.setdefault(key, payload)
|
||||
infos = bytearray()
|
||||
offsets, order, size = {}, [], 0
|
||||
for name, dims, payload, key in entries:
|
||||
if key not in payloads:
|
||||
refuse("INTERNAL %s: no payload staged for its key" % name)
|
||||
if key not in offsets:
|
||||
offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
|
||||
size = offsets[key] + len(payloads[key])
|
||||
order.append(key)
|
||||
infos += enc_str(name) + struct.pack("<I", len(dims))
|
||||
for d in dims:
|
||||
infos += struct.pack("<Q", d)
|
||||
infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
|
||||
|
||||
body = bytes(head) + bytes(infos)
|
||||
data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
|
||||
with open(path, "wb") as f:
|
||||
f.write(body)
|
||||
f.write(b"\0" * (data_start - len(body)))
|
||||
# The offsets in the tensor infos are the file's law, not a hint: each
|
||||
# payload must land at its own padded offset. Writing them back to back
|
||||
# instead leaves every reader that trusts the info — imatrix.cpp does —
|
||||
# reading its neighbour's floats: plausible values, no error, wrong
|
||||
# quantisation.
|
||||
here = data_start
|
||||
for key in order:
|
||||
f.write(b"\0" * (data_start + offsets[key] - here))
|
||||
f.write(payloads[key])
|
||||
here = data_start + offsets[key] + len(payloads[key])
|
||||
return f.tell()
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
|
||||
ap.add_argument("calib_dir")
|
||||
ap.add_argument("-o", "--out", required=True)
|
||||
ap.add_argument("--datasets", default="",
|
||||
help="comma-separated names for imatrix.datasets")
|
||||
ap.add_argument("--chunk-size", type=int, default=4096,
|
||||
help="rows per chunk for imatrix.chunk_size (default 4096)")
|
||||
ap.add_argument("--min-rows", type=int, default=0,
|
||||
help="own-gram row threshold (default 4*k, the engine's"
|
||||
" kOwnRowsPerK law)")
|
||||
ap.add_argument("-v", "--verbose", action="store_true")
|
||||
a = ap.parse_args()
|
||||
|
||||
if os.path.exists(a.out):
|
||||
# The repo's deletion law: an artifact that exists is moved to trash by
|
||||
# a human, never silently overwritten by a re-run of the tool that made
|
||||
# it, so a second convert cannot destroy the imatrix the first one fed.
|
||||
refuse("EXISTS %s: an imatrix is already there; move it to"
|
||||
" .AGENTTRASH/ or name another -o" % a.out)
|
||||
if a.chunk_size <= 0:
|
||||
refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
|
||||
layers, rotated = load_dir(a.calib_dir, a.verbose)
|
||||
entries, census = build_entries(layers, a.min_rows, a.verbose)
|
||||
if not entries:
|
||||
refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
|
||||
if rotated:
|
||||
sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
|
||||
" diagonal is the rotated activation's second moment,"
|
||||
" not the model's own\n" % rotated)
|
||||
|
||||
chunk_count = census["rows"] // a.chunk_size
|
||||
if chunk_count > 0xFFFFFFFF:
|
||||
# imatrix.chunk_count is a u32 in the reader's grammar, so there is no
|
||||
# legal way to record a bigger one. Reaching this means either a chunk
|
||||
# size picked for a far larger run or a row total read out of the wrong
|
||||
# header offset — rows is a u64 and a u32 parse is off by 2**32 — and
|
||||
# the reader would trust whatever number landed in the field.
|
||||
refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
|
||||
" imatrix.chunk_count field; raise --chunk-size"
|
||||
% (chunk_count, census["rows"], a.chunk_size))
|
||||
nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
|
||||
a.chunk_size, chunk_count)
|
||||
sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
|
||||
" rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
|
||||
% (len(entries), census["layers"], census["experts"],
|
||||
census["inherited"], census["rows"], census["routing"],
|
||||
chunk_count, a.chunk_size, nbytes, a.out))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+122
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env bash
|
||||
# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
|
||||
# acceptance gate: three KLD reports against ONE reference recording, one verdict.
|
||||
#
|
||||
# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
|
||||
# kldref, in the same drained window that records the calibration grams). Every scored
|
||||
# container is measured against that one recording, so the KL scale runs from 0 = the
|
||||
# checkpoint itself downward in faithfulness. Three containers ride it:
|
||||
#
|
||||
# fp8.json the published FP8-native release — the achievable end of the interval
|
||||
# iq4r.json the payload frank-archon serves — the bar to beat
|
||||
# cand.json the arm under test — the subject
|
||||
#
|
||||
# The FP8 number is what makes the interval absolute instead of relative: "half the gap"
|
||||
# is half of the distance between the 4-bit payload and the near-lossless release, not half
|
||||
# of an invented distance to the checkpoint. That is also why the FP8 payload is FETCHED and
|
||||
# scored, never converted: it is a second opinion about what this checkpoint tolerates, not a
|
||||
# step of the campaign. It lands LAST (disk sequencing) — by then the 335 GiB bf16 container
|
||||
# is gone, so its 172.8 GiB fits.
|
||||
#
|
||||
# The law, in order of severity, on the mean KL of one slice (--slice all | a corpus source):
|
||||
# 1. KL(fp8) < KL(iq4r) the anchors are ordered. If the near-lossless
|
||||
# release does NOT sit closer to the checkpoint than
|
||||
# the 4-bit payload, the corpus, the tokenizer or the
|
||||
# boot shape is wrong and no arm number means anything.
|
||||
# 2. KL(fp8) < KL(cand) < KL(iq4r) the candidate lies strictly inside the interval —
|
||||
# outside it, something is mis-measured, not better
|
||||
# 3. KL(cand) <= KL(iq4r) - R x gap it closes at least R (--recover, default 0.5) of the
|
||||
# gap = KL(iq4r) - KL(fp8): at least half the way from
|
||||
# what is served toward the release (agreed 2026-10-27)
|
||||
# 4. top-1(cand) >= top-1(iq4r) the agreement the harness keys on does not fall
|
||||
#
|
||||
# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
|
||||
# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
|
||||
# same step size, --deterministic, no speculation) that makes the three numbers comparable.
|
||||
# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
|
||||
#
|
||||
# Usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
|
||||
set -uo pipefail
|
||||
|
||||
[ "$#" -ge 3 ] || { echo "usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2; exit 2; }
|
||||
python3 - "$@" <<'PY'
|
||||
import json, sys
|
||||
|
||||
a = sys.argv[1:]
|
||||
opts, files = [], []
|
||||
i = 0
|
||||
while i < len(a):
|
||||
if a[i].startswith("--"):
|
||||
opts += [a[i], a[i + 1]]
|
||||
i += 2
|
||||
else:
|
||||
files.append(a[i])
|
||||
i += 1
|
||||
if len(files) != 3:
|
||||
sys.exit("usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
|
||||
o = dict(zip(opts[0::2], opts[1::2]))
|
||||
slice_ = o.get("--slice", "all")
|
||||
recover = float(o.get("--recover", 0.5))
|
||||
if not 0.0 < recover <= 1.0:
|
||||
sys.exit(f"--recover must be in (0,1], got {recover}")
|
||||
|
||||
|
||||
def die(msg):
|
||||
print(msg, file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def read(path):
|
||||
try:
|
||||
d = json.load(open(path))
|
||||
except Exception as e:
|
||||
die(f"cannot read the KLD report {path}: {e}")
|
||||
if slice_ == "all":
|
||||
g = d.get("all")
|
||||
else:
|
||||
g = (d.get("by_source") or {}).get(slice_)
|
||||
if g is None:
|
||||
die(f"{path} has no slice {slice_!r}; it has: "
|
||||
f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
|
||||
if g is None or "kld" not in g or "top1_agreement" not in g:
|
||||
die(f"{path} has no {slice_!r} summary — wrong report version?")
|
||||
return d, g
|
||||
|
||||
|
||||
rows = []
|
||||
for label, path in zip(("fp8", "iq4r", "cand"), files):
|
||||
d, g = read(path)
|
||||
rows.append((label, path, d.get("model", "?"), g["kld"]["mean"], g["kld"]["p99_9"],
|
||||
g["top1_agreement"], g["positions"]))
|
||||
|
||||
print(f"reference: BF16 container; slice: {slice_}")
|
||||
print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8} positions")
|
||||
for label, path, model, kl, p999, top1, n in rows:
|
||||
print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}% {n}")
|
||||
r = {label: (kl, top1) for label, _p, _m, kl, _q, top1, _n in rows}
|
||||
kl_fp8, kl_iq4r, kl_cand = r["fp8"][0], r["iq4r"][0], r["cand"][0]
|
||||
t_iq4r, t_cand = r["iq4r"][1], r["cand"][1]
|
||||
gap = kl_iq4r - kl_fp8
|
||||
ok = True
|
||||
|
||||
|
||||
def law(n, text, passed):
|
||||
global ok
|
||||
ok = ok and passed
|
||||
print(f" [{'PASS' if passed else 'FAIL'}] {n}. {text}")
|
||||
|
||||
|
||||
law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
|
||||
gap > 0)
|
||||
law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
|
||||
kl_fp8 < kl_cand < kl_iq4r)
|
||||
need = kl_iq4r - recover * gap
|
||||
law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
|
||||
f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
|
||||
if gap > 0 else "gap is not positive; cannot score recovery",
|
||||
gap > 0 and kl_cand <= need)
|
||||
law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
|
||||
t_cand >= t_iq4r)
|
||||
print("KLD GATE " + ("PASS" if ok else "FAIL"))
|
||||
sys.exit(0 if ok else 1)
|
||||
PY
|
||||
@@ -147,10 +147,21 @@ def main():
|
||||
deep = corpus(160_000)
|
||||
wide = corpus(1_100_000)
|
||||
|
||||
# 3.28 chars/token is the measured ratio of this corpus through the
|
||||
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
|
||||
# deploy); 98.5 % of the window leaves the response room under the ceiling.
|
||||
chars = int(0.985 * maxlen * 3.28)
|
||||
# The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
|
||||
# (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
|
||||
# 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
|
||||
# stale constant sends leg B past --max-model-len, where the engine answers a
|
||||
# perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
|
||||
# context size (200000)") and the leg reads as a capacity failure. One small
|
||||
# request prices the ratio; 97 % of the window leaves the response room.
|
||||
sample = prompt(wide, 4000, "CAL", "cal-marker")
|
||||
cal, _, _ = post({"model": MODEL,
|
||||
"messages": [{"role": "user", "content": sample}],
|
||||
"max_tokens": 1, "temperature": 0.0,
|
||||
"chat_template_kwargs": {"enable_thinking": False}})
|
||||
ratio = len(sample) / cal["usage"]["prompt_tokens"]
|
||||
print(f"[cal] corpus ratio {ratio:.2f} chars/token")
|
||||
chars = int(0.97 * maxlen * ratio)
|
||||
# Each marker's FIRST hyphen-separated word is unique, because show()
|
||||
# scores the hit on that word — a shared prefix would score every slot HIT
|
||||
# on any answer that says the word.
|
||||
@@ -176,7 +187,7 @@ def main():
|
||||
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
|
||||
print(f"[B] total {time.time() - t0:.1f}s")
|
||||
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
|
||||
dchars = int(0.25 * maxlen * 3.28)
|
||||
dchars = int(0.25 * maxlen * ratio)
|
||||
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
|
||||
for i in range(seqs)]
|
||||
print(f"[D] {seqs} concurrent streams x 512 tokens, "
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# qwen4exp-qwen38-w5nl64-bf16trunk.recipe — every routed expert, BOTH projections, at five bits:
|
||||
# u5 codes into libquant's 32-level w5nl table, an E4M3 scale a group of 64 under the fixed 2^-13
|
||||
# second level, rotated by the 128-wide Hadamard, chosen by GPTQ against $CALIB and swept three
|
||||
# times by coordinate descent. This is `w5nl64a8h` in the engine's served-encoding table — the top of
|
||||
# the four-bit class — applied wider than any published radiance container of this checkpoint: the
|
||||
# author's own w5 rung (`qwen4exp-w5dn64-i8-hc8m.recipe`) moves only `down`, because that rung was
|
||||
# priced as a byte-for-error trade against prefill; this campaign buys the whole rung and pays for
|
||||
# it out of the pinned host pool (+15.3 GiB of expert plane, `--host-pool-mib 61440` per rank), not
|
||||
# out of the KV pool.
|
||||
#
|
||||
# Every other line is the served container's recipe (dump it with
|
||||
# `rad-info --recipe /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`), minus the rows this
|
||||
# campaign leaves at the checkpoint:
|
||||
#
|
||||
# * the trunk's block linears, the shared expert and the hyper-connection mixers are NOT listed —
|
||||
# dropping their i8/fp8 lines leaves them bf16, which is what the checkpoint holds. Measured bill
|
||||
# +1.978 GiB of always-resident weight = ~131k tokens of KV off the pool.
|
||||
# * `ssm_inz`/`ssm_out` STAY int8: the delta-net gates are the largest trunk rows (+1.9 GiB, 126k
|
||||
# tokens) and the least sensitive per byte.
|
||||
# * `mtp.draft_head` STAYS 2-bit and the PLE n-gram table STAYS one fixed E4M3 scale: the draft
|
||||
# head's rung is cheap and its acceptance cost is a decode question this campaign does not touch,
|
||||
# and bf16 for the n-gram table would pin +47.69 GiB of host RAM for a lookup.
|
||||
#
|
||||
# Protected experts: none in this file. `scripts/apply-protected-cast.sh` appends the derived set
|
||||
# (48, two per layer so each rank's 256-expert half stays even — the four-bit GEMM takes the rest of
|
||||
# a half as two equal tables by parity) from the fresh 12M-token energy/routing census, and they must
|
||||
# sit ABOVE the expert row because the first rule that matches a weight decides it.
|
||||
#
|
||||
# CALIB=/home/dank/campaign/qwen38-w5nl64/calib \
|
||||
# rad-convert /home/dank/models/Qwen3.8-Flash-Next \
|
||||
# --recipe this.file --imatrix .../imatrix.gguf -o out.rad
|
||||
#
|
||||
# The arm driver (`scripts/convert-qwen38-w5nl64.sh`) generates the per-arm variants of the expert
|
||||
# row from this template — cd=3 (control), cd=6, and `act_order=1` with NO cd= (libquant refuses the
|
||||
# pair, lq_registry.cpp:105) — rather than keeping three near-duplicate recipes that can drift apart.
|
||||
#
|
||||
# The whole campaign, in the order that fits on one disk (every step runs on dankserver):
|
||||
#
|
||||
# 1. scripts/download-qwen38-bf16.sh --fetch the vendor BF16 checkpoint, 335.3 GiB
|
||||
# 2. scripts/build-qwen38-bf16-rad.sh the BF16 reference container, another 335.3 GiB
|
||||
# 3. one drained window, two boots:
|
||||
# scripts/run-qwen38-bf16.sh kldref the KLD reference recording (the zero of scale)
|
||||
# scripts/run-qwen38-bf16.sh calib the 12M-position calibration tap -> grams
|
||||
# 4. trash qwen38-bf16.rad the peak-disk law: the converts need its room
|
||||
# 5. scripts/grams-to-imatrix.py grams -> imatrix.gguf in the campaign dir
|
||||
# 6. scripts/convert-qwen38-w5nl64.sh the arms, planned and billed before each one runs
|
||||
# 7. fetch the published FP8-native release, then per container:
|
||||
# scripts/run-qwen38-bf16.sh score RAD OUT fp8 anchor, served iq4r, each arm
|
||||
# then scripts/kld-gate-qwen38.sh fp8.json iq4r.json cand.json
|
||||
# 8. a winner is a trio edit plus `scripts/deploy.sh dankserver --service frank-archon`; the
|
||||
# rollback is the commit before it. If the win needs the bigger pinned plane, raise
|
||||
# FRANK_ARCHON_HOST_POOL_MIB and FRANK_ARCHON_TTM_TARGET_MIB in the SAME commit: the preflight
|
||||
# floor is 2xHOST_POOL + 48,824 + prefix cache + MNBT + the store, and a pool raised without
|
||||
# the TTM target is a boot refusal that Restart=always repeats forever, invisible to deploy.sh.
|
||||
|
||||
# PROTECTED-BEGIN — scripts/apply-protected-cast.sh splices the derived cast lines in here.
|
||||
# They MUST sit above the expert rule: the first rule whose glob matches a weight decides it, so a
|
||||
# cast line below the five-bit rule would never win. The set is derived from the campaign's own
|
||||
# 12M-token recording (`derive-protected-experts.py --target-count 48` over the fresh grams), and the
|
||||
# driver splices it into a per-arm COPY so this template stays the clean, reviewable baseline.
|
||||
# PROTECTED-END
|
||||
|
||||
# Every routed expert, trunk and MTP layer alike, both projections, at five bits. The experts' scales
|
||||
# sit between 2^-15 and 2^-10.6 under w4nl's top level of 22 and half a binade lower under w5nl's 30
|
||||
# — inside E4M3's normal range under 2^-13 either way. The MTP layer has no recording and is rounded
|
||||
# to nearest, which libquant says on stderr; it is one layer of 49 and only ever changes drafts.
|
||||
blk.*.ffn_*_exps.*.weight gptq table=w5nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
|
||||
|
||||
# The delta-net's in/out gates, int8 with a bf16 scale a group of 128 searched for least error — the
|
||||
# served recipe's lines, unchanged.
|
||||
blk.*.ssm_inz.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||
blk.*.ssm_out.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||
|
||||
# The MTP head's two mixing rows, E4M3 as served. They only ever change the drafts.
|
||||
mtp.fc_hidden.weight rtn codes=fp8_e4m3 group=128 scale=f32
|
||||
mtp.fc_embedding.weight rtn codes=fp8_e4m3 group=128 scale=f32
|
||||
|
||||
# The n-gram table at one fixed E4M3 scale, and the 2-bit draft head — both the served lines.
|
||||
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
|
||||
mtp.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
|
||||
Executable
+342
@@ -0,0 +1,342 @@
|
||||
#!/usr/bin/env bash
|
||||
# run-qwen38-bf16.sh — RUNS ON DANKSERVER. Boot a container over the campaign's
|
||||
# ONE measuring shape for the phases that need both R9700s. The engine owns the
|
||||
# cards, so frank-archon is stopped first and restored on every exit path — the
|
||||
# autoresearch pattern: stop, never mask (the unit file in /etc/systemd/system
|
||||
# is a symlink into the checkout and mask fails on a relinked unit; the deploy
|
||||
# restores it).
|
||||
#
|
||||
# kldref --kld-record the reference log-probabilities: the engine
|
||||
# runs /home/dank/corpora/out/kld.jsonl (197 prompts, 50,432
|
||||
# scored positions, score tail 256) through prefill, writes
|
||||
# /home/dank/campaign/qwen38-w5nl64/kld-ref, and EXITS. The
|
||||
# model behind it is the BF16 container from
|
||||
# scripts/build-qwen38-bf16-rad.sh — the vendor checkpoint's
|
||||
# own numbers, which is what makes it the zero point.
|
||||
# score RAD OUT --kld-ref score ONE container against that recording and
|
||||
# EXIT: the FP8-native anchor, the served iq4r, and every
|
||||
# arm. Deliberately the SAME engine pin, the same TP, the
|
||||
# same step size, the same fp8 KV, --deterministic and no
|
||||
# speculation as kldref — KL and top-1 agreement are only
|
||||
# comparable across boots when everything except the weights
|
||||
# is identical, so this mode is one argv with --model and
|
||||
# the two --kld-* flags changed, never a second shape.
|
||||
# RAD must sit under the models dir (the container reads it
|
||||
# through the ro /models mount); OUT lands on a writable
|
||||
# mount because --kld-out F also writes F.rows beside it.
|
||||
# calib serve on :7770 with the calibration tap open
|
||||
# (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
|
||||
# separate driver — scripts/calib-push.py --port 7770
|
||||
# --container qwen38-bf16 --corpus
|
||||
# /home/dank/corpora/out/calib.jsonl — pushes the corpus; at
|
||||
# 12M pooled positions the engine writes the Grams and the
|
||||
# tap switches itself off (calib-push prints TAP_DONE). Stop
|
||||
# this script after that; the trap brings frank-archon back.
|
||||
#
|
||||
# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
|
||||
# weights disk tier, deterministic (the resident set pinned to the loader's plan,
|
||||
# so a recorded reference and a scored candidate see the same one), no
|
||||
# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
|
||||
# and the production chat template rather than the one baked into the container —
|
||||
# every scored container is asked the same tokenised questions the reference
|
||||
# answered. The window is 32,768: kld.jsonl is 197 records with a longest of
|
||||
# 38,263 characters (longest prompt FIELD 36,855, median record 27,228). Its
|
||||
# token length — 10,554 for that longest record, counted by the engine itself —
|
||||
# is in the funding paragraph below; the kldref gate re-checks the file against
|
||||
# the window before it boots anything.
|
||||
#
|
||||
# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
|
||||
# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
|
||||
# removed: the card is split between the weights and the cache from what the
|
||||
# model needs." The budget resolver derives that split from --host-pool-mib and
|
||||
# --gpu-headroom-mib, and this rig inherits BOTH from frank-archon.env, the file
|
||||
# that owns them: the cache's funding is `elastic - expert_floor` and the expert
|
||||
# floor falls one MiB for every MiB of host pool (frank-archon.md, Laws), so the
|
||||
# pool buys KV as well as residency — a private pair is not a free knob. What
|
||||
# buying one cost, measured on the first kldref boot the same day with the w4
|
||||
# era's 12288 / 768: "I kv: the pool backs 6% of the worst case (4 sequences all
|
||||
# at 32768 tokens), 0.2 full-length sessions; a larger --host-pool-mib gives the
|
||||
# cache more" — a 270 MiB, 7,436-position cache, after which the recorder refused
|
||||
# every one of the 197 corpus documents ("E kld: document 0 is 10554 tokens and
|
||||
# the KV pool holds 7436, so it can never be scored") and exited 1. Production's
|
||||
# 49,152 per rank is affordable exactly when the rig runs: the rig stops
|
||||
# frank-archon first, and that stop frees the same 96 GiB the service was pinning.
|
||||
#
|
||||
# The 32,768-token window it pairs with is not up for grabs: the corpus's longest
|
||||
# prompt is 10,554 tokens by the engine's own count (the header's chars/4.97
|
||||
# estimate called that record ~7,700; the true ratio for this file is chars/3.6),
|
||||
# so the window clears it by 3x and the cache now funds it.
|
||||
#
|
||||
# The step size is ONE value for the recording and every scoring pass:
|
||||
# FRANK_QWEN38_MNBT, default 2,048. The gate's whole claim is that the reference,
|
||||
# the served iq4r and every arm rode one shape, so the ceiling is set by the
|
||||
# TIGHTEST payload the gate boots rather than by the bf16 reference: the quantised
|
||||
# expert FILE tiers (the served iq4r, the arms) stream only at MNBT<=2048 and
|
||||
# refuse the widened form above it, and 2,048 is what production holds anyway.
|
||||
# Recording the reference at 32,768 would therefore measure a shape the payloads
|
||||
# cannot reproduce; at 2,048 the 50,432 positions cost more steps, which is still
|
||||
# one drained window. Calibration keeps the wide 65,536 — grams are not logits,
|
||||
# nothing downstream is compared against a step size, and that is where a big
|
||||
# step pays (a disk-tier step costs the same at 1 token as at 64k).
|
||||
#
|
||||
# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
|
||||
# (RAD and OUT are score's two positionals: RAD is a container under the models
|
||||
# dir, OUT the report path whose directory is created if missing)
|
||||
# --dry-run print the exact `podman run` argv, one shell-quoted argument a
|
||||
# line, and exit 0: the service is NOT stopped, nothing is
|
||||
# mounted, nothing is written — the offline test seam. It still
|
||||
# validates the argv's own shape (paths through the /models mount,
|
||||
# a step size that is a number), because an argv that could not
|
||||
# boot is not worth printing.
|
||||
# --image REF override the digest pin (a probe boot, or the test seam)
|
||||
#
|
||||
# knobs: FRANK_QWEN38_MODELS (the dir mounted at /models), FRANK_QWEN38_RAD (the
|
||||
# bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
|
||||
# calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
|
||||
# the container reads through /models), FRANK_QWEN38_LOG_DIR, and
|
||||
# FRANK_QWEN38_MNBT — the step size shared by kldref and score (2,048 is
|
||||
# the file tier's ceiling and production's own value, four slots and all),
|
||||
# and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
|
||||
# see the corpus census above, and raise it together with the corpus).
|
||||
# kldref refuses (exit 2) when the longest record of the corpus it is about to
|
||||
# read is longer than MAX_MODEL_LEN x 3 characters — chars/3 being the
|
||||
# pessimistic token:char ratio, so the script says no before the engine finds out
|
||||
# inside a drained window, after the checkpoint work is already done. score
|
||||
# refuses when the reference dir is absent, when RAD is not on disk or not under
|
||||
# the models mount, or when OUT already exists: the trash law means a second
|
||||
# score over the same name is a deliberate act, and two numbers in one file is
|
||||
# how a campaign loses its own history.
|
||||
# Engine stdout goes to $LOG_DIR/qwen38_bf16_<mode>.log — the boot ledger is the
|
||||
# record of what the resolver actually gave the cards; the journal never sees it.
|
||||
# exit: the engine's code; 2 = refused before the service was touched
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/frank-archon.env"
|
||||
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
||||
|
||||
MODE=""
|
||||
POS=()
|
||||
DRY=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
kldref|calib|score) MODE="$1" ;;
|
||||
--dry-run) DRY=1 ;;
|
||||
--image) IMAGE="$2"; shift ;;
|
||||
*)
|
||||
# score's two positional arguments, and nothing else may be bare.
|
||||
if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
|
||||
POS+=("$1")
|
||||
else
|
||||
echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF] (unknown argument: $1)" >&2
|
||||
exit 2
|
||||
fi ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
if [ -z "$MODE" ]; then
|
||||
echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF] (no mode given)" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$MODE" = score ] && [ "${#POS[@]}" -ne 2 ]; then
|
||||
echo "usage: ${0##*/} score RAD OUT — RAD is the container to score, OUT the report to write (got ${#POS[@]} argument(s))" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
|
||||
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
|
||||
KLD_CORPUS="${FRANK_QWEN38_KLD_CORPUS:-/home/dank/corpora/out/kld.jsonl}"
|
||||
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
|
||||
TPL="${FRANK_QWEN38_TPL:-$MODELS/tpl.jinja}"
|
||||
TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
|
||||
LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
|
||||
LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
|
||||
KLD_REF_DIR="$CAMPAIGN/kld-ref"
|
||||
CALIB_DIR="$CAMPAIGN/calib"
|
||||
C=qwen38-bf16
|
||||
SERVICE=frank-archon
|
||||
# The card split, taken from the file that owns it instead of a private pair:
|
||||
# --host-pool-mib also funds the KV cache (see the header), so hardcoding it
|
||||
# here once starved the recorder. The `:?` guards keep an empty env fatal.
|
||||
HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
|
||||
GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
|
||||
|
||||
# ONE step size for both logits modes. The file tier streams only at <=2,048 and
|
||||
# the gate scores file-tier payloads (the served iq4r, the arms) against the
|
||||
# reference, so the reference is recorded at the ceiling the payloads can meet —
|
||||
# never two numbers that a KL between them would have to forgive.
|
||||
MNBT="${FRANK_QWEN38_MNBT:-2048}"
|
||||
if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
|
||||
echo "REFUSED: FRANK_QWEN38_MNBT must be a token count (got: $MNBT)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
|
||||
# env file says so for production), so this is the number the corpus must fit in.
|
||||
MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
|
||||
if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
|
||||
echo "REFUSED: FRANK_QWEN38_MAX_MODEL_LEN must be a token count (got: $MAX_MODEL_LEN)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Every container this campaign boots is read through the ro /models mount, so
|
||||
# the host path must live under it; the container-side spelling is what goes
|
||||
# into the argv. Same rule for the container a score pass opens.
|
||||
under_models() {
|
||||
case "$1" in
|
||||
"$MODELS"/*) printf '/models/%s' "${1#"$MODELS"/}" ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# same_dir A B — the two name one directory, judged only when BOTH exist: a dry
|
||||
# run has no fixtures on disk, and two absent paths are not the same place.
|
||||
same_dir() {
|
||||
[ -d "$1" ] && [ -d "$2" ] && [ "$(cd "$1" && pwd)" = "$(cd "$2" && pwd)" ]
|
||||
}
|
||||
|
||||
if [ "$MODE" = score ]; then
|
||||
SCORE_RAD="${POS[0]}"
|
||||
SCORE_OUT="${POS[1]}"
|
||||
OUT_DIR="$(dirname "$SCORE_OUT")"
|
||||
MODEL_IN="$(under_models "$SCORE_RAD")" || {
|
||||
echo "REFUSED: container to score $SCORE_RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
|
||||
exit 2
|
||||
}
|
||||
else
|
||||
MODEL_IN="$(under_models "$RAD")" || {
|
||||
echo "REFUSED: reference container $RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
|
||||
exit 2
|
||||
}
|
||||
fi
|
||||
|
||||
# The measuring shape. Identical for kldref and score by construction — this
|
||||
# array is the only place the campaign's KL numbers are defined.
|
||||
ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
|
||||
--host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
|
||||
--deterministic --num-speculative-tokens 0
|
||||
--kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
|
||||
--ngram-placement ram
|
||||
--override-chat-template /models/tpl.jinja)
|
||||
MOUNTS=(-v "$MODELS:/models:ro")
|
||||
TAPENV=()
|
||||
case "$MODE" in
|
||||
kldref)
|
||||
# Recording is prefill-only over four slots and the shared step, which is
|
||||
# production's own operating point (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4
|
||||
# with FRANK_ARCHON_MAX_BATCHED_TOKENS=2048). Corpus read-only, record dir rw.
|
||||
ARGS+=(--model "$MODEL_IN"
|
||||
--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
|
||||
--max-num-seqs 4 --max-num-batched-tokens "$MNBT")
|
||||
MOUNTS+=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
|
||||
-v "$KLD_REF_DIR:/data/kld-ref")
|
||||
;;
|
||||
score)
|
||||
# The same argv with three words changed: which container, and that this
|
||||
# time the engine reads a recording and writes a report instead of the
|
||||
# reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
|
||||
# able to touch the zero point — and the report gets its own writable bind.
|
||||
if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR"; then
|
||||
echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
|
||||
exit 2
|
||||
fi
|
||||
ARGS+=(--model "$MODEL_IN"
|
||||
--kld-ref /data/kld-ref --kld-out "/data/out/$(basename "$SCORE_OUT")"
|
||||
--max-num-seqs 4 --max-num-batched-tokens "$MNBT")
|
||||
MOUNTS+=(-v "$KLD_REF_DIR:/data/kld-ref:ro"
|
||||
-v "$OUT_DIR:/data/out")
|
||||
;;
|
||||
calib)
|
||||
# Calibration is exempt from the shared step: the tap counts pooled
|
||||
# positions and nothing downstream is compared against a step size, so the
|
||||
# wide step is free throughput (a disk-tier step costs the same at 1 token
|
||||
# as at 64k).
|
||||
ARGS+=(--model "$MODEL_IN" --port 7770
|
||||
--max-num-seqs 8 --max-num-batched-tokens 65536)
|
||||
MOUNTS+=(-v "$CALIB_DIR:/data/calib")
|
||||
TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
|
||||
-e RADIANCE_CALIB_TOKENS=12000000)
|
||||
;;
|
||||
esac
|
||||
|
||||
CMD=(podman run --rm --name "$C"
|
||||
--network host --init
|
||||
--security-opt seccomp=unconfined
|
||||
--ulimit memlock=-1:-1
|
||||
--device /dev/kfd --device /dev/dri
|
||||
"${MOUNTS[@]}" "${TAPENV[@]}"
|
||||
"$IMAGE" "${ARGS[@]}")
|
||||
|
||||
# ---- gate: the window must hold the corpus (kldref alone can meet it) --------
|
||||
# score reads a recording instead of a corpus, and every arm declares at the
|
||||
# checkpoint's training context, so neither can meet a sequence too long to boot.
|
||||
# Measured on the raw JSON line — the pessimistic direction, and the same figure
|
||||
# the census in the header quotes. It runs on a dry run too, because a refusal
|
||||
# this cheap should never have to wait for the drained window to be discovered.
|
||||
if [ "$MODE" = kldref ] && [ -f "$KLD_CORPUS" ]; then
|
||||
read -r REC_NO REC_CHARS < <(awk '{ if (length($0) > best) { best = length($0); where = NR } } END { print where + 0, best + 0 }' "$KLD_CORPUS")
|
||||
LIMIT_CHARS=$(( MAX_MODEL_LEN * 3 ))
|
||||
if [ "$REC_CHARS" -gt "$LIMIT_CHARS" ]; then
|
||||
echo "REFUSED: record $REC_NO of $KLD_CORPUS is $REC_CHARS chars, past the $LIMIT_CHARS a $MAX_MODEL_LEN-token window promises at the pessimistic chars/3 — raise FRANK_QWEN38_MAX_MODEL_LEN, and the container's declared --max-ctx with it, before booting the reference" >&2
|
||||
exit 2
|
||||
fi
|
||||
echo "corpus fits: longest record $REC_CHARS chars (number $REC_NO) inside the $MAX_MODEL_LEN-token window (limit $LIMIT_CHARS chars)"
|
||||
fi
|
||||
|
||||
if [ "$DRY" = 1 ]; then
|
||||
echo "dry run: $MODE — the production unit is left running, nothing is mounted, nothing is written"
|
||||
printf '%q\n' "${CMD[@]}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- gates that only a real run needs ---------------------------------------
|
||||
if [ "$MODE" = score ]; then
|
||||
if [ ! -d "$KLD_REF_DIR" ]; then
|
||||
echo "REFUSED: no KLD reference at $KLD_REF_DIR — record it first in a drained window: scripts/run-qwen38-bf16.sh kldref" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ ! -f "$SCORE_RAD" ]; then
|
||||
echo "REFUSED: no container to score at $SCORE_RAD" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ -e "$SCORE_OUT" ]; then
|
||||
echo "REFUSED: $SCORE_OUT already exists — one report per name; move the old one to trash before re-scoring" >&2
|
||||
exit 2
|
||||
fi
|
||||
mkdir -p "$OUT_DIR"
|
||||
else
|
||||
if [ ! -f "$RAD" ]; then
|
||||
echo "REFUSED: no reference container at $RAD — build it first: scripts/build-qwen38-bf16-rad.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
[ "$MODE" = kldref ] && [ ! -f "$KLD_CORPUS" ] && {
|
||||
echo "REFUSED: no KLD corpus at $KLD_CORPUS — the record would be empty" >&2
|
||||
exit 2
|
||||
}
|
||||
fi
|
||||
if [ ! -f "$TEMPLATE" ]; then
|
||||
echo "REFUSED: no chat template at $TEMPLATE" >&2
|
||||
exit 2
|
||||
fi
|
||||
mkdir -p "$LOG_DIR"
|
||||
# The prod template, staged where the ro /models mount reaches it: a reference
|
||||
# recorded with the container's baked template scores a different tokenisation
|
||||
# than the arms served with froggeric's, and the KL would be measuring that.
|
||||
cp -f "$TEMPLATE" "$TPL" || {
|
||||
echo "REFUSED: cannot stage the chat template at $TPL (the container reads it through /models)" >&2
|
||||
exit 2
|
||||
}
|
||||
[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
|
||||
[ "$MODE" = calib ] && mkdir -p "$CALIB_DIR"
|
||||
|
||||
restore() { sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true; }
|
||||
trap restore EXIT
|
||||
echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
|
||||
sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
|
||||
podman rm -f "$C" >/dev/null 2>&1 || true
|
||||
|
||||
"${CMD[@]}" 2>&1 | tee "$LOG"
|
||||
rc=${PIPESTATUS[0]}
|
||||
echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
|
||||
exit "$rc"
|
||||
@@ -15,18 +15,21 @@
|
||||
# log states every budget before anything is allocated (weights vs KV
|
||||
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
||||
# the P2P answer per pair) — read it after every resize.
|
||||
# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000
|
||||
# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in
|
||||
# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab
|
||||
# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair
|
||||
# and the budget resolver derives it from headroom, the cache bound and the host
|
||||
# pool, so the slab size and the pool's token count are LEDGER lines: after every
|
||||
# boot read them out of the log and check the pool still backs 3 x 200,000.
|
||||
# Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a
|
||||
# >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache
|
||||
# in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
|
||||
# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
|
||||
# resolver derives it from headroom, the cache bound and the host pool, so the slab
|
||||
# size and the pool's token count are LEDGER lines. After every boot read them, plus
|
||||
# the two lines that size a prefill step — "activation arena: ... lends ... while
|
||||
# steps are at most N tokens" (the arena levels the expert slab borrows) and
|
||||
# "prefill staging: ... a step of more than N tokens ..." (the chunk size below which
|
||||
# pooled experts are read synchronously) — and the cache's session backing.
|
||||
#
|
||||
# The root unit (scripts/frank-archon.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
|
||||
# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
|
||||
# pages, which is ALSO why this script refuses to start when ttm's
|
||||
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 48 GiB expert pools + the 47.7 GiB PLE
|
||||
# table + 8 GiB prefix host tier + the checkpoint store (slots x 112 MiB) — ~159 GiB of
|
||||
# pinned pages, which is ALSO why this script refuses to start when ttm's
|
||||
# pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
|
||||
set -euo pipefail
|
||||
|
||||
|
||||
@@ -136,10 +136,17 @@ fi
|
||||
# The engine re-checks size+sha256 against that catalog at boot; it cannot
|
||||
# fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
|
||||
# stays unset — no outbound connection at boot),
|
||||
# 2. the distro XRT's three libraries, each mounted twice (where the image
|
||||
# looks for XRT, and at its own path — the libraries find each other there;
|
||||
# an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
|
||||
# container, which the vendor calls out explicitly),
|
||||
# 2. the distro XRT's three libraries, each mounted THREE times: where the
|
||||
# image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
|
||||
# libraries find each other there), and at /opt/xilinx/lib/<multiarch>
|
||||
# — the NPU child searches that third prefix too, and since the host's
|
||||
# libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
|
||||
# names: omitting it crashes the lane with `No such library
|
||||
# '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
|
||||
# supervisor takes the dead lane down on the whole container (measured
|
||||
# 2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
|
||||
# of symlinks into the system dir dangles inside the container, which the
|
||||
# vendor calls out explicitly),
|
||||
# 3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
|
||||
# hang the machine while FCLK changes speed, and the engine refuses to
|
||||
# start when it moves — it cannot read host sysfs from inside the
|
||||
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
|
||||
exit 1
|
||||
fi
|
||||
REAL="$(readlink -f "$XRT_DIR/$L")"
|
||||
NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
|
||||
MULTIARCH="${XRT_DIR##*/}"
|
||||
NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
|
||||
-v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
|
||||
done
|
||||
HELD=0
|
||||
for D in "$SYSFS"/class/drm/card*/device; do
|
||||
|
||||
@@ -293,6 +293,14 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/caddyfile.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# The GPU exporter is a host process; its seam pins the collector, the
|
||||
# user unit, the deployer function, the planner route, the Prometheus
|
||||
# scrape jobs and the host-system GPU panels — a cross-file contract, so
|
||||
# the whole seam rides on any of its own files.
|
||||
scripts/gpu-exporter.py|scripts/gpu-exporter.service)
|
||||
add_test tests/gpu_exporter.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
dankclaw/udev/*)
|
||||
# Host-level udev rules are applied by manual sudo install + udevadm
|
||||
# trigger, not dispatched as a service; nothing here is bats-testable.
|
||||
|
||||
@@ -52,28 +52,31 @@ Where the contract lives:
|
||||
The arithmetic (the coupled contract every endpoint's maxTokens comment
|
||||
encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
|
||||
endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
|
||||
follows is a **worked example** recomputed from the live files on 2026-10-18 —
|
||||
a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
|
||||
follows is a **worked example** recomputed from the live files on 2026-10-21 —
|
||||
a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
|
||||
maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
|
||||
|
||||
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
|
||||
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
|
||||
95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
|
||||
bundle clamps the percentage to 1–99 and converts it to a token threshold.
|
||||
2. **Output budget** = capped at the target model's `maxTokens` — every role
|
||||
here compacts to frank-zealot, so the budget is its 16,384.
|
||||
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
|
||||
window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
|
||||
200,704. The registry's own comments design around an 85 % trigger (archon:
|
||||
170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
|
||||
does hold. While `config.yml` and those comments disagree, treat 85 % as the
|
||||
design point and re-run this arithmetic after ANY window change — a 400 from
|
||||
the compaction target mid-conversation is the symptom.
|
||||
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
|
||||
dragoon session the fitting window is the ENGINE ceiling, not a slot: the
|
||||
190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
|
||||
error plus the 45,056 request budget lands inside — that headroom is the
|
||||
reason the ceiling moved to native on 2026-10-21. For a session compacting
|
||||
INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
|
||||
does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
|
||||
200,704. Treat 85 % as the design point until config.yml and the registry
|
||||
comments agree, and re-run this arithmetic after ANY window change — a 400
|
||||
from the compaction target mid-conversation is the symptom.
|
||||
(Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
|
||||
177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
|
||||
32,768 ≤ 200,704 zealot.)
|
||||
|
||||
Procedure (recompute from the LIVE files every time — windows rotate with
|
||||
every swap; the block above is the 2026-10-18 state):
|
||||
every swap; the block above is the 2026-10-21 state):
|
||||
|
||||
1. Read the `config.yml` compaction block + `models.yml` entries for source
|
||||
and target. Every endpoint's `contextWindow` comment carries its own
|
||||
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
|
||||
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
|
||||
dankclaw — the comment/config disagreed. The comments now name the real
|
||||
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
|
||||
truth; fix comments in the same change.** A fresh instance of the same trap is
|
||||
open 2026-10-18: every `maxTokens` comment still says "compaction fires at
|
||||
85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
|
||||
truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
|
||||
disagreement is not academic: it is why long dragoon sessions kept drawing
|
||||
400s at the old 200,704 ceiling — the trigger fires late AND the est
|
||||
under-counts, and a window parked near the ceiling has no room for either.
|
||||
`maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
|
||||
input. Boot-shape OOM laddering is a different problem
|
||||
([[Llama Cpp Serving Sizing]]).
|
||||
|
||||
@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
|
||||
## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
|
||||
|
||||
- **Hard-fail keys** — the whole unit silently fails to generate
|
||||
(`systemctl --user cat <unit>` shows "could not be found"): any
|
||||
dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
|
||||
in group 'Container'`); `Command=` is unsupported, use `Exec=`.
|
||||
- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
|
||||
live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
|
||||
generator log `unsupported key … in group 'Container'` plus
|
||||
`processing encountered some errors` and generate NOTHING for that
|
||||
manager. The running containers keep going as orphans, so the failure
|
||||
first appears on the next deploy as `Unit <x>.service not found` for a
|
||||
service that is visibly up. Other offenders: any dash-prefixed
|
||||
`[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
|
||||
The seam `tests/quadlet_image_pins.bats` fails the build on any
|
||||
generator-rejected key in any host's units.
|
||||
Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
|
||||
`Exec=` — repeating the binary makes it a positional arg
|
||||
(`unexpected /bin/blackbox_exporter`, live 2026-10-21).
|
||||
- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
|
||||
spaces into the healthcheck argv; a quoted probe
|
||||
`/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
|
||||
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
"unsupported key"`, `systemctl --user cat probe.service` (shows the
|
||||
generated ExecStart verbatim), then remove + reload.
|
||||
|
||||
## 4a. Image pins (fleet law, 2026-10-21)
|
||||
|
||||
Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
|
||||
locally built images (`localhost/*`, `franksdank/*`) are exempt — their
|
||||
build is the pin. A rolling tag lets a rebuild move under a running
|
||||
service with nobody approving it, and it makes a CVE audit ungradeable.
|
||||
Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
|
||||
tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
|
||||
version the config label carries in the comment beside it.
|
||||
|
||||
## 5. Rootless Quadlet log retrieval
|
||||
|
||||
Step 0 — establish ownership: `systemctl --user status <name>.service`
|
||||
|
||||
@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
|
||||
docs, unit docstrings, and the repo's rule file (the most expensive place to
|
||||
be stale).
|
||||
|
||||
## Three rules that outlive the incident that taught them
|
||||
|
||||
* **A target can read down beside a healthy service.** Read `lastError` from
|
||||
`/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
|
||||
protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
|
||||
* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
|
||||
process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
|
||||
not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
|
||||
host volume: silences + notification log, [[Alertmanager]]).
|
||||
* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
|
||||
unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
|
||||
[[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
|
||||
the process shape can change between two censuses ten minutes apart.
|
||||
|
||||
## Order and verification
|
||||
|
||||
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
|
||||
|
||||
@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
|
||||
|
||||
## Recommended additions
|
||||
|
||||
Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
|
||||
reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
|
||||
2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
|
||||
Quantized preferred. The iGPU's GTT aperture is not free RAM beside
|
||||
frank-dragoon — size every pick against *What actually fits beside
|
||||
frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
|
||||
all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
|
||||
`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
|
||||
`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
|
||||
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
|
||||
never on the command line.
|
||||
|
||||
Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
|
||||
the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
|
||||
them at GTT capacity, and they double the disk bill).
|
||||
the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
|
||||
bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
|
||||
is not the real budget.
|
||||
|
||||
## What actually fits beside frank-dragoon
|
||||
|
||||
The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
|
||||
an APU the aperture and `MemAvailable` are the same physical RAM, and
|
||||
frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
|
||||
live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
|
||||
~10 MB, so the budget for a generation run is roughly what `free` says.
|
||||
|
||||
Weights must land inside that budget with slack for compute buffers and
|
||||
activations (the mmproj-class lesson: price a model at ~1.5× its file set):
|
||||
|
||||
| Pipeline (weights) | Fits beside frank-dragoon? |
|
||||
|---|---|
|
||||
| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
|
||||
| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
|
||||
| Any BF16 combination | No |
|
||||
| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
|
||||
|
||||
**The law: one process owns the memory — same rule as the 3090 bench law.**
|
||||
For a Qwen-Image render: stop and mask frank-dragoon
|
||||
(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
|
||||
`Restart=always` from firing mid-render), generate, unmask, verify
|
||||
`curl 192.168.0.143:7770/health`. The proven workflows above then run with
|
||||
the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
|
||||
- Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
|
||||
- The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
|
||||
- The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
|
||||
- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
|
||||
memory.** On the APU it is system RAM, and frank-dragoon pins most of it
|
||||
(`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
|
||||
Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
|
||||
stop-mask-render-verify procedure and the fit table are in
|
||||
[[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
|
||||
The picker is built from the `model` table in `webui.db` (one row per model)
|
||||
plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
|
||||
connection discovery — so the picker shows exactly what is seeded there.
|
||||
As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
|
||||
2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
|
||||
`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
|
||||
`shared/homepage/services.json`)*.
|
||||
Since 2026-10-21 the picker exposes **all three fleet models to every
|
||||
account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
|
||||
`frank-archon` to everyone; the other two were already public). Proven with
|
||||
a `role=user` token: `/api/models` lists all three ids and a chat turn to
|
||||
`frank-archon` answers end-to-end.
|
||||
|
||||
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
|
||||
| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
|
||||
|---|---|---|---|---|
|
||||
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
|
||||
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
|
||||
| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
|
||||
| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
|
||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
|
||||
|
||||
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
|
||||
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
|
||||
is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
|
||||
that day's payload swap (see [[frank-zealot]] for what was deployed and when).
|
||||
|
||||
The `model` table carries exactly two decoration rows, `frank-zealot` and
|
||||
`frank-dragoon`. Both the context length and the description live **inside
|
||||
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
|
||||
The `model` table carries three decoration rows — `frank-zealot`,
|
||||
`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
|
||||
Both the context length and the description live **inside JSON columns** —
|
||||
`params` holds `{"context": {"length": …}}` and `meta` holds
|
||||
`{"description": …}`; there is no `context_length` or `description` column, so
|
||||
`sqlite3` one-liners against those names fail. `params.context.length` is a copy
|
||||
of the endpoint's per-slot window, so the authoritative number lives on the
|
||||
@@ -380,6 +382,20 @@ neither.
|
||||
name: /selected model/i })`) and the welcome text "How can I help you
|
||||
today?"; that snapshot is also the cheapest proof the picker is populated.
|
||||
|
||||
- **2026-10-21 — picker opened to all three models, for everyone.** Per
|
||||
Frank: `frank-archon` accessible to everyone. Three-leg repair on
|
||||
`webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
|
||||
`enable: true`, a `model` row `frank-archon` (params
|
||||
`{"context":{"length":200000}}`, meta description from the endpoint card),
|
||||
and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
|
||||
principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
|
||||
role row. Verified as a plain `role=user` token: `/api/models` lists all
|
||||
three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
|
||||
package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
|
||||
come from **PID 1's environ** (`podman exec` sessions do not inherit the
|
||||
app process env; the `.webui_secret_key` file happens to equal it — use it
|
||||
explicitly, `podman exec -e`).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
|
||||
tags:
|
||||
- dankserver
|
||||
- plex
|
||||
timestamp: 2026-09-15
|
||||
timestamp: 2026-10-21
|
||||
---
|
||||
|
||||
# Plex
|
||||
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
||||
| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
|
||||
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
||||
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
||||
| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
|
||||
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
||||
|
||||
@@ -3,7 +3,7 @@ type: reference
|
||||
title: Observability Provisioning
|
||||
description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
|
||||
tags: [observability, prometheus, grafana, provisioning, deployment]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
timestamp: 2026-10-21T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Observability Provisioning
|
||||
@@ -20,6 +20,8 @@ committed.
|
||||
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
|
||||
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
||||
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
||||
| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
|
||||
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
|
||||
|
||||
Targets and datasources reference **container names** (`grafana:3000`
|
||||
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
|
||||
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
|
||||
reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
|
||||
the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
|
||||
(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
|
||||
HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
|
||||
[[Stale Signal Audit]]).
|
||||
HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
|
||||
deployed 2026-10-21; see [[Stale Signal Audit]]).
|
||||
|
||||
## The deploy path
|
||||
|
||||
@@ -43,8 +45,9 @@ from `deploy_observability`:
|
||||
- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
|
||||
- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
|
||||
- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
|
||||
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
|
||||
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
|
||||
- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
|
||||
|
||||
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
|
||||
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
|
||||
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
|
||||
the backend (`http://prometheus:9090`), not loopback.
|
||||
|
||||
**Add a dashboard** — drop the JSON into
|
||||
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
|
||||
unified storage, not (only) the SQLite table — grep the API, not the DB, when
|
||||
verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
|
||||
2026-10-18: the storage-backend claim needs an admin password to confirm
|
||||
against `grafana.db` and the API side by side)*
|
||||
`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
|
||||
the next `--service grafana` deploy provisions it. This is a repaired law, not
|
||||
a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
|
||||
`containers` board — merged, tested, documented as "working" — had **never
|
||||
been provisioned**; the other boards existed on disk only because someone had
|
||||
hand-copied them once. A repo JSON is not a live dashboard until the copier
|
||||
carries it, and `deploy.sh` cannot see the gap because it only reports the
|
||||
copy it performed.
|
||||
|
||||
## Verify through the APIs, not the filesystem
|
||||
|
||||
@@ -95,90 +101,32 @@ against `grafana.db` and the API side by side)*
|
||||
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
|
||||
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
|
||||
# grafana: datasource + dashboard provisioned
|
||||
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
|
||||
# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
|
||||
# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
|
||||
# account is the Authentik RP login). The proven surface is a real browser
|
||||
# session: warm the Anubis challenge on the vhost root, then hit
|
||||
# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
|
||||
# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
|
||||
```
|
||||
|
||||
The two Prometheus queries were run on 2026-10-18 and are what every number on
|
||||
this page comes from: 4 alerting + 2 recording rules, and `up` for
|
||||
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
|
||||
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
|
||||
listed in [[Prometheus]]. The Grafana query needs the local admin password and
|
||||
was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
|
||||
proof of installation only — see the next paragraph.
|
||||
`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
|
||||
that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
|
||||
2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
|
||||
filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
|
||||
only — see *Add a dashboard* for why a repo JSON is not a live board.
|
||||
|
||||
Health answers from a stale process and provisioned resources can hide in
|
||||
unified storage — a file existing on disk is not proof the running server
|
||||
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
||||
table (refused = outage, 404/HTML = config defect).
|
||||
|
||||
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
|
||||
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
|
||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
||||
via `host.containers.internal`) → signal-cli, addressed by number
|
||||
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
|
||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
|
||||
`curl -s http://192.168.0.143:18093/health` → `ok`, and
|
||||
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
|
||||
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
|
||||
`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
|
||||
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
||||
the blackbox exporter. See [[Prometheus]] → Targets.
|
||||
|
||||
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
|
||||
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
|
||||
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
|
||||
`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
|
||||
is started with the flag, so the scrape was refused at the protocol layer (the
|
||||
flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
|
||||
stayed firing:
|
||||
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
|
||||
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
|
||||
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
|
||||
metrics opt-in landed that day; the target
|
||||
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
|
||||
How many payload generations it spanned is not readable from any host
|
||||
artifact. *(unverified 2026-10-18)*
|
||||
|
||||
Two laws fall out of it. A target can read down beside a healthy service, so
|
||||
read `lastError` from `/api/v1/targets` before believing an outage. And rule
|
||||
state lives in Prometheus' process, not on disk — a restart re-stamps
|
||||
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
|
||||
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
|
||||
`--storage.path=/alertmanager` on a host volume keeps silences and the
|
||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
|
||||
`/opt/alertmanager/data`.)
|
||||
|
||||
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
||||
metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
|
||||
([[frank-dragoon]]), which is why
|
||||
`frank-dragoon` has always been green. The active-alert count is a
|
||||
moment-in-time value and should never be documented as a constant:
|
||||
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
|
||||
`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
|
||||
minutes later, and was empty again by 11:56Z; the relay logged the matching
|
||||
firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
|
||||
|
||||
**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
|
||||
production `frank-zealot` root unit was stopped **on purpose** for a benchmark
|
||||
window on the 3090 — the repo Benchmarking Policy says stop the production
|
||||
service before benching — and `:7771` answered from a manually booted candidate
|
||||
arm (`systemctl is-active frank-zealot` → `failed` after
|
||||
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
|
||||
`TimeoutStopSec`, the known shutdown shape for these engines). So
|
||||
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
|
||||
shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
|
||||
and it is the repo contract.
|
||||
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
|
||||
reinstalls and starts it, restoring boot persistence and self-healing. Never
|
||||
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
|
||||
changed between two censuses ten minutes apart.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — stack topology and the alert delivery chain
|
||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
|
||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
|
||||
down-beside-healthy / alert-state-persistence / candidate-arm rules
|
||||
- [[Quadlet Lifecycle]] — the units that host these services
|
||||
- [[Deploy State Triage]] — how the planner selects these services
|
||||
@@ -2,7 +2,7 @@
|
||||
type: reference
|
||||
title: "Observability on Dankclaw"
|
||||
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
|
||||
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
|
||||
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
|
||||
timestamp: 2026-10-20
|
||||
---
|
||||
|
||||
@@ -10,10 +10,10 @@ timestamp: 2026-10-20
|
||||
|
||||
## Summary
|
||||
|
||||
Seven containers and one host-side process: Prometheus (metrics), Grafana
|
||||
Seven containers and two host-side processes: Prometheus (metrics), Grafana
|
||||
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
|
||||
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
|
||||
carrying every firing group to Signal. All of it is Quadlet-managed,
|
||||
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
|
||||
plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
|
||||
boot-persistent and loopback-only, deployed by
|
||||
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
|
||||
end to end; the log half is configured, healthy, and carries no data.
|
||||
@@ -35,6 +35,7 @@ that proves it today.
|
||||
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
|
||||
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
|
||||
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
|
||||
| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
|
||||
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
|
||||
|
||||
Every unit lives in `dankclaw/observability/quadlet/`, carries
|
||||
@@ -45,10 +46,11 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
|
||||
the same day, and `homepage-status.timer` — the root unit that probes all of
|
||||
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
|
||||
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
|
||||
Dankserver runs two members of this family, node-exporter and cadvisor, as
|
||||
its own units (`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
|
||||
Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
|
||||
(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
|
||||
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
|
||||
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`).
|
||||
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
|
||||
`gpu-exporter` (a user unit installed the same way as `signal-relay`).
|
||||
|
||||
## Exposure
|
||||
|
||||
@@ -65,10 +67,26 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
|
||||
in the host network namespace, and it binds exactly two addresses rather than
|
||||
the wildcard — see [[Node-Exporter]].
|
||||
|
||||
The relay's bind is the one address a bridge container can reach on the host:
|
||||
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
|
||||
and the podman bridge subnet `10.89.0.0/16` before the DROP
|
||||
(`scripts/deploy-remote.sh:255-262`).
|
||||
**Same-host reachability from a bridge container is measured, not inferred**
|
||||
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
|
||||
(the podman gateway) reaches host services whether they bind host-net or are
|
||||
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
|
||||
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
|
||||
every same-host Prometheus target uses the gateway and only cross-host targets
|
||||
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
|
||||
relay at `192.168.0.143:18093` is reached from alertmanager's network only
|
||||
while `install_signal_relay`'s INPUT rules are in force: loopback, the host
|
||||
itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
|
||||
trailing blanket DROP. The law the poisoned host taught (2026-10-21):
|
||||
`iptables -I` prepends, so inserting the DROP last lands it first — the old
|
||||
install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
|
||||
deployer's health gate failed beside a LISTENing socket, and the drain
|
||||
aborted twice before `nft list ruleset` named the cause. The install now
|
||||
rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
|
||||
DROP appended); verify any change to it with the sandboxed bats case, not
|
||||
with `-C` greps — `-C` answers existence, never order. Never point a scrape
|
||||
target at a host LAN IP from its own host on the strength of one working
|
||||
example; dial the gateway and probe.
|
||||
|
||||
## Deployment
|
||||
|
||||
@@ -85,13 +103,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
|
||||
[[Observability Provisioning]] for what is provisioned as code and how to
|
||||
verify it through the APIs.
|
||||
|
||||
## Current state (2026-10-20)
|
||||
## Current state (2026-10-21)
|
||||
|
||||
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
|
||||
eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
|
||||
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
|
||||
and both node exporters. The four downs are structural (the `comfyui` and
|
||||
`services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
|
||||
every job up — the three LLM endpoints (frank-archon on `vllm:*`,
|
||||
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
|
||||
both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
|
||||
the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
|
||||
downs are retired — see [[Prometheus]] → Targets.
|
||||
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
|
||||
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
|
||||
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
|
||||
@@ -99,12 +118,15 @@ verify it through the APIs.
|
||||
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
|
||||
relay, and `journalctl --user -u signal-relay` records successful
|
||||
`sent to +12062476336:` lines for every rule in the set.
|
||||
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
|
||||
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
|
||||
`containers`, `observability` and `logs-overview` dashboards, all
|
||||
installed from `shared/grafana/provisioning/`. The Loki datasource is how
|
||||
you read logs: Loki has no web UI of its own, so Grafana Explore is the
|
||||
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
|
||||
* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
|
||||
datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
|
||||
`llm-fleet`, `host-system`, `containers`, `observability` and
|
||||
`logs-overview` boards. Until 2026-10-21 the copier carried only
|
||||
`llm-fleet.json` by name and the `containers` board had never been
|
||||
provisioned; the deployer now globs the directory. The Loki datasource is
|
||||
how you read logs: Loki has no web UI of its own, so Grafana Explore is the
|
||||
human surface (the guarded `loki.franksdank.cloud` vhost serves only the
|
||||
API). Verify method and the copier law: [[Observability Provisioning]].
|
||||
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
|
||||
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
|
||||
per-container CPU/memory here, and on this rootless-Podman host it works on one
|
||||
@@ -117,25 +139,38 @@ verify it through the APIs.
|
||||
`container` label. Network and filesystem are NOT exported for these containers
|
||||
(no netns/classid attribution under the host-PID layout), so the dashboard
|
||||
graphs only CPU, memory and OOM and does not fake the rest.
|
||||
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
|
||||
a rootless Alloy cannot read the `0640` journal files, and the container-log
|
||||
globs point at a root-storage layout that does not exist here. The fix is built
|
||||
and tested in-repo (2026-10-20): Alloy is now a **root system service on
|
||||
`Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
|
||||
Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
|
||||
lines and pushed 26,966 with zero drops. The **live** instance is still the old
|
||||
rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
|
||||
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
|
||||
* **Not configured:** no blackbox exporter (the structural downs have no real
|
||||
uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
|
||||
not yet deployed (see *Logs* above).
|
||||
* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
|
||||
`deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
|
||||
host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
|
||||
own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
|
||||
without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
|
||||
`mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
|
||||
emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
|
||||
°C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
|
||||
job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
|
||||
`power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
|
||||
the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
|
||||
node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
|
||||
* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
|
||||
pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
|
||||
the container-log globs pointed at a root-storage layout that does not exist
|
||||
here) — fixed 2026-10-20 by making Alloy a **root system service on
|
||||
`Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
|
||||
but the stream carried only `job`/`service_name`, so every per-container
|
||||
view was structurally empty: Loki metrics queries group only by INDEXED
|
||||
stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
|
||||
label (plus a `host` label) at ingestion — `loki.process` in
|
||||
`shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
|
||||
purpose-test law in [[Observability Provisioning]].
|
||||
|
||||
## Citations
|
||||
|
||||
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
|
||||
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
|
||||
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
|
||||
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
|
||||
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
|
||||
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
|
||||
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
|
||||
- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
|
||||
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
|
||||
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
|
||||
@@ -42,29 +42,29 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
||||
|
||||
The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
|
||||
|
||||
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
|
||||
**No job reads down structurally any more** (the three permanent-reds were
|
||||
retired 2026-10-21; census re-run after each deploy):
|
||||
|
||||
| Job | Target | `health` | `lastError`, verbatim |
|
||||
|-----|--------|----------|------------------------|
|
||||
| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
|
||||
| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
|
||||
| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
|
||||
| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
|
||||
| Job | Target | how |
|
||||
|---|---|---|
|
||||
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
|
||||
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
|
||||
|
||||
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
|
||||
Scraping a web root is not a health check: an HTML page fails the scrape
|
||||
protocol (`unsupported Content-Type`), which is what kept those targets red
|
||||
beside healthy services. The classification rule and the honesty law live in
|
||||
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
|
||||
netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
|
||||
the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
|
||||
that gateway lands on the host's loopback listener. The gateway address is
|
||||
not itself bindable from the host netns.
|
||||
Plain targets stay `host:port`, never URLs — the probe job's URL targets are
|
||||
the documented exception, relabeled onto the prober.
|
||||
|
||||
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
|
||||
|
||||
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
|
||||
|
||||
The remedy for a probe-shaped question is the **blackbox exporter**; the
|
||||
classification rule and the honesty law for it live in [[Stale Signal Audit]].
|
||||
Until the exporter is deployed, these three stay red for a structural reason:
|
||||
deleting them would
|
||||
hide a real service, and fixing them by publishing databases or enabling metrics
|
||||
would trade a wrong signal for a real exposure. Targets are `host:port`, never
|
||||
URLs.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path |
|
||||
|
||||
@@ -239,45 +239,92 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
|
||||
sequence stops producing entirely and then re-prefills from token 0, which is a
|
||||
stall, not sludge.
|
||||
|
||||
Knobs, in the order worth trying them (radiance GUIDE §4.5):
|
||||
**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
|
||||
looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
|
||||
the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
|
||||
(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
|
||||
decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
|
||||
with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
|
||||
14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
|
||||
a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
|
||||
worth ~10 % of one step — not a lever.
|
||||
|
||||
- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
|
||||
steps carry decode rows only and admit nothing. Decoders keep their cadence; the
|
||||
prompt pays that in TTFT. Counts as `engine_steps_held_total`.
|
||||
- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
|
||||
`auto` (default) reserves a row plus the draft window per decode row behind it and
|
||||
an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
|
||||
whole step (admission never runs while it does), `N` pins the share. Never under
|
||||
the prefill stager's smallest pass — the startup line names it, and below it every
|
||||
routed layer reads its pooled experts synchronously, which costs the prefill far
|
||||
more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
|
||||
- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
|
||||
curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
|
||||
at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
|
||||
magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
|
||||
expert residency to the arena.
|
||||
- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
|
||||
shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
|
||||
back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
|
||||
at short prompts, a large TTFT gain at the far end of the window.
|
||||
**Four traps in the PromQL above.** Learn them before trusting a bucket:
|
||||
- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
|
||||
histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
|
||||
is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
|
||||
*instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
|
||||
endpoint never stalls anyone".
|
||||
- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
|
||||
`decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
|
||||
scrape in eight — use step ≤ 30 s for a bucket table.
|
||||
- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
|
||||
one shape (`up`, or the first sample of a counter that only the current build exports) before
|
||||
ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
|
||||
turned out to belong entirely to the older one.
|
||||
- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
|
||||
above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
|
||||
|
||||
Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
|
||||
expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
|
||||
decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
|
||||
And **re-check
|
||||
`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
|
||||
pool are the same memory, and the failure looks like turn two taking as long as turn
|
||||
one.
|
||||
Three instruments the PromQL above cannot reach:
|
||||
|
||||
Availability is a version question, not a config question: `--busy-prefill-chunk` and
|
||||
`--prefill-interleave` are absent from older pins — prove it on the running container
|
||||
(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
|
||||
tree, and check the source clone's depth before dating a flag from `git log -S`
|
||||
(a `--depth 1` clone reports its own tip for every search).
|
||||
- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
|
||||
`token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
|
||||
prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
|
||||
the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
|
||||
and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
|
||||
configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
|
||||
- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
|
||||
(linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
|
||||
cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
|
||||
prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
|
||||
prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` -
|
||||
exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner:
|
||||
[[frank-archon]].
|
||||
- **Bucket the arriving request too** — `histogram_quantile` over
|
||||
`vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
|
||||
sequences that are already running; the one waiting for its first token pays the same steps and
|
||||
shows in no decode number.
|
||||
|
||||
Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
|
||||
interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
|
||||
for this fleet's shape are that page's.
|
||||
Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
|
||||
|
||||
- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
|
||||
rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
|
||||
cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
|
||||
Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
|
||||
- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
|
||||
setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
|
||||
of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
|
||||
alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
|
||||
`engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
|
||||
stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
|
||||
`prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
|
||||
every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
|
||||
decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
|
||||
chunk blindly buys latency in both directions.
|
||||
- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
|
||||
carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
|
||||
measurement says a span starts to win), while **every other step is planned at the shared
|
||||
budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
|
||||
knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
|
||||
pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
|
||||
- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
|
||||
arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
|
||||
decode — [[frank-archon]]).
|
||||
|
||||
Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
|
||||
reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
|
||||
quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
|
||||
slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
|
||||
arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
|
||||
taking exactly as long as turn one.
|
||||
|
||||
Availability is a version question, not a config question: `--busy-prefill-chunk`,
|
||||
`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
|
||||
container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
|
||||
`configuration` block, never from a vendored tree, and check a source clone's depth before dating a
|
||||
flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
|
||||
|
||||
Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
|
||||
|
||||
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
|
||||
reserves the registry price, not the actual pool. Raising the pool consumes
|
||||
real free VRAM without growing the arena; the registry over-prices the pool
|
||||
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
|
||||
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
|
||||
block-alignment loss when converting a token target to bytes). The ask lands
|
||||
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
|
||||
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
|
||||
real free VRAM without growing the arena; the registry OVER-PRICES the pool
|
||||
(era G measured ~14 % under the planned B/token — the gap is the lesson, not
|
||||
the number; **never reuse a B/token ratio across geometries**), and there is a
|
||||
~4 MB block-alignment loss to subtract when converting a token target to
|
||||
bytes. The ask lands block-aligned: verify against the engine's own
|
||||
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
|
||||
|
||||
**Driver-reported free VRAM after hours of traffic is NOT allocatable
|
||||
arena.** It is the estimator gap: the registry over-prices KV and charges
|
||||
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
|
||||
**KV accounting — pin the pool BYTES, floor-sized:** set
|
||||
`--kv-cache-memory-bytes` PER RANK to the contract floor
|
||||
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
|
||||
come out of the arena/headroom and buy nothing. Measured B/token on this
|
||||
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
|
||||
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
|
||||
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
|
||||
planner's KV reserve is for ONE max-model-len request.
|
||||
come out of the arena/headroom and buy nothing. The B/token rate is a property
|
||||
of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
|
||||
the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
|
||||
Read your live rate from the boot ledger every time; the decomposition (a
|
||||
dtype-independent mamba/side stream rides on top of the attention KV) is why
|
||||
the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
|
||||
is for ONE max-model-len request.
|
||||
|
||||
**Parallelism ceilings you cannot flag past:**
|
||||
|
||||
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
|
||||
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
|
||||
+ engine flags in the log.
|
||||
|
||||
**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
|
||||
|
||||
| shape | util | graphs | arena | result |
|
||||
|---|---|---|---|---|
|
||||
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
|
||||
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
|
||||
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
|
||||
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
|
||||
|
||||
bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
|
||||
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
|
||||
~207k per context max.
|
||||
|
||||
## 7. Arena audit procedure (when asked for "more experts")
|
||||
|
||||
|
||||
@@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
|
||||
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|
||||
|---|---|
|
||||
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
|
||||
| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
|
||||
| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
|
||||
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
|
||||
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
|
||||
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
|
||||
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
|
||||
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
|
||||
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
|
||||
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
|
||||
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
|
||||
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
|
||||
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
|
||||
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
|
||||
| TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
|
||||
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
|
||||
|
||||
Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
|
||||
the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
|
||||
`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
|
||||
**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
|
||||
2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
|
||||
(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
|
||||
(≤ 4,194,304 px, 16,384 patches/pass).
|
||||
Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
|
||||
the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
|
||||
`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
|
||||
The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
|
||||
sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
|
||||
per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
|
||||
`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
|
||||
host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
|
||||
(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
|
||||
(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
|
||||
directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
|
||||
the bigger cache, paid over PCIe when those experts are hit.
|
||||
|
||||
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
|
||||
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
|
||||
@@ -145,6 +150,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
|
||||
|
||||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||||
|---|---|---|---|---|---|
|
||||
| 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
|
||||
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
|
||||
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
|
||||
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
|
||||
@@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
|
||||
buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
|
||||
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
|
||||
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
|
||||
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
|
||||
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
|
||||
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
|
||||
hand. Three boots of evidence:
|
||||
- **At 1.8.1 the split belongs to the resolver, and the source fixes its order.**
|
||||
`core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena →
|
||||
staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a
|
||||
floor, not a share**, and divides the `elastic` remainder in three lines (146-156):
|
||||
|
||||
| shape | demand | cache given | backing |
|
||||
```cpp
|
||||
exp = std::min(b.expert_floor, b.elastic);
|
||||
kv = std::min(kv_cap, std::max(b.elastic - exp, kv_one));
|
||||
exp = std::min(std::max(b.expert_ceiling, b.expert_floor), b.elastic - kv);
|
||||
```
|
||||
|
||||
`kv_cap` is `KVManager::ceiling()` = `state + Σ_paged max_seqs × ceil(max_ctx / block) × bytes`
|
||||
(`core/mem/kv.cpp:266`), so `--max-num-seqs` and `--max-model-len` set the cache's **demand**, never
|
||||
its funding. The funding is `elastic − expert_floor`, and the floor is
|
||||
`min(plane + spare, plane + spare + 2·unit_max − host_room)` (`core/place/planner.cpp:100-104`, with
|
||||
`host_room = host_pool − file_stage_reserve` at `core/engine_bringup.cpp:1418-1422`). **That is the
|
||||
cache's lever: every MiB added to `--host-pool-mib` drops the expert floor one-for-one and the cache
|
||||
takes it** — minus `stage`, the prefill stager's ring (two layers of the experts left off the card),
|
||||
which is charged back out of the cache whenever the cache lends (`vram_budget.cpp:193-203`). Boots of
|
||||
evidence, all on 1.8.1:
|
||||
|
||||
| shape | demand | cache given | engine's `backs %` |
|
||||
|---|---|---|---|
|
||||
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
|
||||
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
|
||||
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
|
||||
| 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
|
||||
| 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
|
||||
| 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
|
||||
|
||||
So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
|
||||
never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
|
||||
slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
|
||||
grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
|
||||
`--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
|
||||
hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
|
||||
era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
|
||||
the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
|
||||
`--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
|
||||
The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
|
||||
`ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
|
||||
ledger line `kv: the pool backs N % … the cache can grow to … N tokens`, never arithmetic.** To buy
|
||||
window: raise `--host-pool-mib` (cost: pinned host RAM, and experts evicted to host cross PCIe —
|
||||
`experts-from-host` was 5.99 GiB a rank at 44,928 and is 8.99 at 49,152) or cut the per-token cost
|
||||
(`fp8` KV). Raising slots raises demand **and** costs 10 × 111.98 MiB of pinned checkpoint-store RAM
|
||||
per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in
|
||||
the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape:
|
||||
`/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused,
|
||||
and the 400 class is a single request over `--max-model-len`.
|
||||
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
|
||||
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
|
||||
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
|
||||
@@ -280,41 +303,104 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
|
||||
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
|
||||
GiB, refused).
|
||||
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
|
||||
3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
|
||||
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
|
||||
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
|
||||
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
|
||||
decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
|
||||
samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
|
||||
`prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
|
||||
bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
|
||||
0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
|
||||
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
|
||||
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
|
||||
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
|
||||
[[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
|
||||
and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
|
||||
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
|
||||
shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
|
||||
the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
|
||||
step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
|
||||
the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
|
||||
The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
|
||||
knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
|
||||
edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
|
||||
it is trusted.
|
||||
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
|
||||
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
|
||||
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
|
||||
(single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
|
||||
today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
|
||||
floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
|
||||
moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
|
||||
the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
|
||||
mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
|
||||
retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
|
||||
fp8-native start.
|
||||
- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not
|
||||
protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s,
|
||||
`engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` /
|
||||
`prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
|
||||
chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
|
||||
step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the
|
||||
harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s
|
||||
scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light
|
||||
prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two
|
||||
sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
|
||||
else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
|
||||
17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from
|
||||
30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
|
||||
victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
|
||||
and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while
|
||||
median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
|
||||
`vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
|
||||
chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
|
||||
and `busy_cap` is the fourth law.
|
||||
- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves
|
||||
`1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
|
||||
budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
|
||||
"not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
|
||||
the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
|
||||
anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live
|
||||
proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
|
||||
with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies
|
||||
"whenever the chunk shares the step".
|
||||
- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
|
||||
ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
|
||||
`max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode
|
||||
rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
|
||||
prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
|
||||
the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
|
||||
routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
|
||||
pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
|
||||
vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
|
||||
beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
|
||||
- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
|
||||
lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
|
||||
it **synchronously before the step** — units revert to their host copies (an address change, not a
|
||||
copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big
|
||||
step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
|
||||
with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics`
|
||||
says the binding constraint in one line: "a prefill step streams every routed layer's pooled
|
||||
experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
|
||||
Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
|
||||
- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
|
||||
N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
|
||||
costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
|
||||
two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
|
||||
~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
|
||||
measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
|
||||
substitute for it.
|
||||
- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
|
||||
intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
|
||||
intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
|
||||
starts to win), while **every other step is planned at the shared budget of 2048** — which is a
|
||||
lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot
|
||||
express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
|
||||
cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
|
||||
span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
|
||||
second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at
|
||||
75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
|
||||
does not boot at 3 slots; this shape is 2.
|
||||
- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
|
||||
the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
|
||||
replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
|
||||
**linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
|
||||
`core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
|
||||
`log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as
|
||||
deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
|
||||
· … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
|
||||
`prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the
|
||||
cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly
|
||||
**`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs
|
||||
retention only - the writes happen at every interval whatever the policy says (only `--prefill-span`
|
||||
skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
|
||||
alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB
|
||||
against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` -
|
||||
the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
|
||||
Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a
|
||||
chain that keeps resuming at one deep point stops paying for it. Shortening
|
||||
`--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
|
||||
`interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
|
||||
interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
|
||||
- **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance
|
||||
constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store
|
||||
RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over
|
||||
`seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so
|
||||
single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises
|
||||
the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic −
|
||||
expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured
|
||||
2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured
|
||||
2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of
|
||||
expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old
|
||||
"seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
|
||||
specialization, and radiance booted seqs 2 on its first fp8-native start.
|
||||
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
|
||||
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
|
||||
pinned weights, and the doubling is invisible in the weight count. Read the engine log for
|
||||
@@ -396,12 +482,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
|
||||
theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
|
||||
60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
|
||||
software theory.
|
||||
software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo
|
||||
dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
|
||||
between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
|
||||
before crediting any software revert with a fix — and before believing a clean run of the arm
|
||||
you just tested, because the confounder is still in the slot.
|
||||
- **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
|
||||
`Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
|
||||
all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
|
||||
them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
|
||||
decode, `exact` is the bit-exact alternative.
|
||||
- **The link is already at the board's ceiling — do not chase the `link-below-max` warning.**
|
||||
Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
|
||||
the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states.
|
||||
`/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its
|
||||
x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
|
||||
side of the chain is x8 max, which the same endpoint reports two findings later as
|
||||
`link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
|
||||
the only lever on a link-bound prefill is **less traffic over the link** (more expert
|
||||
residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the
|
||||
driver, the two cards reach each other's memory), so the collectives do not hairpin through host
|
||||
RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
|
||||
all-reduce wins the link, the staged experts wait.
|
||||
- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
|
||||
cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre
|
||||
and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured
|
||||
during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU
|
||||
contention is not the current decode culprit**, so do not spend an experiment window on it;
|
||||
re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
|
||||
different from the step-sharing one.
|
||||
- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
|
||||
the disk tier silently never opens. The tier does survive a payload swap when arch and
|
||||
tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
|
||||
@@ -445,11 +554,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
## Contract (the wire)
|
||||
|
||||
- **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
|
||||
`/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
|
||||
JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
|
||||
(`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
|
||||
`engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
|
||||
bare `curl -fsS …/health` gate is honest and needs no body grep.
|
||||
`/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health`
|
||||
answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while
|
||||
stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load
|
||||
(`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses
|
||||
connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep.
|
||||
- **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
|
||||
returns `configuration` (every flag, so drift from the env file is checkable without the host),
|
||||
`environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card
|
||||
`link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings`
|
||||
(level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses
|
||||
PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`,
|
||||
`tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is
|
||||
structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
|
||||
- **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
|
||||
only the alias; container, unit, scripts and env keep the service name). The env's contract
|
||||
line is that **radiance validates the requested model name, so every client — harness, bench
|
||||
@@ -476,12 +593,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
|
||||
— at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
|
||||
- **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
|
||||
longer than the KV pool's token count is accepted then refused as too large to fit; under the
|
||||
3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
|
||||
longer than the KV pool's token count is accepted then refused as too large to fit; a request
|
||||
beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here)
|
||||
and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height
|
||||
sessions, so four conversations can be resident but a fourth near-full-height one can still
|
||||
**preempt a running sequence from the back** to free its blocks
|
||||
(`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then
|
||||
re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`);
|
||||
`radiance:num_requests_preempted` is only the instant depth.
|
||||
- **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
|
||||
Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
|
||||
published-port quirk is same-host only). The boot ledger — the referee for every shape number
|
||||
above — is in the container log (`podman logs frank-archon`)
|
||||
Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
|
||||
netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
|
||||
`radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
|
||||
running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only
|
||||
query misses every latency answer. The step-phase counters are all scraped:
|
||||
`engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`,
|
||||
`engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`,
|
||||
`prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the
|
||||
engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
|
||||
`token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
|
||||
prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
|
||||
`/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape
|
||||
number here — is in the container log (`podman logs frank-archon`);
|
||||
`/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
|
||||
|
||||
## Files
|
||||
|
||||
@@ -10,14 +10,15 @@ tags:
|
||||
- gfx1151
|
||||
- halogen
|
||||
- endpoint
|
||||
timestamp: 2026-10-19
|
||||
timestamp: 2026-10-21
|
||||
---
|
||||
|
||||
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
|
||||
|
||||
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
|
||||
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
|
||||
(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
|
||||
(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
|
||||
This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
|
||||
|
||||
## Current shape
|
||||
|
||||
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
|
||||
unauthenticated token port 8730 stays on container loopback. Engine stdout →
|
||||
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
|
||||
|
||||
- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
|
||||
`HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
|
||||
enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
|
||||
exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
|
||||
- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
|
||||
2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
|
||||
`HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
|
||||
(`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
|
||||
exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
|
||||
conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
|
||||
- Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
|
||||
omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
|
||||
pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
|
||||
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
|
||||
(Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
|
||||
~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
|
||||
Caddy/observability).
|
||||
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
|
||||
(= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
|
||||
`reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
|
||||
rides [[frank-zealot]].
|
||||
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
|
||||
— sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
|
||||
`tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
|
||||
together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
|
||||
`high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
|
||||
- Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
|
||||
pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
|
||||
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
|
||||
@@ -124,14 +128,18 @@ created it — earlier topologies in .
|
||||
prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
|
||||
85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
|
||||
re-measure the warm ratio after any pool change.
|
||||
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
|
||||
155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
|
||||
its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
|
||||
3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
|
||||
over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
|
||||
target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
|
||||
is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
|
||||
under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
|
||||
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
|
||||
clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
|
||||
traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
|
||||
thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
|
||||
window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
|
||||
theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
|
||||
At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
|
||||
200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
|
||||
both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
|
||||
≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
|
||||
`tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
|
||||
from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
|
||||
- **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
|
||||
503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
|
||||
together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
|
||||
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
|
||||
(`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
|
||||
compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
|
||||
return as a crash-loop.
|
||||
2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
|
||||
container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
|
||||
`/usr/lib/x86_64-linux-gnu`, where all three exist.
|
||||
2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
|
||||
distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
|
||||
`libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
|
||||
symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
|
||||
searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
|
||||
2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
|
||||
engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
|
||||
A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
|
||||
3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
|
||||
while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
|
||||
engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**
|
||||
|
||||
@@ -17,6 +17,9 @@ logging {
|
||||
// Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
|
||||
// host network, so it pushes by loopback, never by container name — a root
|
||||
// container is not on the rootless dankclaw_default network.
|
||||
// GRAFANA_HOST_LABEL is replaced with this host's name by
|
||||
// copy_alloy_config at deploy time; it must not appear verbatim in the
|
||||
// installed config.
|
||||
loki.write "default" {
|
||||
endpoint {
|
||||
url = "http://127.0.0.1:3100/loki/api/v1/push"
|
||||
@@ -32,7 +35,34 @@ loki.write "default" {
|
||||
loki.source.journal "journal" {
|
||||
format_as_json = true
|
||||
labels = {
|
||||
job = "systemd-journal",
|
||||
job = "systemd-journal",
|
||||
host = "GRAFANA_HOST_LABEL",
|
||||
}
|
||||
forward_to = [loki.process.relabel.receiver]
|
||||
}
|
||||
|
||||
// Promote the fields that ARE the point of the log stream to real Loki labels.
|
||||
// Without this the journal arrives as one opaque JSON line per entry and every
|
||||
// per-container view is structurally impossible: Loki metrics queries can only
|
||||
// group by INDEXED stream labels, so `sum by (container_name)` returns a single
|
||||
// anonymous series and the logs-overview board read empty beside 6.5k
|
||||
// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
|
||||
// podman's journald logging writes (uppercase); host units lack it and stay on
|
||||
// the host-only stream. `format_as_json = true` above guarantees every line
|
||||
// reaching this stage is a JSON object, so there is no non-JSON branch to
|
||||
// protect — alloy 1.21's json stage has no `on_error` attribute (validated
|
||||
// against the pinned image); a malformed line would drop, which is the
|
||||
// correct fate for a line Loki could not index anyway.
|
||||
loki.process "relabel" {
|
||||
stage.json {
|
||||
expressions = {
|
||||
container = "CONTAINER_NAME",
|
||||
}
|
||||
}
|
||||
stage.labels {
|
||||
values = {
|
||||
container = "container",
|
||||
}
|
||||
}
|
||||
forward_to = [loki.write.default.receiver]
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
# Blackbox exporter modules — the honest answer to "is this HTTP service
|
||||
# answering?". A Prometheus scrape cannot express it (an HTML front page is a
|
||||
# 200 that fails the scrape protocol), which is why the old `services` and
|
||||
# `comfyui` jobs sat permanently red on `unsupported Content-Type`.
|
||||
# See serverAppConfigs-docs/Architecture/Stale Signal Audit.md §1.
|
||||
# The listen address is NOT set here: the quadlet unit's Exec passes
|
||||
# --listen-address (LAN bind — see the unit header for why loopback is
|
||||
# unreachable from the bridge-networked Prometheus). Keep this file to
|
||||
# modules only, so the two sources cannot disagree.
|
||||
|
||||
modules:
|
||||
http_2xx:
|
||||
prober: http
|
||||
timeout: 10s
|
||||
http:
|
||||
preferred_ip_protocol: ip4
|
||||
valid_status_codes: [] # default: 2xx + 3xx
|
||||
@@ -104,14 +104,14 @@
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Memory vs limit (%)",
|
||||
"description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.",
|
||||
"title": "Memory vs host total (%)",
|
||||
"description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
|
||||
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)",
|
||||
"expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
|
||||
"legendFormat": "{{host}} / {{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
|
||||
@@ -18,6 +18,16 @@
|
||||
"includeAll": true,
|
||||
"multi": true,
|
||||
"current": { "text": "All", "value": "$__all" }
|
||||
},
|
||||
{
|
||||
"name": "gpuhost",
|
||||
"label": "gpu host",
|
||||
"type": "query",
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"query": { "query": "label_values(gpu_utilization_percent, host)", "refId": "A" },
|
||||
"includeAll": true,
|
||||
"multi": true,
|
||||
"current": { "text": "All", "value": "$__all" }
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -44,7 +54,7 @@
|
||||
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
|
||||
"targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
|
||||
"targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
|
||||
},
|
||||
{
|
||||
"id": 4, "type": "stat", "title": "Open file handles",
|
||||
@@ -139,6 +149,56 @@
|
||||
{ "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
|
||||
{ "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13, "type": "timeseries", "title": "GPU busy %",
|
||||
"description": "Per-GPU utilization from gpu-exporter (amdgpu gpu_busy_percent / nvidia-smi). The picker above selects the host; each series is one card.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 36 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "expr": "gpu_utilization_percent{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 14, "type": "timeseries", "title": "GPU VRAM used",
|
||||
"description": "Device memory in use. On the R9700s (frank-archon) and the 3090 (frank-zealot) this is the loaded model's footprint.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 36 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "expr": "gpu_memory_used_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 15, "type": "timeseries", "title": "GPU VRAM used %",
|
||||
"description": "Used / total device memory. Near 100% is the pre-OOM wall.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 44 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "expr": "100 * gpu_memory_used_bytes{host=~\"$gpuhost\"} / gpu_memory_total_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 16, "type": "timeseries", "title": "GPU temperature",
|
||||
"description": "Package temperature. AMD edge temp (hwmon temp1_input); NVIDIA gpu temp.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 44 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "celsius", "min": 0 }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "expr": "gpu_temperature_celsius{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 17, "type": "timeseries", "title": "GPU power draw",
|
||||
"description": "Board power draw. Emitted only where readable: NVIDIA and the Dankclaw iGPU. The R9700s have no sysfs power sensor, so they are simply absent here — never a fabricated 0.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 52 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "watt", "min": 0 }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "expr": "gpu_power_watts{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -194,7 +194,7 @@
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Firing fleet alerts",
|
||||
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
|
||||
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
|
||||
@@ -2,74 +2,242 @@
|
||||
"uid": "logs-overview",
|
||||
"title": "Logs \u2014 Loki (systemd journal)",
|
||||
"description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
|
||||
"tags": ["logs", "loki"],
|
||||
"tags": [
|
||||
"logs",
|
||||
"loki"
|
||||
],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"refresh": "30s",
|
||||
"time": { "from": "now-1h", "to": "now" },
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "container",
|
||||
"label": "container",
|
||||
"type": "query",
|
||||
"datasource": { "type": "loki", "uid": "loki" },
|
||||
"query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"query": {
|
||||
"query": "label_values({job=\"systemd-journal\"}, container)",
|
||||
"refId": "LokiVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"includeAll": true,
|
||||
"multi": true,
|
||||
"refresh": 2,
|
||||
"current": { "text": "All", "value": "$__all" }
|
||||
"current": {
|
||||
"text": "All",
|
||||
"value": "$__all"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
|
||||
"description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
|
||||
"gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
|
||||
"datasource": { "type": "loki", "uid": "loki" },
|
||||
"fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
|
||||
"id": 1,
|
||||
"type": "timeseries",
|
||||
"title": "Log volume by container (lines/s)",
|
||||
"description": "Per-container journal line rate. The `container` label is promoted at ingestion by Alloy (loki.process) from podman's CONTAINER_NAME journald field - Loki metrics can only group by indexed stream labels.",
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"custom": {
|
||||
"fillOpacity": 20,
|
||||
"stacking": {
|
||||
"mode": "normal"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} [$__auto]))",
|
||||
"legendFormat": "{{container_name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2, "type": "timeseries", "title": "Total log rate",
|
||||
"id": 2,
|
||||
"type": "timeseries",
|
||||
"title": "Total log rate",
|
||||
"description": "All journal lines per second.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
|
||||
"datasource": { "type": "loki", "uid": "loki" },
|
||||
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 9
|
||||
},
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))",
|
||||
"legendFormat": "all lines/s",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3, "type": "timeseries", "title": "Error-level line rate",
|
||||
"id": 3,
|
||||
"type": "timeseries",
|
||||
"title": "Error-ish line rate (text match)",
|
||||
"description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
|
||||
"datasource": { "type": "loki", "uid": "loki" },
|
||||
"fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 9
|
||||
},
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "red"
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))",
|
||||
"legendFormat": "error lines/s",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4, "type": "timeseries", "title": "Errors by container",
|
||||
"id": 4,
|
||||
"type": "timeseries",
|
||||
"title": "Errors by container",
|
||||
"description": "Which unit is throwing. Rank by the top offenders.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
|
||||
"datasource": { "type": "loki", "uid": "loki" },
|
||||
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 17
|
||||
},
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} |~ \"(?i)(error|panic|fatal)\" [$__auto]))",
|
||||
"legendFormat": "{{container_name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5, "type": "stat", "title": "Error lines (last 15m)",
|
||||
"id": 5,
|
||||
"type": "stat",
|
||||
"title": "Error lines (last 15m)",
|
||||
"description": "Absolute count over the window, not a rate.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
|
||||
"datasource": { "type": "loki", "uid": "loki" },
|
||||
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 17
|
||||
},
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 50
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 500
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "errors/15m",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6, "type": "logs", "title": "Journal (filtered)",
|
||||
"id": 6,
|
||||
"type": "logs",
|
||||
"title": "Journal (filtered)",
|
||||
"description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
|
||||
"gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
|
||||
"datasource": { "type": "loki", "uid": "loki" },
|
||||
"options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
|
||||
"targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 25
|
||||
},
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"wrapLogMessage": true,
|
||||
"sortOrder": "Descending",
|
||||
"enableLogDetails": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "{job=\"systemd-journal\", container=~\"$container\"}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -2,95 +2,384 @@
|
||||
"uid": "observability",
|
||||
"title": "Observability \u2014 self-monitoring",
|
||||
"description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
|
||||
"tags": ["observability", "meta"],
|
||||
"tags": [
|
||||
"observability",
|
||||
"meta"
|
||||
],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"refresh": "30s",
|
||||
"time": { "from": "now-6h", "to": "now" },
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1, "type": "stat", "title": "Targets up",
|
||||
"gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
|
||||
"targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
|
||||
"id": 1,
|
||||
"type": "stat",
|
||||
"title": "Targets up",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "count(up == 1)",
|
||||
"instant": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2, "type": "stat", "title": "Targets down",
|
||||
"description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
|
||||
"gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
|
||||
"targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
|
||||
"id": 2,
|
||||
"type": "stat",
|
||||
"title": "Targets down",
|
||||
"description": "Every scrape target that is down. ZERO IS THE HEALTHY STATE: the HTML-probe jobs and the disabled Gitea metrics were retired 2026-10-21 (HTTP liveness now rides the blackbox exporter, job \"services\"). A non-zero count here is a real regression - read /api/v1/targets lastError before triaging.",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 4,
|
||||
"y": 0
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "count(up == 0)",
|
||||
"instant": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3, "type": "stat", "title": "TSDB head series",
|
||||
"id": 3,
|
||||
"type": "stat",
|
||||
"title": "TSDB head series",
|
||||
"description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
|
||||
"gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
|
||||
"targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 5,
|
||||
"x": 8,
|
||||
"y": 0
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "prometheus_tsdb_head_series",
|
||||
"instant": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4, "type": "stat", "title": "Discovered targets",
|
||||
"gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
|
||||
"id": 4,
|
||||
"type": "stat",
|
||||
"title": "Discovered targets",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 5,
|
||||
"x": 13,
|
||||
"y": 0
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(prometheus_sd_discovered_targets)",
|
||||
"instant": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5, "type": "stat", "title": "Prometheus mem",
|
||||
"gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
|
||||
"targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
|
||||
"id": 5,
|
||||
"type": "stat",
|
||||
"title": "Prometheus mem",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "process_resident_memory_bytes{job=\"prometheus\"}",
|
||||
"instant": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6, "type": "stat", "title": "Grafana up",
|
||||
"gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
|
||||
"targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
|
||||
"id": 6,
|
||||
"type": "stat",
|
||||
"title": "Grafana up",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 21,
|
||||
"y": 0
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"mappings": [
|
||||
{
|
||||
"type": "value",
|
||||
"options": {
|
||||
"1": {
|
||||
"text": "UP",
|
||||
"color": "green"
|
||||
},
|
||||
"0": {
|
||||
"text": "DOWN",
|
||||
"color": "red"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "up{job=\"grafana\"}",
|
||||
"instant": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7, "type": "table", "title": "Target health",
|
||||
"id": 7,
|
||||
"type": "table",
|
||||
"title": "Target health",
|
||||
"description": "Every scrape target and its current up. Sort by Value to surface regressions.",
|
||||
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
|
||||
"targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"options": {
|
||||
"sortBy": [
|
||||
{
|
||||
"displayName": "Value",
|
||||
"sortOrder": "Asc"
|
||||
}
|
||||
]
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "up",
|
||||
"format": "table",
|
||||
"instant": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
|
||||
"id": 8,
|
||||
"type": "timeseries",
|
||||
"title": "Scrape interval drift (p99 actual)",
|
||||
"description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
|
||||
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
|
||||
"targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 4
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}",
|
||||
"legendFormat": "{{instance}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9, "type": "timeseries", "title": "Scrape samples/s",
|
||||
"id": 9,
|
||||
"type": "timeseries",
|
||||
"title": "Scrape samples/s",
|
||||
"description": "Total samples ingested across all jobs.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 13
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(scrape_samples_scraped[5m]))",
|
||||
"legendFormat": "samples/s",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 10, "type": "timeseries", "title": "Loki request rate",
|
||||
"id": 10,
|
||||
"type": "timeseries",
|
||||
"title": "Loki request rate",
|
||||
"description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 13
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))",
|
||||
"legendFormat": "{{route}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 11, "type": "timeseries", "title": "Exporter process CPU",
|
||||
"id": 11,
|
||||
"type": "timeseries",
|
||||
"title": "Exporter process CPU",
|
||||
"description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
|
||||
"targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 21
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))",
|
||||
"legendFormat": "{{job}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -60,20 +60,33 @@ scrape_configs:
|
||||
- targets: ["loki:3100"]
|
||||
metrics_path: "/metrics"
|
||||
|
||||
# ComfyUI (no native /metrics; probe the UI)
|
||||
- job_name: "comfyui"
|
||||
metrics_path: "/"
|
||||
static_configs:
|
||||
- targets: ["comfyui:8188"]
|
||||
|
||||
# Blackbox-style probes for service up/down (published LAN ports;
|
||||
# gitea and comfyui are covered by their own jobs above)
|
||||
# HTTP liveness, probed honestly through the blackbox exporter. Scraping a
|
||||
# web root is NOT a health check — an HTML front page fails the scrape
|
||||
# protocol (`unsupported Content-Type`), which is exactly why the old
|
||||
# `services` and `comfyui` jobs were red forever beside healthy services
|
||||
# (Stale Signal Audit §1). The prober is a host-netns container listening on
|
||||
# the podman gateway (the scrape address below - the netavark hairpin gap
|
||||
# bars this container from its own host's LAN IP, cadvisor job note) and on
|
||||
# the LAN; from that netns it reaches the host LAN IP, and published ports
|
||||
# via 127.0.0.1 - bridge container names do NOT resolve in the host netns,
|
||||
# which is why comfyui's probe is a loopback URL. comfyui's own
|
||||
# metrics-scrape job is retired, one owner per service.
|
||||
- job_name: "services"
|
||||
metrics_path: "/"
|
||||
metrics_path: /probe
|
||||
params:
|
||||
module: [http_2xx]
|
||||
static_configs:
|
||||
- targets:
|
||||
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
||||
- "192.168.0.143:3005" # OpenWebUI
|
||||
- "http://192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
||||
- "http://192.168.0.143:3005" # OpenWebUI
|
||||
- "http://127.0.0.1:8188" # ComfyUI UI (published 0.0.0.0:8188; probed from the prober's host netns)
|
||||
relabel_configs:
|
||||
- source_labels: [__address__]
|
||||
target_label: __param_target
|
||||
- source_labels: [__param_target]
|
||||
target_label: instance
|
||||
- target_label: __address__
|
||||
replacement: "host.containers.internal:9115"
|
||||
|
||||
# LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
|
||||
# this container, not the guarded vhosts: these are machine clients and
|
||||
@@ -140,12 +153,15 @@ scrape_configs:
|
||||
# system.slice daemons and machine.slice scopes carry no such name and are
|
||||
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
|
||||
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
|
||||
# the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
|
||||
# is why node-dankclaw scrapes via host.containers.internal), a published port
|
||||
# IS reachable from this bridge container at the host LAN address.
|
||||
# the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
|
||||
# DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
|
||||
# connection whether the port is a host-net bind or a published port (the
|
||||
# original note here claimed published ports were exempt; measured 2026-10-21
|
||||
# from inside this container: LAN refused, gateway answers). Cross-host
|
||||
# targets (dankserver) have no hairpin problem and stay plain LAN.
|
||||
- job_name: "cadvisor-dankclaw"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.143:8080"]
|
||||
- targets: ["host.containers.internal:8080"]
|
||||
labels:
|
||||
host: dankclaw
|
||||
metric_relabel_configs:
|
||||
@@ -191,3 +207,25 @@ scrape_configs:
|
||||
action: keep
|
||||
- regex: 'id|name|image|container_label.*|machine_id|boot_id'
|
||||
action: labeldrop
|
||||
|
||||
# Per-GPU metrics (gpu_utilization_percent / gpu_memory_used_bytes /
|
||||
# gpu_memory_total_bytes / gpu_temperature_celsius / gpu_power_watts) from the
|
||||
# host-process gpu-exporter (scripts/gpu-exporter.py). One job per host, same
|
||||
# reach rules as node/cadvisor: Dankclaw is scraped at the podman gateway
|
||||
# because the netavark hairpin quirk makes Prometheus fail to dial its own
|
||||
# host LAN IP; Dankserver is plain LAN. The gpu label (PCI BDF on AMD, index on
|
||||
# NVIDIA) + vendor/model come from the exporter, so the GPU row on the
|
||||
# host-system dashboard filters by host. Power is emitted only where readable
|
||||
# (NVIDIA + the Dankclaw iGPU); the R9700s have no sysfs power, so they simply
|
||||
# have no gpu_power_watts series — no fabricated zero.
|
||||
- job_name: "gpu-dankclaw"
|
||||
static_configs:
|
||||
- targets: ["host.containers.internal:9401"]
|
||||
labels:
|
||||
host: dankclaw
|
||||
|
||||
- job_name: "gpu-dankserver"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.235:9401"]
|
||||
labels:
|
||||
host: dankserver
|
||||
+12
-1
@@ -193,7 +193,18 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
|
||||
}
|
||||
|
||||
@test "prometheus targets are host:port (no URL schemes)" {
|
||||
! grep -nE '^\s*-\s*"?http' "$REPO_ROOT/shared/prometheus/prometheus.yml"
|
||||
# Plain scrape targets are host:port; a URL target is only meaningful for
|
||||
# the blackbox prober job, whose relabel_configs copy it to __param_target
|
||||
# and point __address__ at the prober. Anywhere else a URL target is the
|
||||
# old HTML-probe defect wearing a new hat.
|
||||
awk '
|
||||
/job_name:/ { job=$0; sub(/.*"/, "", job); sub(/".*/, "", job); probe = 0 }
|
||||
/metrics_path: \/probe/ { probe = 1 }
|
||||
/^[[:space:]]*-[[:space:]]*"http/ && !probe {
|
||||
print FILENAME ":" FNR ": URL target outside a prober job: " $0; bad = 1
|
||||
}
|
||||
END { exit bad }
|
||||
' "$REPO_ROOT/shared/prometheus/prometheus.yml"
|
||||
}
|
||||
|
||||
@test "observability containers run as container root (rootless = host user)" {
|
||||
|
||||
@@ -125,3 +125,48 @@ load helpers
|
||||
[ ! -e "$sandbox/frank-default-sightsound" ]
|
||||
[ -e "$BATS_TEST_TMPDIR/.AGENTTRASH/$(date +%Y%m%d)-retired-deploy-state/frank-default-sightsound" ]
|
||||
}
|
||||
|
||||
@test "the client and remote allowlists agree per target" {
|
||||
# Two gates guard `deploy.sh --service X`: the client's valid_service and
|
||||
# the remote runner's service_is_allowed. When only one side gained a
|
||||
# service the other answered "is not deployed on <target>" and the only
|
||||
# repair path was a fleet-wide drain — cadvisor and gpu-exporter both fell
|
||||
# through this crack within a day of each other. The lists MUST agree;
|
||||
# searxng-network is the one sanctioned exception (client-only network
|
||||
# unit, no remote dispatch handler).
|
||||
run env OUT="$BATS_TEST_TMPDIR" bash -c '
|
||||
grep -oE "dank(claw|server):[a-z0-9-]+" \
|
||||
<(sed -n "/^valid_service()/,/^}/p" "$1/scripts/deploy.sh") \
|
||||
| grep -v "dankclaw:searxng-network" | sort -u > "$OUT/cli"
|
||||
grep -oE "dank(claw|server):[a-z0-9-]+" \
|
||||
<(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
|
||||
| sort -u > "$OUT/rem"
|
||||
diff "$OUT/cli" "$OUT/rem"
|
||||
' _ "$REPO_ROOT"
|
||||
[ "$status" -eq 0 ] || { echo "deploy.sh vs deploy-remote.sh allowlist drift:"; echo "$output"; return 1; }
|
||||
# Drift aside, the gate must actually know the host-process exporters —
|
||||
# their first-ever deploy has to pass the client gate.
|
||||
for pair in dankclaw:cadvisor dankclaw:gpu-exporter dankserver:cadvisor dankserver:gpu-exporter; do
|
||||
grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
|
||||
done
|
||||
}
|
||||
|
||||
@test "every gated dankclaw service is dispatched" {
|
||||
# A service can clear both allowlists and still die inside the deployer:
|
||||
# dankclaw has no compose file, and deploy_compose_service (the case
|
||||
# catch-all) hard-fails with "no quadlet handler" since 2026-09-16. So
|
||||
# every name the remote gate allows for dankclaw MUST be named by a
|
||||
# deploy_service pattern — blackbox-exporter (2026-10-21) shipped with
|
||||
# both gates and the health gate wired but no dispatch case, and only
|
||||
# the deploy attempt found it.
|
||||
run env OUT="$BATS_TEST_TMPDIR" bash -c '
|
||||
grep -oE "dankclaw:[a-z0-9-]+" \
|
||||
<(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
|
||||
| cut -d: -f2 | sort -u > "$OUT/gated"
|
||||
sed -n "/^deploy_service()/,/^}/p" "$1/scripts/deploy-remote.sh" \
|
||||
| grep -oE "^ +[a-z][a-z0-9|-]*\)" | tr -d " )" | tr "|" "\n" \
|
||||
| sort -u > "$OUT/dispatched"
|
||||
comm -23 "$OUT/gated" "$OUT/dispatched"
|
||||
' _ "$REPO_ROOT"
|
||||
[ -z "$output" ] || { echo "gated dankclaw services with no deploy_service case:"; echo "$output"; return 1; }
|
||||
}
|
||||
@@ -145,7 +145,7 @@ nosudo_bin() {
|
||||
# (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
|
||||
# seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
|
||||
# tied inside the +-3% noise floor — slots buy concurrency, not speed.
|
||||
grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
|
||||
@@ -158,7 +158,7 @@ nosudo_bin() {
|
||||
# so the pool's token count is a boot-ledger line — verified live by the
|
||||
# capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
|
||||
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=49152$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
|
||||
# Prefix cache: pinned host RAM only (the disk tier was removed), and the
|
||||
@@ -177,8 +177,11 @@ nosudo_bin() {
|
||||
[ "$ttmt" -gt "$ttmm" ]
|
||||
pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
|
||||
ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE")
|
||||
# 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
|
||||
ledger=$(( 2 * pool + 48824 + ph + 2048 ))
|
||||
# 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host + MNBT + the auto
|
||||
# checkpoint store (MAX_SEQS x 10 slots of 112 MiB -- the store is pinned host
|
||||
# memory too, and it grows with the slot count).
|
||||
seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
|
||||
ledger=$(( 2 * pool + 48824 + ph + 2048 + seqs * 10 * 112 ))
|
||||
[ "$ttmm" -ge "$ledger" ]
|
||||
# The .rad size floor is positive (a truncated convert must fail fast).
|
||||
grep -q '^FRANK_ARCHON_RAD_MIN_MIB=' "$ENV_FILE"
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
|
||||
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
|
||||
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE
|
||||
# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
|
||||
# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
|
||||
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
|
||||
# shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
|
||||
#
|
||||
@@ -85,7 +85,7 @@ good_env() {
|
||||
"FRANK_DRAGOON_VISION_BYTES=0"
|
||||
"FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
|
||||
"HALOGEN_MODEL_ID=frank-dragoon"
|
||||
"HALOGEN_CTX=200704"
|
||||
"HALOGEN_CTX=262144"
|
||||
"HALOGEN_KV_POOL_POSITIONS=401408"
|
||||
"HALOGEN_KV_SLOTS=2"
|
||||
"HALOGEN_SPEC_ADAPT=0"
|
||||
@@ -159,9 +159,11 @@ good_env() {
|
||||
reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
# The pool is ONE budget behind the slots (boot ledger form: "2 slots over
|
||||
# ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
|
||||
# a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
|
||||
# (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
|
||||
# resident at once — frank-zealot parity on total in-flight context. What
|
||||
# a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
|
||||
# native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
|
||||
# MAXIMUM harness-length (200,000-token) conversations are resident at once;
|
||||
# two literal ceiling-length requests would queue — widening that is a pool
|
||||
# raise, not a window move.
|
||||
# still binds: the pool must back ONE full-length request (below that the
|
||||
# advertised per-request ceiling would be unreachable, hence pool >= ctx),
|
||||
# it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
|
||||
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
[[ "$output" == *"--name frank-dragoon-halogen"* ]]
|
||||
[[ "$output" == *"--publish 7770:8731"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
|
||||
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
|
||||
@test "the shipped env keeps the harness fit law inside the ceiling" {
|
||||
# HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
|
||||
# naming the limit, never a silent truncation). The law is INVARIANT to
|
||||
# the value: the harness window must sit ~45k UNDER it, because the
|
||||
# harness clamps max_tokens = min(45056, window − est − 4096), compacts
|
||||
# at 85 % of its declared window, and its chars/4 est UNDER-counts the
|
||||
# rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
|
||||
# est 152,938 on 2026-10-07 — with window == ceiling, every request over
|
||||
# ~65 % of the window was a guaranteed 400 before compaction could fire).
|
||||
# The shipped pair is ctx 200704 / window 155648 (lowered from
|
||||
# 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
|
||||
# monotone maxima of est + budget (est ≤ window − 4096):
|
||||
# full budget at the compaction threshold: 1.03*132300 + 45056
|
||||
# clamped budget at the clamp's zero point: 1.03*151552 + 0
|
||||
# and the compaction request that rides on frank-zealot (its
|
||||
# 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
|
||||
grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
|
||||
ctx=200704
|
||||
hw=155648; mt=45056
|
||||
thr=$(( hw * 85 / 100 ))
|
||||
# naming the limit, never a silent truncation). The law: the harness
|
||||
# window must sit ~45k UNDER the ceiling, because the harness clamps
|
||||
# max_tokens = min(45056, window − est − 4096), compacts at the trigger
|
||||
# percent of its declared window, and its chars/4 est UNDER-counts the
|
||||
# rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
|
||||
# at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
|
||||
# request is a guaranteed 400 before compaction can fire). The 2026-10-21
|
||||
# pair: ceiling 262144 (native) under a harness window of 200,000.
|
||||
# The declared window is read from the endpoint page — the one owner —
|
||||
# so env, models.yml and page that disagree FAIL HERE, not as a 400
|
||||
# mid-session (the recurring failure this seam exists for).
|
||||
grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
|
||||
ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
|
||||
grep -qE 'contextWindow: 200000' "$page"
|
||||
hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
|
||||
mt=45056
|
||||
# Invariant checked at the live 95 % trigger (config.yml
|
||||
# thresholdPercent) — strictly stronger than the 85 % design point.
|
||||
# Both monotone maxima of est + budget (est ≤ window − 4096):
|
||||
thr=$(( hw * 95 / 100 ))
|
||||
[ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
|
||||
zero=$(( hw - 4096 ))
|
||||
[ $(( zero * 103 / 100 )) -le "$ctx" ]
|
||||
# The compaction request rides on frank-zealot's 200,704-token slot:
|
||||
# input ≈ threshold − keepRecent(20k).
|
||||
[ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
|
||||
}
|
||||
|
||||
@@ -423,9 +430,12 @@ npu_fixture() {
|
||||
# the stock root:render 0660 mode, and the serve gate checks exactly that.
|
||||
touch "$TEST_TMP/accel0"
|
||||
chmod 666 "$TEST_TMP/accel0"
|
||||
mkdir -p "$TEST_TMP/xrt"
|
||||
# basename carries a multiarch shape: the third mount prefix is derived
|
||||
# from the XRT dir's basename (the NPU child searches
|
||||
# /opt/xilinx/lib/<multiarch>/, measured 2026-10-21).
|
||||
mkdir -p "$TEST_TMP/x86_64-linux-gnu"
|
||||
for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
|
||||
touch "$TEST_TMP/xrt/$l"
|
||||
touch "$TEST_TMP/x86_64-linux-gnu/$l"
|
||||
done
|
||||
mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
|
||||
mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
|
||||
@@ -435,7 +445,7 @@ npu_fixture() {
|
||||
"HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
|
||||
"FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
|
||||
"FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
|
||||
"FRANK_DRAGOON_XRT_DIR=$TEST_TMP/xrt"
|
||||
"FRANK_DRAGOON_XRT_DIR=$TEST_TMP/x86_64-linux-gnu"
|
||||
"FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
|
||||
)
|
||||
}
|
||||
@@ -465,11 +475,15 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
! grep -q 'accel0' "$TEST_TMP/serve.log"
|
||||
}
|
||||
|
||||
@test "the lane arms the accel node, the staged model dir and each XRT library twice" {
|
||||
# Each library mounts at the image's XRT path AND at its own path: the
|
||||
# distro libraries find each other where they were installed, and an
|
||||
# /opt/xilinx/xrt of symlinks into the system dir dangles inside the
|
||||
# container (the vendor calls that out by name).
|
||||
@test "the lane arms the accel node, the staged model dir and each XRT library three times" {
|
||||
# Each library mounts at the image's XRT path, at its own path (the distro
|
||||
# libraries find each other where they were installed), and at
|
||||
# /opt/xilinx/lib/<multiarch>/ — the prefix the NPU child searches and
|
||||
# names when it is missing (`No such library
|
||||
# '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'`, measured
|
||||
# 2026-10-21: the lane died and the supervisor took the whole container
|
||||
# with it, 88-restart loop). An /opt/xilinx/xrt of symlinks into the
|
||||
# system dir dangles inside the container (the vendor calls that out).
|
||||
podman_stub
|
||||
good_env
|
||||
npu_fixture
|
||||
@@ -484,8 +498,9 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
[[ "$output" != *":/models/npu"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
|
||||
for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
|
||||
[[ "$output" == *"-v $TEST_TMP/xrt/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
|
||||
[[ "$output" == *"-v $TEST_TMP/xrt/$l:$TEST_TMP/xrt/$l:ro"* ]]
|
||||
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
|
||||
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:$TEST_TMP/x86_64-linux-gnu/$l:ro"* ]]
|
||||
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/lib/x86_64-linux-gnu/$l:ro"* ]]
|
||||
done
|
||||
# The Flash payload still rides the same container: the lane is additive.
|
||||
[[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
|
||||
@@ -560,7 +575,7 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
podman_stub
|
||||
good_env
|
||||
npu_fixture
|
||||
rm "$TEST_TMP/xrt/libxrt_driver_xdna.so.2"
|
||||
rm "$TEST_TMP/x86_64-linux-gnu/libxrt_driver_xdna.so.2"
|
||||
run_with_log
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]
|
||||
|
||||
@@ -77,6 +77,7 @@ FRANKBOOK_UNITS=(
|
||||
}
|
||||
|
||||
@test "readarr keeps its pinned image" {
|
||||
grep -q '^Image=lscr.io/linuxserver/readarr:amd64-1.0.980$' \
|
||||
# 2026-10-21 fleet law: version tag AND digest (tests/quadlet_image_pins.bats)
|
||||
grep -qE '^Image=lscr\.io/linuxserver/readarr:amd64-1\.0\.980@sha256:[0-9a-f]{64}$' \
|
||||
"$REPO_ROOT/frankbook-pro/readarr/quadlet/readarr.container" || return 1
|
||||
}
|
||||
Executable
+227
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the GPU metrics exporter's black-box contract.
|
||||
#
|
||||
# The exporter reads amdgpu sysfs (one card per /sys/class/drm/cardN that has a
|
||||
# device/gpu_busy_percent file) and, on the box that has a working NVIDIA stack,
|
||||
# one `nvidia-smi --query-gpu` CSV, and emits the unified `gpu_*` Prometheus
|
||||
# families the host-system dashboard graphs. Both sources are injectable so the
|
||||
# whole thing is testable with fixtures on a GPU-less dev box.
|
||||
#
|
||||
# Only the inputs a consumer sees are pinned: the metric families, their label
|
||||
# set, and that absent hardware yields ABSENT series (never a fabricated 0 — a
|
||||
# wrong number on a dashboard gets believed). The R9700s expose no sysfs
|
||||
# power1_average, so those cards legitimately carry no gpu_power_watts line.
|
||||
|
||||
load helpers
|
||||
|
||||
EXPORTER="$REPO_ROOT/scripts/gpu-exporter.py"
|
||||
|
||||
# Build a fixture amdgpu sysfs tree. $BATS_TEST_TMPDIR/sys
|
||||
mk_amd() { # gpu busy vram_used vram_total power_uw_or_blank temp_mc
|
||||
local card=$1 busy=$2 used=$3 total=$4 power=$5 temp=$6
|
||||
local d="$SYS/card$card/device"
|
||||
mkdir -p "$d/hwmon/hwmon9"
|
||||
printf '%s\n' "$busy" > "$d/gpu_busy_percent"
|
||||
printf '%s\n' "$used" > "$d/mem_info_vram_used"
|
||||
printf '%s\n' "$total" > "$d/mem_info_vram_total"
|
||||
printf '%s\n' "$power" > "$d/power1_average" # blank => absent on R9700
|
||||
[ -n "$temp" ] && printf '%s\n' "$temp" > "$d/hwmon/hwmon9/temp1_input"
|
||||
printf 'DRIVER=amdgpu\nPCI_ID=1002:7551\nPCI_SLOT_NAME=0000:03:00.0\n' > "$d/uevent"
|
||||
}
|
||||
|
||||
setup() {
|
||||
SYS="$BATS_TEST_TMPDIR/sys"
|
||||
mkdir -p "$SYS"
|
||||
}
|
||||
|
||||
@test "exporter exists and compiles" {
|
||||
[ -f "$EXPORTER" ]
|
||||
python3 -m py_compile "$EXPORTER"
|
||||
}
|
||||
|
||||
@test "amd card with full sysfs yields util/vram/temp/power with the right labels and units" {
|
||||
mk_amd 1 42 1073741824 34359738368 250000000 71000 # 250 W, 71 C
|
||||
run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
|
||||
[ "$status" -eq 0 ] || { echo "$output"; return 1; }
|
||||
# util is a raw percent, VRAM verbatim bytes, temp milli->C, power microW->W.
|
||||
echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 42$' || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 1073741824$' || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_memory_total_bytes\{[^}]*\} 34359738368$' || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*\} 71$' || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*\} 250$' || { echo "$output"; return 1; }
|
||||
# identity labels: the PCI slot is a stable unique id, vendor marks the family.
|
||||
echo "$output" | grep -Eq 'gpu_utilization_percent\{.*gpu="0000:03:00\.0"' || { echo "$output"; return 1; }
|
||||
}
|
||||
|
||||
@test "card with an empty power file and no hwmon yields util/vram but omits power and temp" {
|
||||
mk_amd 1 99 34108338176 34208743424 "" "" # the R9700 shape
|
||||
run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
|
||||
[ "$status" -eq 0 ] || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*\} 99$' || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 34108338176$'
|
||||
# Absent means ABSENT, not a zero the dashboard would plot as a real reading.
|
||||
! echo "$output" | grep -Eq '^gpu_power_watts\{' || { echo "fabricated power line: $output"; return 1; }
|
||||
! echo "$output" | grep -Eq '^gpu_temperature_celsius\{' || { echo "fabricated temp line: $output"; return 1; }
|
||||
}
|
||||
|
||||
@test "nvidia csv yields util/vram/temp/power tagged vendor=nvidia" {
|
||||
nvidia="$BATS_TEST_TMPDIR/nvidia-smi"
|
||||
cat > "$nvidia" <<'STUB'
|
||||
echo '0, NVIDIA GeForce RTX 3090, 93, 23004, 24576, 76, 397.40'
|
||||
STUB
|
||||
chmod +x "$nvidia"
|
||||
run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$nvidia"
|
||||
[ "$status" -eq 0 ] || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="nvidia"[^}]*\} 93$' || { echo "$output"; return 1; }
|
||||
# memory.used MiB -> bytes (23004 * 1048576)
|
||||
echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*vendor="nvidia"[^}]*\} 24121442304$' || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*vendor="nvidia"[^}]*\} 76$'
|
||||
echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*vendor="nvidia"[^}]*\} 397\.4$'
|
||||
echo "$output" | grep -Eq 'model="NVIDIA GeForce RTX 3090"'
|
||||
}
|
||||
|
||||
@test "a failing nvidia command is tolerated: amd still emits and exit is 0" {
|
||||
mk_amd 1 50 1000000 2000000 "" 40000
|
||||
failing="$BATS_TEST_TMPDIR/nvidia-fail"
|
||||
printf '#!/bin/sh\nexit 1\n' > "$failing"; chmod +x "$failing"
|
||||
run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$failing"
|
||||
[ "$status" -eq 0 ] || { echo "$output"; return 1; }
|
||||
echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 50$'
|
||||
! echo "$output" | grep -q 'vendor="nvidia"'
|
||||
}
|
||||
|
||||
@test "no GPUs at all yields only HELP/TYPE headers and exits 0" {
|
||||
run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
|
||||
[ "$status" -eq 0 ] || { echo "$output"; return 1; }
|
||||
# families are declared even when empty, so the dashboard/queries stay valid.
|
||||
echo "$output" | grep -q '# TYPE gpu_utilization_percent gauge'
|
||||
echo "$output" | grep -q '# TYPE gpu_memory_used_bytes gauge'
|
||||
! echo "$output" | grep -Eq '^gpu_utilization_percent\{'
|
||||
}
|
||||
|
||||
@test "output is valid Prometheus text" {
|
||||
mk_amd 1 42 1000 2000 "" 50000
|
||||
run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
|
||||
[ "$status" -eq 0 ]
|
||||
# every non-comment line is "<name>{labels} value" or "<name> value"
|
||||
printf '%s\n' "$output" | python3 -c '
|
||||
import sys, re
|
||||
pat = re.compile(r"^[a-z_]+(\{[^}]*\})? [-+0-9.eE+]+$")
|
||||
for i, l in enumerate(sys.stdin):
|
||||
l = l.rstrip("\n")
|
||||
if not l or l.startswith("#"):
|
||||
continue
|
||||
assert pat.match(l), "malformed line %d: %r" % (i, l)
|
||||
print("prometheus text ok")
|
||||
'
|
||||
}
|
||||
|
||||
# ---- deployment + wiring contracts -------------------------------------
|
||||
# The exporter is a HOST-PROCESS user unit (not a quadlet): it must read the
|
||||
# host's own DRM sysfs and shell to nvidia-smi, which a container could not do
|
||||
# without GPU passthrough. These tests pin the cross-file wiring a reader
|
||||
# cannot eyeball — a lone change on one side leaves the metric silently absent.
|
||||
|
||||
@test "the base unit runs the exporter from the checkout and never binds a wildcard" {
|
||||
local u="$REPO_ROOT/scripts/gpu-exporter.service"
|
||||
[ -f "$u" ]
|
||||
grep -q '^ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve$' "$u" \
|
||||
|| { echo "unit does not run the exporter --serve from the checkout"; return 1; }
|
||||
grep -q '^Restart=always$' "$u"
|
||||
grep -q '^WantedBy=default.target$' "$u"
|
||||
# Base unit binds loopback only; the per-host LAN address is added by a
|
||||
# deploy drop-in, so a checked-in unit is never a wildcard.
|
||||
grep -q '^Environment=GPU_EXPORTER_BINDS=127.0.0.1$' "$u"
|
||||
! grep -qE '0\.0\.0\.0' "$u"
|
||||
}
|
||||
|
||||
@test "the deployer installs, drops in the LAN bind, and health-gates gpu-exporter" {
|
||||
local handler="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
grep -q '^deploy_gpu_exporter() {' "$handler"
|
||||
# Per-host LAN from $TARGET (never a hardcoded host, never a wildcard).
|
||||
grep -q 'dankclaw) lan=192.168.0.143' "$handler"
|
||||
grep -q 'dankserver) lan=192.168.0.235' "$handler"
|
||||
# Drop-in binds loopback + this host's LAN as one comma list.
|
||||
grep -q "GPU_EXPORTER_BINDS=127.0.0.1,%s" "$handler"
|
||||
grep -q 'systemctl --user enable gpu-exporter.service' "$handler"
|
||||
grep -q 'systemctl --user restart gpu-exporter.service' "$handler"
|
||||
# Health gate waits on the loopback metrics endpoint.
|
||||
grep -q 'http://127.0.0.1:9401/metrics' "$handler"
|
||||
# Dispatched from the central case.
|
||||
grep -q 'gpu-exporter) deploy_gpu_exporter ;;' "$handler"
|
||||
# Allowlisted on BOTH hosts.
|
||||
grep -q 'dankclaw:gpu-exporter' "$handler"
|
||||
grep -q 'dankserver:gpu-exporter' "$handler"
|
||||
}
|
||||
|
||||
@test "the planner selects gpu-exporter for its files on each target" {
|
||||
local t f
|
||||
for t in dankclaw dankserver; do
|
||||
for f in scripts/gpu-exporter.py scripts/gpu-exporter.service; do
|
||||
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target "$t" --paths "$f"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "gpu-exporter" ] || { echo "$t/$f -> '$output'"; return 1; }
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
@test "Prometheus scrapes gpu-exporter where node/cadvisor are reachable on each host" {
|
||||
# Cross-file contract, same posture as node-exporter: the local instance is
|
||||
# reached via the podman gateway (this Prometheus is a bridge container and
|
||||
# a host-process bind is refused at the host LAN IP); the Dankserver
|
||||
# instance is a different machine, so plain LAN routing reaches it. If either
|
||||
# address drifts from the exporter's bind, GPU panels go blank with no error.
|
||||
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
|
||||
grep -q 'job_name: "gpu-dankclaw"' "$cfg" || { echo "no gpu-dankclaw job"; return 1; }
|
||||
grep -q 'job_name: "gpu-dankserver"' "$cfg" || { echo "no gpu-dankserver job"; return 1; }
|
||||
grep -q '"host.containers.internal:9401"' "$cfg" \
|
||||
|| { echo "gpu-dankclaw does not scrape the podman gateway"; return 1; }
|
||||
grep -q '"192.168.0.235:9401"' "$cfg" \
|
||||
|| { echo "gpu-dankserver does not scrape the LAN address"; return 1; }
|
||||
# A vhost target would be red forever (Prometheus cannot solve Anubis).
|
||||
! grep -qE '"[^"]*franksdank[^"]*:9401"' "$cfg"
|
||||
}
|
||||
|
||||
@test "the host-system dashboard graphs the gpu_* families the exporter emits" {
|
||||
local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/host-system.json"
|
||||
[ -f "$dash" ]
|
||||
python3 -c "
|
||||
import json
|
||||
d = json.load(open('$dash'))
|
||||
exprs = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
|
||||
for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
|
||||
'gpu_temperature_celsius', 'gpu_power_watts'):
|
||||
assert any(want in e for e in exprs), ('missing', want, exprs)
|
||||
print('gpu dashboard panels ok')
|
||||
"
|
||||
}
|
||||
|
||||
@test "--serve answers on EVERY declared bind, not just the first" {
|
||||
# The deployed unit serves loopback + the host LAN address, and Prometheus
|
||||
# reaches the host through the gateway — which dials the LAN-class bind.
|
||||
# A server that only runs serve_forever() on the first socket leaves the
|
||||
# second one listen()ing-but-never-accepting: the deploy health gate on
|
||||
# loopback passes, every scrape on the other bind dies on the SYN backlog,
|
||||
# and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
|
||||
# left both gpu-* targets down beside a green deploy.)
|
||||
mk_amd 0 42 1073741824 34359738368 "" 71000
|
||||
port=$((19400 + RANDOM % 500))
|
||||
python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
|
||||
--bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
|
||||
> "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
|
||||
local pid=$!
|
||||
trap 'kill "$pid" 2>/dev/null' EXIT
|
||||
local ok=0
|
||||
for _ in $(seq 1 20); do
|
||||
curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
|
||||
sleep 0.25
|
||||
done
|
||||
[ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
|
||||
# Both binds must ANSWER within 2 s each — the second one is the bug.
|
||||
for b in 127.0.0.1 127.0.0.2; do
|
||||
curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
|
||||
|| { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
|
||||
done
|
||||
kill "$pid" 2>/dev/null
|
||||
trap - EXIT
|
||||
}
|
||||
@@ -21,7 +21,7 @@ load helpers
|
||||
|
||||
UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager)
|
||||
UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager blackbox-exporter)
|
||||
|
||||
# container_section <unit> — the [Container] key/value block only
|
||||
container_section() {
|
||||
@@ -33,7 +33,7 @@ service_section() {
|
||||
awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
|
||||
}
|
||||
|
||||
@test "all seven observability units exist under dankclaw/observability/quadlet/" {
|
||||
@test "all eight observability units exist under dankclaw/observability/quadlet/" {
|
||||
local svc
|
||||
for svc in "${UNITS[@]}"; do
|
||||
[ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
|
||||
@@ -559,6 +559,64 @@ assert any('ALERTS' in e for e in exprs), exprs
|
||||
grep -q -- '--dport 18093 -j DROP' "$handler"
|
||||
}
|
||||
|
||||
@test "the deployer installs the relay firewall with the ACCEPTs deciding before the DROP" {
|
||||
# Behavioral, run for real against a sandbox that plays the kernel: the
|
||||
# rule set is a LIST and iptables -I prepends, so the shape this function
|
||||
# had until 2026-10-21 (-C/-I ACCEPT x3, then -C/-I the blanket DROP)
|
||||
# landed the DROP at position 1 and every packet died there — the drain
|
||||
# failed its health gate beside a LISTENing socket twice. Seeding the
|
||||
# sandbox with that poisoned order also pins the convergent repair: a
|
||||
# re-deploy must rewrite the tail, not skip because -C finds every rule.
|
||||
local sandbox="$BATS_TEST_TMPDIR/fw"
|
||||
mkdir -p "$sandbox/bin"
|
||||
cat > "$sandbox/bin/iptables" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
# ordered rule list in $FW_STATE: -C lookup, -I at head, -A at tail, -D delete.
|
||||
state="${FW_STATE:?}"
|
||||
touch "$state"
|
||||
op="$1"; shift
|
||||
case "$op" in
|
||||
-C) grep -Fxq -- "$*" "$state" ;;
|
||||
-I) grep -Fxq -- "$*" "$state" || { { echo "$*"; cat "$state"; } > "$state.new"; mv "$state.new" "$state"; } ;;
|
||||
-A) grep -Fxq -- "$*" "$state" || echo "$*" >> "$state" ;;
|
||||
-D) if grep -Fxq -- "$*" "$state"; then
|
||||
grep -Fxv -- "$*" "$state" > "$state.new"; mv "$state.new" "$state"
|
||||
else exit 1; fi ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
EOF
|
||||
chmod +x "$sandbox/bin/iptables"
|
||||
# Poisoned state exactly as the pre-fix deployer left Dankclaw.
|
||||
printf '%s\n' \
|
||||
'INPUT -p tcp --dport 18093 -j DROP' \
|
||||
'INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT' \
|
||||
'INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT' \
|
||||
'INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT' \
|
||||
> "$sandbox/rules"
|
||||
(
|
||||
cd "$REPO_ROOT"
|
||||
export PATH="$sandbox/bin:$PATH" FW_STATE="$sandbox/rules" HOME="$sandbox/home"
|
||||
mkdir -p "$HOME"
|
||||
eval "$(sed -n '/^install_signal_relay()/,/^}/p' scripts/deploy-remote.sh)"
|
||||
run_privileged() { "$@"; }
|
||||
systemctl() { :; }
|
||||
curl() { return 0; }
|
||||
sleep() { :; }
|
||||
install_signal_relay
|
||||
)
|
||||
# Exactly one blanket DROP, and it sits after every ACCEPT.
|
||||
[ "$(grep -cF -- 'INPUT -p tcp --dport 18093 -j DROP' "$sandbox/rules")" -eq 1 ]
|
||||
local drop_at total
|
||||
drop_at="$(grep -nF -- 'INPUT -p tcp --dport 18093 -j DROP' "$sandbox/rules" | cut -d: -f1)"
|
||||
total="$(wc -l < "$sandbox/rules")"
|
||||
[ "$drop_at" -eq "$total" ]
|
||||
local s seen
|
||||
for s in 127.0.0.1 192.168.0.143 10.89.0.0/16; do
|
||||
seen="$(grep -nF -- "-s $s -j ACCEPT" "$sandbox/rules" | cut -d: -f1)"
|
||||
[ -n "$seen" ] && [ "$seen" -lt "$drop_at" ]
|
||||
done
|
||||
}
|
||||
|
||||
@test "the planner selects alertmanager for its config and relay files" {
|
||||
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
|
||||
--paths shared/prometheus/alertmanager.yml
|
||||
@@ -595,16 +653,22 @@ assert any('ALERTS' in e for e in exprs), exprs
|
||||
}
|
||||
|
||||
@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
|
||||
# Cross-file contract: cadvisor publishes the LAN port; a cadvisor scrape
|
||||
# job pointing anywhere else reads nothing (unlike node-exporter's host-net
|
||||
# bind, a published port IS reachable from this bridge container at the LAN
|
||||
# address). And the whole dashboard depends on the id -> container relabel:
|
||||
# cadvisor names nothing itself, the quadlet unit name is only in the cgroup
|
||||
# path. If either half changes alone, the Containers dashboard is blank.
|
||||
# Cross-file contract, measured 2026-10-21 from inside the Prometheus
|
||||
# container: a bridge container cannot dial its OWN host's LAN IP — the
|
||||
# hairpin refuses published ports exactly like host-net binds, so
|
||||
# cadvisor-dankclaw must scrape via the podman gateway (the original pin
|
||||
# here demanded the LAN address and shipped a permanently-down target).
|
||||
# dankserver is a different machine: plain LAN there. And the whole
|
||||
# dashboard depends on the id -> container relabel: cadvisor names nothing
|
||||
# itself, the quadlet unit name is only in the cgroup path. If either half
|
||||
# changes alone, the Containers dashboard is blank.
|
||||
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
|
||||
grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
|
||||
grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
|
||||
grep -q '"192.168.0.143:8080"' "$cfg" || { echo "cadvisor-dankclaw does not scrape the published LAN port"; return 1; }
|
||||
grep -q '"host.containers.internal:8080"' "$cfg" \
|
||||
|| { echo "cadvisor-dankclaw does not scrape via the podman gateway"; return 1; }
|
||||
! grep -q '"192.168.0.143:8080"' "$cfg" \
|
||||
|| { echo "cadvisor-dankclaw dials its own host LAN IP — hairpin refuses it"; return 1; }
|
||||
grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
|
||||
# The relabel that turns …/app.slice/<name>.service… into a `container`.
|
||||
grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
|
||||
@@ -638,3 +702,125 @@ assert not any('container_network' in e or 'container_fs_' in e for e in names),
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "cadvisor" ]
|
||||
}
|
||||
|
||||
@test "the grafana copier provisions EVERY dashboard json, not a hand-listed subset" {
|
||||
# Behavioral, not a text grep: the copier is extracted and executed with a
|
||||
# stubbed run_privileged against a sandbox. The fleet provider turns each
|
||||
# file in its directory into a dashboard, so a file the copier skips is a
|
||||
# dashboard that silently never exists (the containers board lived in the
|
||||
# repo, never reached /opt, and its fixes were undeployable).
|
||||
# BATS_TEST_TMPDIR: per-test, removed by bats itself; a mktemp dir with a
|
||||
# RETURN trap races bats' own trap machinery and vanishes mid-function.
|
||||
local sandbox="$BATS_TEST_TMPDIR/sandbox"
|
||||
(
|
||||
cd "$REPO_ROOT"
|
||||
eval "$(sed -n '/^copy_grafana_provisioning()/,/^}/p' scripts/deploy-remote.sh)"
|
||||
run_privileged() { "$@"; }
|
||||
export GRAFANA_PROV_ROOT="$sandbox/opt/grafana"
|
||||
copy_grafana_provisioning
|
||||
)
|
||||
local f
|
||||
for f in "$REPO_ROOT"/shared/grafana/provisioning/dashboards/*.json; do
|
||||
[ -f "$sandbox/opt/grafana/provisioning/dashboards/$(basename "$f")" ]
|
||||
done
|
||||
[ -f "$sandbox/opt/grafana/provisioning/dashboards/provider.yml" ]
|
||||
[ -f "$sandbox/opt/grafana/provisioning/datasources/prometheus.yml" ]
|
||||
}
|
||||
|
||||
@test "every observability unit declares an Image" {
|
||||
# Quadlet refuses to GENERATE a service for a .container without Image=
|
||||
# - the deployer then dies at `systemctl restart` with "Unit not found"
|
||||
# and no diagnosis (blackbox-exporter, live 2026-10-21: merged with the
|
||||
# copier, gates and health check wired, but no Image line).
|
||||
local svc
|
||||
for svc in "${UNITS[@]}"; do
|
||||
grep -q '^Image=' "$UNIT_DIR/$svc.container" \
|
||||
|| { echo "$svc.container has no Image= - quadlet will not generate it"; return 1; }
|
||||
done
|
||||
}
|
||||
|
||||
@test "changes under shared/blackbox/ plan the prober" {
|
||||
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
|
||||
--paths shared/blackbox/blackbox.yml
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "blackbox-exporter" ]
|
||||
}
|
||||
|
||||
@test "the copier runs the same code path the deployer runs" {
|
||||
# Textually pinned so a refactor cannot quietly route the deployer around
|
||||
# the copy_* functions the seams execute.
|
||||
grep -q ' prometheus)' "$REMOTE_DEPLOY"
|
||||
grep -q 'copy_prometheus_config' "$REMOTE_DEPLOY"
|
||||
grep -q 'copy_alertmanager_config' "$REMOTE_DEPLOY"
|
||||
grep -q 'copy_loki_config' "$REMOTE_DEPLOY"
|
||||
grep -q ' blackbox-exporter)' "$REMOTE_DEPLOY"
|
||||
grep -q 'copy_blackbox_config' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the alloy copier installs the host-labeled config, token-free" {
|
||||
# Executes copy_alloy_config for real against a sandbox (stubbed
|
||||
# run_privileged). A surviving GRAFANA_HOST_LABEL token would ship a
|
||||
# config whose host label is a literal placeholder — the exact class of
|
||||
# silent-label defect this tranche retired the board for.
|
||||
local sandbox="$BATS_TEST_TMPDIR/opt"
|
||||
(
|
||||
cd "$REPO_ROOT"
|
||||
eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
|
||||
run_privileged() { "$@"; }
|
||||
# NOTE: VAR=x func does not bind for shell functions — export.
|
||||
export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
|
||||
copy_alloy_config
|
||||
)
|
||||
[ -f "$sandbox/config/config.alloy" ]
|
||||
grep -q 'host = "dankclaw",' "$sandbox/config/config.alloy"
|
||||
! grep -q "GRAFANA_HOST_LABEL" "$sandbox/config/config.alloy"
|
||||
}
|
||||
|
||||
@test "the blackbox copier installs the prober config" {
|
||||
local sandbox="$BATS_TEST_TMPDIR/blackbox"
|
||||
(
|
||||
cd "$REPO_ROOT"
|
||||
eval "$(sed -n '/^copy_blackbox_config()/,/^}/p' scripts/deploy-remote.sh)"
|
||||
run_privileged() { "$@"; }
|
||||
export BLACKBOX_CONF_ROOT="$sandbox"
|
||||
copy_blackbox_config
|
||||
)
|
||||
[ -f "$sandbox/blackbox.yml" ]
|
||||
grep -q '^ http_2xx:' "$sandbox/blackbox.yml"
|
||||
}
|
||||
|
||||
@test "the alloy copier refuses to install a config the engine rejects" {
|
||||
# The syntax gate: alloy attributes are position-sensitive and a
|
||||
# misplaced one only explodes at run time (crash-loop behind a green
|
||||
# deploy). Prove the copier refuses a config the pinned image won't
|
||||
# validate, and installs nothing. Needs the alloy image; skipped
|
||||
# where it is absent (the copier's gate is likewise image-gated).
|
||||
if ! podman image exists localhost/franksdank/alloy:latest; then
|
||||
skip "franksdank/alloy image not present on this box"
|
||||
fi
|
||||
local sandbox="$BATS_TEST_TMPDIR/opt" cfg="$BATS_TEST_TMPDIR/broken.alloy"
|
||||
sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
|
||||
printf 'bogus_attr = "x"\n' >> "$cfg"
|
||||
! (
|
||||
cd "$REPO_ROOT"
|
||||
eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
|
||||
run_privileged() { "$@"; }
|
||||
cp() { if [ "$1" = "$cfg" ]; then return 1; fi; command cp "$@"; }
|
||||
export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
|
||||
copy_alloy_config
|
||||
)
|
||||
[ ! -f "$sandbox/config/config.alloy" ]
|
||||
}
|
||||
|
||||
@test "the installed alloy config is engine-valid" {
|
||||
# Runs the pinned image's own validator over the shipped config. The
|
||||
# 2026-10-21 defect (`on_error` on the labels stage, which alloy 1.21
|
||||
# does not define) passed every textual seam and only failed at boot.
|
||||
if ! podman image exists localhost/franksdank/alloy:latest; then
|
||||
skip "franksdank/alloy image not present on this box"
|
||||
fi
|
||||
local cfg="$BATS_TEST_TMPDIR/rendered.alloy"
|
||||
sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
|
||||
podman run --rm -v "$cfg:/c.alloy:ro" --entrypoint /usr/bin/alloy \
|
||||
localhost/franksdank/alloy:latest validate /c.alloy
|
||||
}
|
||||
@@ -15,8 +15,9 @@ QDIR="$REPO_ROOT/dankclaw/open-webui/quadlet"
|
||||
}
|
||||
|
||||
@test "open-webui uses the CPU image, not :cuda (dankclaw is CPU-only)" {
|
||||
grep -qE '^Image=ghcr\.io/open-webui/open-webui:main$' "$QDIR/open-webui.container" \
|
||||
|| { echo "open-webui.container must use the :main (CPU) image"; return 1; }
|
||||
# 2026-10-21 fleet law: tag AND digest (tests/quadlet_image_pins.bats)
|
||||
grep -qE '^Image=ghcr\.io/open-webui/open-webui:main@sha256:[0-9a-f]{64}$' "$QDIR/open-webui.container" \
|
||||
|| { echo "open-webui.container must use the :main (CPU) image, digest-pinned"; return 1; }
|
||||
if grep -qE '^Image=.*:cuda' "$QDIR/open-webui.container"; then
|
||||
echo "open-webui.container must not use the :cuda image on a CPU-only host"
|
||||
return 1
|
||||
|
||||
Executable
+47
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: fleet-wide quadlet laws learned the hard way.
|
||||
# 1. Every registry image is digest-pinned. A rolling tag (":latest",
|
||||
# ":9-alpine", ":nightly") lets a rebuild move under a running
|
||||
# service at the next restart/reboot with nobody approving it, and it
|
||||
# makes a CVE audit ungradeable (you cannot score a tag). Locally
|
||||
# built images (localhost/*, franksdank/*) are exempt: their build
|
||||
# IS the pin, and they never pull.
|
||||
# 2. No unit carries a key the podman 5.7.0 quadlet generator rejects.
|
||||
# The generator aborts the WHOLE batch on an unsupported key, so one
|
||||
# bad file silently un-generates every other unit in that manager -
|
||||
# live 2026-10-21: HostAddr= in prometheus.container orphaned
|
||||
# prometheus.service while its container kept running, and the next
|
||||
# deploy died on "Unit prometheus.service not found".
|
||||
|
||||
load helpers
|
||||
|
||||
@test "every registry Image= is digest-pinned across all three hosts" {
|
||||
bad=""
|
||||
while IFS= read -r unit; do
|
||||
while IFS= read -r img; do
|
||||
case "$img" in
|
||||
localhost/*|franksdank/*) continue ;;
|
||||
esac
|
||||
case "$img" in
|
||||
*@sha256:*) ;;
|
||||
*) bad="$bad$unit: $img"$'\n' ;;
|
||||
esac
|
||||
done < <(sed -n 's/^Image=//p' "$unit")
|
||||
done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
|
||||
-path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
|
||||
[ -z "$bad" ] || { printf 'unpinned images:\n%s' "$bad" >&2; return 1; }
|
||||
}
|
||||
|
||||
@test "no unit uses a key the 5.7.0 quadlet generator rejects (batch killer)" {
|
||||
# HostAddr rejected by podman 5.7.0 (live 2026-10-21); Device= and
|
||||
# Command= are the older v1 findings, guarded host-locally too.
|
||||
# Keep this list current with every generator finding.
|
||||
bad=""
|
||||
while IFS= read -r unit; do
|
||||
for key in HostAddr Device Command; do
|
||||
grep -q "^${key}=" "$unit" && bad="$bad$unit: ${key}="$'\n'
|
||||
done
|
||||
done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
|
||||
-path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
|
||||
[ -z "$bad" ] || { printf 'generator-rejected keys:\n%s' "$bad" >&2; return 1; }
|
||||
}
|
||||
Executable
+527
@@ -0,0 +1,527 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the Qwen3.8-Flash-Next five-bit campaign (feat/qwen38-w5nl64).
|
||||
#
|
||||
# The campaign converts the vendor BF16 checkpoint into a w5nl64a8h .rad on
|
||||
# dankserver while frank-archon keeps serving. Every gate that protects the box
|
||||
# or the result lives in a script under scripts/, and each gate is tested here
|
||||
# through that script's own exit status and printed output — no engine, no GPU,
|
||||
# no network. Fixtures are synthetic; the expected numbers are worked examples,
|
||||
# not values recomputed the way the script computes them.
|
||||
#
|
||||
# Seams under test (the campaign's public boundaries):
|
||||
# scripts/download-qwen38-bf16.sh fetch/verify of the BF16 checkpoint
|
||||
# scripts/grams-to-imatrix.py calibration recording -> imatrix GGUF
|
||||
# scripts/derive-protected-experts.py --target-count sweep + the parity law
|
||||
# scripts/build-qwen38-bf16-rad.sh the reference container's convert argv
|
||||
# scripts/run-qwen38-bf16.sh the drained-serve argv, mode selection
|
||||
# scripts/convert-qwen38-w5nl64.sh the arm driver's refusal gates
|
||||
# scripts/kld-gate-qwen38.sh the three-anchor interval law
|
||||
# scripts/deploy-plan.py campaign files must deploy nothing
|
||||
|
||||
load helpers
|
||||
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-qwen38-bf16.sh"
|
||||
PLANNER="$REPO_ROOT/scripts/deploy-plan.py"
|
||||
|
||||
# make_shard <dir> <name> <total-bytes> — a REAL safetensors file whose size on
|
||||
# disk is exactly <total-bytes>: 8-byte header length, JSON header, tensor data.
|
||||
# Real, because the verifier reads each shard's own header to learn how many
|
||||
# bytes it claims to be, which is the only per-file truth available offline.
|
||||
make_shard() {
|
||||
mkdir -p "$1"
|
||||
python3 - "$1/$2" "$3" <<'PY'
|
||||
import json, struct, sys
|
||||
path, want = sys.argv[1], int(sys.argv[2])
|
||||
payload = max(0, want - 96)
|
||||
header = b""
|
||||
for _ in range(16):
|
||||
# The header is part of the file, so its own length moves the target: solve
|
||||
# payload = want - 8 - len(header(payload)) by fixed-point iteration.
|
||||
header = json.dumps({"w": {"dtype": "F32", "shape": [payload // 4],
|
||||
"data_offsets": [0, payload]}},
|
||||
separators=(",", ":")).encode()
|
||||
want_payload = want - 8 - len(header)
|
||||
if want_payload == payload:
|
||||
break
|
||||
if want_payload < 0:
|
||||
raise SystemExit("fixture too small to carry a header")
|
||||
payload = want_payload
|
||||
else:
|
||||
raise SystemExit("fixture header never converged")
|
||||
with open(path, "wb") as f:
|
||||
f.write(struct.pack("<Q", len(header)) + header + b"\x00" * payload)
|
||||
PY
|
||||
}
|
||||
|
||||
# make_index <dir> <shard>... — a safetensors index that references them
|
||||
make_index() {
|
||||
local dir="$1"; shift
|
||||
python3 - "$dir/model.safetensors.index.json" "$@" <<'PY'
|
||||
import json, sys
|
||||
out, names = sys.argv[1], sys.argv[2:]
|
||||
json.dump({"metadata": {"total_size": 0},
|
||||
"weight_map": {f"model.layers.{i}.attn.q.weight": n
|
||||
for i, n in enumerate(names)}},
|
||||
open(out, "w"))
|
||||
PY
|
||||
}
|
||||
|
||||
# three_fixture <dir> — three valid 1000-byte shards plus the index naming them
|
||||
three_fixture() {
|
||||
make_shard "$1" model-00001-of-00003.safetensors 1000
|
||||
make_shard "$1" model-00002-of-00003.safetensors 1000
|
||||
make_shard "$1" model-00003-of-00003.safetensors 1000
|
||||
make_index "$1" model-00001-of-00003.safetensors \
|
||||
model-00002-of-00003.safetensors \
|
||||
model-00003-of-00003.safetensors
|
||||
}
|
||||
|
||||
# --- download / verify -------------------------------------------------------
|
||||
|
||||
@test "the checkpoint verifier accepts a complete checkpoint and reports its footprint" {
|
||||
C="$BATS_TEST_TMPDIR/complete"
|
||||
three_fixture "$C"
|
||||
# 3 x 1000 bytes of shards. The index file itself is deliberately not
|
||||
# counted, so a Hub-side edit to a config file cannot fail this gate.
|
||||
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"VERIFY_OK"*"3 shards"*"3000 bytes"* ]]
|
||||
}
|
||||
|
||||
@test "the verifier names the shard whose own header disagrees with its size on disk" {
|
||||
C="$BATS_TEST_TMPDIR/truncated"
|
||||
three_fixture "$C"
|
||||
# Cut the tail, keep the header: that is a torn transfer, which is what the
|
||||
# box actually gets from an interrupted fetch. Zero-filling instead would
|
||||
# destroy the header and be caught as "not a safetensors file" — a different
|
||||
# gate, and it would prove nothing about the size check.
|
||||
python3 -c 'import sys
|
||||
p = sys.argv[1]
|
||||
keep = open(p, "rb").read(400)
|
||||
open(p, "wb").write(keep)' "$C/model-00002-of-00003.safetensors"
|
||||
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"TRUNCATED"*"model-00002-of-00003.safetensors"* ]]
|
||||
}
|
||||
|
||||
@test "the verifier refuses a missing shard and names it" {
|
||||
C="$BATS_TEST_TMPDIR/missing"
|
||||
three_fixture "$C"
|
||||
mv "$C/model-00003-of-00003.safetensors" "$C/hidden.bin"
|
||||
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"MISSING"*"model-00003-of-00003.safetensors"* ]]
|
||||
}
|
||||
|
||||
@test "the verifier refuses when the shard census disagrees with the expected count" {
|
||||
# The fragment law this repo learned the hard way: large quants ship as
|
||||
# .partNofM fragments and a naive glob reports success on a partial tree.
|
||||
C="$BATS_TEST_TMPDIR/count"
|
||||
three_fixture "$C"
|
||||
make_shard "$C" model-00004-of-00004.safetensors 1000
|
||||
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"SHARD_COUNT"*"4"*"3"* ]]
|
||||
}
|
||||
|
||||
@test "the verifier refuses a shard the index names that never landed" {
|
||||
C="$BATS_TEST_TMPDIR/orphan-index"
|
||||
three_fixture "$C"
|
||||
rm "$C/model-00003-of-00003.safetensors"
|
||||
make_index "$C" model-00001-of-00003.safetensors \
|
||||
model-00002-of-00003.safetensors \
|
||||
model-00003-of-00003.safetensors
|
||||
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 2 --bytes 2000
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"MISSING"*"model-00003-of-00003.safetensors"* ]]
|
||||
}
|
||||
|
||||
@test "the verifier treats an HTTP error body written as a shard as a failure, not a download" {
|
||||
# curl -o FILE exits 0 on 401/403 and writes the error body; three shards
|
||||
# of identical tiny size must never look like a checkpoint.
|
||||
C="$BATS_TEST_TMPDIR/errorbody"
|
||||
mkdir -p "$C"
|
||||
for n in 1 2 3; do
|
||||
printf '{"error":"You are not logged in"}' \
|
||||
> "$C/model-0000${n}-of-00003.safetensors"
|
||||
done
|
||||
make_index "$C" model-00001-of-00003.safetensors \
|
||||
model-00002-of-00003.safetensors \
|
||||
model-00003-of-00003.safetensors
|
||||
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"NOT_SAFETENSORS"* ]]
|
||||
}
|
||||
|
||||
@test "the verifier refuses a shard census whose bytes do not total what the Hub declared" {
|
||||
# Every shard header can be self-consistent and the tree still be the wrong
|
||||
# payload: the total is the cross-check against the Hub's declared size.
|
||||
C="$BATS_TEST_TMPDIR/wrongtotal"
|
||||
three_fixture "$C"
|
||||
run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 2500
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"TOTAL_BYTES"*"3000"*"2500"* ]]
|
||||
}
|
||||
|
||||
@test "the fetch path refuses when the destination cannot hold the checkpoint" {
|
||||
# The campaign's peak-disk law starts at the fetch: the BF16 checkpoint and
|
||||
# the reference container built from it are both ~335 GiB, and the tree that
|
||||
# does not fit is discovered hours into a transfer.
|
||||
C="$BATS_TEST_TMPDIR/tiny"
|
||||
three_fixture "$C"
|
||||
run env FRANK_QWEN38_MIN_FREE_MIB=999999999 \
|
||||
bash "$DOWNLOAD" --fetch --no-hub --dir "$C" --shards 3 --bytes 3000
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"*"free"* ]]
|
||||
}
|
||||
|
||||
# --- deployment scoping ------------------------------------------------------
|
||||
|
||||
@test "the campaign tooling deploys nothing: merging it cannot drain frank-archon" {
|
||||
# Existence is asserted first: a planner that plans nothing because the
|
||||
# files vanished would otherwise pass.
|
||||
[ -f "$REPO_ROOT/scripts/download-qwen38-bf16.sh" ]
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
scripts/download-qwen38-bf16.sh \
|
||||
scripts/build-qwen38-bf16-rad.sh \
|
||||
scripts/run-qwen38-bf16.sh \
|
||||
scripts/convert-qwen38-w5nl64.sh \
|
||||
scripts/kld-gate-qwen38.sh \
|
||||
scripts/grams-to-imatrix.py \
|
||||
scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe \
|
||||
tests/qwen38_campaign.bats
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
# --- protected experts -------------------------------------------------------
|
||||
|
||||
DERIVE="$REPO_ROOT/scripts/derive-protected-experts.py"
|
||||
APPLY="$REPO_ROOT/scripts/apply-protected-cast.sh"
|
||||
CONVERT="$REPO_ROOT/scripts/convert-qwen38-w5nl64.sh"
|
||||
GATE="$REPO_ROOT/scripts/kld-gate-qwen38.sh"
|
||||
TEMPLATE="$REPO_ROOT/scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe"
|
||||
|
||||
# make_gram <dir> <layer> <expert> <energy> <rows> — a RADGRAM2 down-Gram whose
|
||||
# trace (its down-input energy) is <energy> and whose routing count is <rows>.
|
||||
make_gram() {
|
||||
python3 - "$@" <<'PY'
|
||||
import os, struct, sys
|
||||
d, layer, e, energy, rows = sys.argv[1:6]
|
||||
os.makedirs(d, exist_ok=True)
|
||||
hdr = struct.pack("<8sIIQI4x", b"RADGRAM2", 2, 0, int(rows), 0)
|
||||
mat = struct.pack("<4f", float(energy), 0.0, 0.0, 0.0)
|
||||
with open(os.path.join(d, f"gram.r0.blk.{layer}.ffn_down_exps.{e}.bin"), "wb") as f:
|
||||
f.write(hdr + mat)
|
||||
PY
|
||||
}
|
||||
|
||||
# eight_grams <dir> — one layer, eight experts inside rank half 0, energies
|
||||
# 60/15/10/8/4/2/1/0.5 against equal routing. Under the default bars (3% of the
|
||||
# layer's energy AND 3x its routing share) exactly ONE expert is disproportionate,
|
||||
# so the parity law has to hand it a partner.
|
||||
eight_grams() {
|
||||
local d="$1"
|
||||
make_gram "$d" 0 0 60 100
|
||||
make_gram "$d" 0 1 15 100
|
||||
make_gram "$d" 0 2 10 100
|
||||
make_gram "$d" 0 3 8 100
|
||||
make_gram "$d" 0 4 4 100
|
||||
make_gram "$d" 0 5 2 100
|
||||
make_gram "$d" 0 6 1 100
|
||||
make_gram "$d" 0 7 0.5 100
|
||||
}
|
||||
|
||||
cast_count() { grep -c 'cast dtype=bf16' <<< "$output"; }
|
||||
|
||||
@test "the bar-only pass keeps the disproportionate expert and repairs the odd rank half" {
|
||||
C="$BATS_TEST_TMPDIR/calib"
|
||||
eight_grams "$C"
|
||||
run python3 "$DERIVE" "$C"
|
||||
[ "$status" -eq 0 ]
|
||||
# e0 holds 59.7% of the layer's energy on 12.5% of the routing: the only expert
|
||||
# that clears three times its routing share. e1 joins it because half 0 was odd.
|
||||
[ "$(cast_count)" -eq 2 ]
|
||||
[[ "$output" == *"blk.0.ffn_*_exps.0.weight"* ]]
|
||||
[[ "$output" == *"blk.0.ffn_*_exps.1.weight"* ]]
|
||||
[[ "$output" != *"ffn_*_exps.2.weight"* ]]
|
||||
[[ "$output" == *"half 0 odd -> added e1"* ]]
|
||||
}
|
||||
|
||||
@test "--target-count lands the byte bill that was asked for, and never buys more" {
|
||||
# The count is a disk and VRAM decision (each protected expert costs bf16 twice,
|
||||
# once per rank half's copy), so landing it matters as much as not missing it.
|
||||
C="$BATS_TEST_TMPDIR/calib"
|
||||
eight_grams "$C"
|
||||
run python3 "$DERIVE" "$C" --target-count 6
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(cast_count)" -eq 6 ]
|
||||
[[ "$output" == *"count fill -> added e2"* ]]
|
||||
[[ "$output" != *"ffn_*_exps.6.weight"* ]]
|
||||
# the energy share the set bottoms out at is the bar the campaign paid for
|
||||
[[ "$output" == *"total protected: 6 (target 6, energy cutoff"* ]]
|
||||
}
|
||||
|
||||
@test "--target-count cannot invent experts: the pool's ceiling is reported, not padded" {
|
||||
C="$BATS_TEST_TMPDIR/calib"
|
||||
eight_grams "$C"
|
||||
run python3 "$DERIVE" "$C" --target-count 20
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(cast_count)" -eq 8 ]
|
||||
[[ "$output" == *"total protected: 8 (target 20"* ]]
|
||||
}
|
||||
|
||||
@test "a protected set that cannot be made even is refused instead of emitted" {
|
||||
# One disproportionate expert in half 0, its only companion in half 1: no
|
||||
# candidate could make half 0 even, and an odd half hands the four-bit GEMM two
|
||||
# tables that are not equal. Refusing is the only honest answer.
|
||||
C="$BATS_TEST_TMPDIR/calib"
|
||||
make_gram "$C" 0 0 100 100
|
||||
make_gram "$C" 0 300 1 1000
|
||||
run python3 "$DERIVE" "$C"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"PARITY LAW BROKEN"* ]]
|
||||
[[ "$output" == *"blk.0 half 0 holds 1"* ]]
|
||||
[ "$(cast_count)" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "the splice rewrites the run's copy above the expert rule and refuses a second run" {
|
||||
C="$BATS_TEST_TMPDIR/calib"
|
||||
eight_grams "$C"
|
||||
W="$BATS_TEST_TMPDIR/recipe-work.recipe"
|
||||
cp "$TEMPLATE" "$W"
|
||||
run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 0
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q "derive-protected-experts.py rewrote" "$W"
|
||||
# The derived lines must precede the expert rule: the first rule whose glob
|
||||
# matches a weight decides it, so a cast line below the expert line never fires.
|
||||
cast_line=$(grep -n 'cast dtype=bf16' "$W" | head -1 | cut -d: -f1)
|
||||
expert_line=$(grep -n 'ffn_\*_exps\.\*\.weight' "$W" | head -1 | cut -d: -f1)
|
||||
[ "$cast_line" -lt "$expert_line" ]
|
||||
# The committed template stays the clean, reviewable baseline.
|
||||
! grep -q "derive-protected-experts.py rewrote" "$TEMPLATE"
|
||||
# A second run on the same copy would derive from a recording it already spent.
|
||||
run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 0
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"refusing to re-derive"* ]]
|
||||
}
|
||||
|
||||
@test "the splice refuses before touching the recipe when the count is not reached" {
|
||||
C="$BATS_TEST_TMPDIR/calib"
|
||||
eight_grams "$C"
|
||||
W="$BATS_TEST_TMPDIR/recipe-work.recipe"
|
||||
cp "$TEMPLATE" "$W"
|
||||
run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 20
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"target was 20"* ]]
|
||||
! grep -q "derive-protected-experts.py rewrote" "$W"
|
||||
}
|
||||
|
||||
# --- the arm driver ----------------------------------------------------------
|
||||
|
||||
# drv_fixture <dir> — the smallest tree the driver's gates accept: a checkpoint
|
||||
# with its index and tokenizer, three grams, an imatrix inside the campaign dir
|
||||
# the container mounts, and an empty output dir.
|
||||
drv_fixture() {
|
||||
local d="$1"
|
||||
mkdir -p "$d/ck/Qwen3.8-Flash-Next" "$d/campaign" "$d/calib" "$d/out"
|
||||
: > "$d/ck/Qwen3.8-Flash-Next/model.safetensors.index.json"
|
||||
: > "$d/ck/Qwen3.8-Flash-Next/tokenizer.json"
|
||||
make_gram "$d/calib" 0 0 60 100
|
||||
make_gram "$d/calib" 0 1 15 100
|
||||
make_gram "$d/calib" 0 2 10 100
|
||||
# four experts, not three: a thin pool that cannot be made even trips the parity
|
||||
# audit, and this fixture is meant to reach the later gates
|
||||
make_gram "$d/calib" 0 3 8 100
|
||||
: > "$d/campaign/imatrix.gguf"
|
||||
}
|
||||
|
||||
# run_drv <driver args and K=V overrides, any order> — the driver with every path
|
||||
# pointed at the fixture. Override words are separated out because `env` only
|
||||
# accepts assignments ahead of the command; later assignments win, so a test can
|
||||
# break exactly one input.
|
||||
run_drv() {
|
||||
local d="$BATS_TEST_TMPDIR/drv" assigns=() flags=() w
|
||||
for w in "$@"; do
|
||||
if [[ "$w" =~ ^[A-Za-z_][A-Za-z0-9_]*= ]]; then assigns+=("$w"); else flags+=("$w"); fi
|
||||
done
|
||||
run env CKPT="$d/ck/Qwen3.8-Flash-Next" CAMPAIGN="$d/campaign" STATE="$d/campaign" \
|
||||
CALIB="$d/calib" OUTDIR="$d/out" IMATRIX="$d/campaign/imatrix.gguf" \
|
||||
MIN_GRAMS=2 TARGET_COUNT=0 ${assigns[@]+"${assigns[@]}"} \
|
||||
bash "$CONVERT" ${flags[@]+"${flags[@]}"}
|
||||
}
|
||||
|
||||
@test "the dry run prints the plan and then the convert for every arm" {
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
run_drv --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"== arm cd3 =="* ]]
|
||||
[[ "$output" == *"== arm cd6 =="* ]]
|
||||
[[ "$output" == *"== arm ao1 =="* ]]
|
||||
# one --plan-only bill per arm, and one convert line per arm beside it
|
||||
[ "$(grep -c -- '--plan-only' <<< "$output")" -eq 3 ]
|
||||
[ "$(grep -c 'rad-convert' <<< "$output")" -eq 6 ]
|
||||
awk '/arm cd3/ {next}
|
||||
/qwen38-w5nl64-cd3\.rad/ { if ($0 ~ /--plan-only/) p = NR; else c = NR }
|
||||
END { exit !(p && c && p < c) }' <<< "$output"
|
||||
}
|
||||
|
||||
@test "the cd6 arm re-sends the whole expert option set with cd=6; the control sends none" {
|
||||
# --quant replaces the recipe line wholesale, so an override that forgot an
|
||||
# option would quietly convert with the default for it.
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
run_drv --arm cd3 --arm cd6 --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
cd6line=$(grep -- '-cd6\.rad --quant' <<< "$output" | head -1)
|
||||
[[ "$cd6line" == *"table=w5nl"* ]]
|
||||
[[ "$cd6line" == *"group=64"* ]]
|
||||
[[ "$cd6line" == *"scale=fp8_e4m3"* ]]
|
||||
[[ "$cd6line" == *"transform=fwht128"* ]]
|
||||
[[ "$cd6line" == *"calib=\$CALIB"* ]]
|
||||
[[ "$cd6line" == *",cd=6"* ]]
|
||||
cd3line=$(grep -- '-cd3\.rad' <<< "$output" | grep -v -- '--plan-only' | head -1)
|
||||
[[ "$cd3line" != *"--quant"* ]]
|
||||
}
|
||||
|
||||
@test "the ao1 arm carries act_order=1 and no cd=, because libquant refuses the pair" {
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
run_drv --arm ao1 --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
line=$(grep -- '-ao1\.rad --quant' <<< "$output" | head -1)
|
||||
[[ "$line" == *"act_order=1"* ]]
|
||||
[[ "$line" != *",cd="* ]]
|
||||
}
|
||||
|
||||
@test "the driver splices the protected plane into its own copy, never the template" {
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
run_drv --dry-run
|
||||
[[ "$output" == *"apply-protected-cast.sh $BATS_TEST_TMPDIR/drv/calib "* ]]
|
||||
[[ "$output" == *"/campaign/recipe-"* ]]
|
||||
[[ "$output" != *"$TEMPLATE"* ]]
|
||||
}
|
||||
|
||||
@test "the driver refuses while the BF16 reference container still holds the volume" {
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
: > "$BATS_TEST_TMPDIR/drv/out/qwen38-bf16.rad"
|
||||
run_drv --dry-run
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"peak-disk"* ]]
|
||||
}
|
||||
|
||||
@test "a thin recording is refused: the driver never converts uncalibrated" {
|
||||
# rad-convert converts WITHOUT a calibration recording with one stderr note,
|
||||
# which is silently worse than a GPTQ run that failed.
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
run_drv MIN_GRAMS=9 --dry-run
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"run-qwen38-bf16.sh calib"* ]]
|
||||
}
|
||||
|
||||
@test "an imatrix outside the mounted campaign dir is refused before any convert" {
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
mv "$BATS_TEST_TMPDIR/drv/campaign/imatrix.gguf" "$BATS_TEST_TMPDIR/imatrix.gguf"
|
||||
run_drv IMATRIX="$BATS_TEST_TMPDIR/imatrix.gguf" --dry-run
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"not in the campaign dir"* ]]
|
||||
}
|
||||
|
||||
@test "the driver never overwrites an arm that already exists" {
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
: > "$BATS_TEST_TMPDIR/drv/out/qwen38-w5nl64-cd6.rad"
|
||||
run_drv --arm cd6 --dry-run
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"move it to trash first"* ]]
|
||||
}
|
||||
|
||||
@test "an unknown arm name is refused before the gates run" {
|
||||
# No fixture built: the name check must not need a checkpoint to exist.
|
||||
run_drv --arm cd9 --dry-run
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"unknown arm: cd9"* ]]
|
||||
}
|
||||
|
||||
@test "the driver stops when the protected derivation cannot land the target" {
|
||||
drv_fixture "$BATS_TEST_TMPDIR/drv"
|
||||
run_drv PY=python3 TARGET_COUNT=20 --arm cd3
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"target was 20"* ]]
|
||||
[[ "$output" == *"would convert unprotected"* ]]
|
||||
[[ "$output" != *"--entrypoint"* ]]
|
||||
}
|
||||
|
||||
# --- the acceptance gate -----------------------------------------------------
|
||||
|
||||
# kld_report <path> <model> <kl mean> <top1 fraction> — one score report, in the
|
||||
# shape `run-qwen38-bf16.sh score` writes: an `all` summary plus a per-source one.
|
||||
kld_report() {
|
||||
python3 - "$@" <<'PY'
|
||||
import json, sys
|
||||
path, model, kl, top1 = sys.argv[1:5]
|
||||
kl, top1 = float(kl), float(top1)
|
||||
group = {"kld": {"mean": kl, "p99_9": kl * 4}, "top1_agreement": top1, "positions": 50432}
|
||||
json.dump({"model": model, "all": group,
|
||||
"by_source": {"ops": dict(group, positions=20000)}}, open(path, "w"))
|
||||
PY
|
||||
}
|
||||
|
||||
gate_case() { # gate_case <kl fp8> <kl iq4r> <kl cand> <top1 cand> [extra flags...]
|
||||
D="$BATS_TEST_TMPDIR/kl"
|
||||
mkdir -p "$D"
|
||||
kld_report "$D/fp8.json" fp8-native "$1" 0.981
|
||||
kld_report "$D/iq4r.json" iq4r-moe "$2" 0.962
|
||||
kld_report "$D/cand.json" w5nl64 "$3" "$4"
|
||||
run bash "$GATE" "$D/fp8.json" "$D/iq4r.json" "$D/cand.json" "${@:5}"
|
||||
}
|
||||
|
||||
@test "an arm strictly inside the interval that closes half the gap passes" {
|
||||
gate_case 0.0021 0.0190 0.0090 0.971
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"closes 59.2%"* ]]
|
||||
[[ "$output" == *"KLD GATE PASS"* ]]
|
||||
}
|
||||
|
||||
@test "an arm worse than the served payload fails the interval and the recovery" {
|
||||
gate_case 0.0021 0.0190 0.0210 0.971
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"[FAIL] 2."* ]]
|
||||
[[ "$output" == *"[FAIL] 3."* ]]
|
||||
[[ "$output" == *"KLD GATE FAIL"* ]]
|
||||
}
|
||||
|
||||
@test "anchors that are not ordered stop the gate: the instrument is wrong, not the arm" {
|
||||
# If the near-lossless FP8 release does not sit CLOSER to the bf16 reference
|
||||
# than the 4-bit payload does, the corpus, tokenizer or boot shape is broken and
|
||||
# no arm number can mean anything.
|
||||
gate_case 0.0300 0.0190 0.0120 0.971
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"[FAIL] 1."* ]]
|
||||
[[ "$output" == *"gap is not positive"* ]]
|
||||
}
|
||||
|
||||
@test "an arm measured outside the interval fails even when it looks better" {
|
||||
# Closer to the reference than the achievable anchor is a measurement fault.
|
||||
gate_case 0.0021 0.0190 0.0010 0.971
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"[FAIL] 2."* ]]
|
||||
}
|
||||
|
||||
@test "an arm that buys KL with top-1 agreement fails" {
|
||||
gate_case 0.0021 0.0190 0.0090 0.940
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"[PASS] 3."* ]]
|
||||
[[ "$output" == *"[FAIL] 4."* ]]
|
||||
}
|
||||
|
||||
@test "--recover tightens the same numbers until they fail" {
|
||||
gate_case 0.0021 0.0190 0.0090 0.971 --recover 0.8
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"needs 80%"* ]]
|
||||
[[ "$output" == *"[FAIL] 3."* ]]
|
||||
}
|
||||
|
||||
@test "a slice the report does not carry is named, not guessed at" {
|
||||
gate_case 0.0021 0.0190 0.0090 0.971 --slice code
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"has no slice 'code'"* ]]
|
||||
[[ "$output" == *"it has: ops"* ]]
|
||||
}
|
||||
Executable
+445
@@ -0,0 +1,445 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: scripts/grams-to-imatrix.py — the calibration recording (RADGRAM2 grams)
|
||||
# folded into the GGUF v3 imatrix `rad-convert --imatrix F` reads.
|
||||
#
|
||||
# The only contract that matters is the reader's: core/format/imatrix.cpp joins
|
||||
# `<base>.in_sum2` with `<base>.counts`, takes cols from in_sum2 dims[0] and
|
||||
# n_expert from dims[1], demands ggml f32 tensors and GGUF version 3, and
|
||||
# silently IGNORES an entry whose width does not match the weight it is asked
|
||||
# about. So every test here parses the artifact back with its own tiny GGUF
|
||||
# reader and asserts bytes — a file that merely exists and exits 0 proves
|
||||
# nothing about a quantiser that will read it hours later on another machine.
|
||||
#
|
||||
# Fixtures are synthetic: k=4 grams with real RADGRAM2 headers (rows is a u64
|
||||
# at offset 16 — pack it as u32 and 100 rows read back as 4294967296000) and
|
||||
# real f32 payloads whose OFF-DIAGONAL is filled with 99.0, so an entry built
|
||||
# from anything but the diagonal is instantly visible. Expected numbers are
|
||||
# worked examples from the fixture tables below, never recomputed the way the
|
||||
# tool computes them.
|
||||
|
||||
load helpers
|
||||
|
||||
TOOL="$REPO_ROOT/scripts/grams-to-imatrix.py"
|
||||
|
||||
# make_gram <path> <k> <rank> <rows> <domain> <diag,csv>
|
||||
# A RADGRAM2 file: 32-byte header <8sIIQI (magic, k, rank, rows, domain, pad)
|
||||
# then k*k f32 row-major. Off-diagonals are 99.0 — a sentinel, not data.
|
||||
make_gram() {
|
||||
python3 - "$@" <<'PY'
|
||||
import struct, sys
|
||||
path, k, rank, rows, domain = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), \
|
||||
int(sys.argv[4]), int(sys.argv[5])
|
||||
diag = [float(x) for x in sys.argv[6].split(",")]
|
||||
assert len(diag) == k, "diagonal must hold k values"
|
||||
vals = [99.0] * (k * k)
|
||||
for i, d in enumerate(diag):
|
||||
vals[i * k + i] = d
|
||||
hdr = struct.pack("<8sIIQI", b"RADGRAM2", k, rank, rows, domain) + struct.pack("<I", 0)
|
||||
with open(path, "wb") as f:
|
||||
f.write(hdr + struct.pack("<%df" % (k * k), *vals))
|
||||
PY
|
||||
}
|
||||
|
||||
# full_recording <dir> — the fixture every happy-path test reads.
|
||||
#
|
||||
# layer 0 (k=4, own-gram floor 4*k = 16 rows):
|
||||
# pooled r0 rows 40 diag 1,2,3,4 + r1 rows 8 diag .5,.5,.5,.5
|
||||
# -> summed gram (the engine's calib_sum law): diag 1.5,2.5,3.5,4.5,
|
||||
# pooled rows 48
|
||||
# expert 0 r0 rows 100 diag 10,20,30,40 (>= 16: own gram kept)
|
||||
# expert 1 r0 rows 3 diag 7,7,7,7 (< 16: inherits pooled)
|
||||
# expert 2 r1 rows 64 diag 1,1,1,2 (>= 16: own gram kept)
|
||||
# expert 3 no file at all (inherits pooled, count 0)
|
||||
# expert 4 r1 rows 100 diag 5,6,7,8 (>= 16: own gram kept)
|
||||
# -> n_expert 5, counts 100 3 64 0 100, inherited 2, routing 267
|
||||
# layer 1: pooled r0 rows 48 diag 2,2,2,2 ; expert 0 on r1 rows 48 diag 1,1,1,1
|
||||
# -> n_expert 1, counts 48, inherited 0, routing 48
|
||||
# Corpus rows = the largest single-layer pooled rows = 48.
|
||||
full_recording() {
|
||||
local d="$1"
|
||||
mkdir -p "$d"
|
||||
make_gram "$d/gram.r0.blk.0.ffn_down_exps.bin" 4 0 40 0 "1,2,3,4"
|
||||
make_gram "$d/gram.r1.blk.0.ffn_down_exps.bin" 4 1 8 0 "0.5,0.5,0.5,0.5"
|
||||
make_gram "$d/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 100 0 "10,20,30,40"
|
||||
make_gram "$d/gram.r0.blk.0.ffn_down_exps.1.bin" 4 0 3 0 "7,7,7,7"
|
||||
make_gram "$d/gram.r1.blk.0.ffn_down_exps.2.bin" 4 1 64 0 "1,1,1,2"
|
||||
make_gram "$d/gram.r1.blk.0.ffn_down_exps.4.bin" 4 1 100 0 "5,6,7,8"
|
||||
make_gram "$d/gram.r0.blk.1.ffn_down_exps.bin" 4 0 48 0 "2,2,2,2"
|
||||
make_gram "$d/gram.r1.blk.1.ffn_down_exps.0.bin" 4 1 48 0 "1,1,1,1"
|
||||
}
|
||||
|
||||
# gguf_read <file> [vals <tensor> <float-index> <count>] — an independent GGUF
|
||||
# v3 reader: KV block, tensor infos, and (on request) the f32 values behind a
|
||||
# tensor's own declared offset. Prints KV/HDR/TENSOR lines. Nothing here shares
|
||||
# code with the writer, which is the only reason the byte assertions mean
|
||||
# anything: imatrix.cpp walks the same three structures.
|
||||
gguf_read() {
|
||||
python3 - "$@" <<'PY'
|
||||
import struct, sys
|
||||
d = open(sys.argv[1], "rb").read()
|
||||
def u32(o): return struct.unpack_from("<I", d, o)[0]
|
||||
def u64(o): return struct.unpack_from("<Q", d, o)[0]
|
||||
def st(o):
|
||||
n = u64(o)
|
||||
return d[o + 8:o + 8 + n].decode(), o + 8 + n
|
||||
assert d[:4] == b"GGUF", "magic is %r" % d[:4]
|
||||
ver, n_t, n_kv = u32(4), u64(8), u64(16)
|
||||
p, align, out = 24, 32, []
|
||||
for _ in range(n_kv):
|
||||
key, p = st(p)
|
||||
ty = u32(p); p += 4
|
||||
if ty == 4: # uint32
|
||||
v = u32(p); p += 4
|
||||
if key == "general.alignment":
|
||||
align = v
|
||||
elif ty == 8: # string
|
||||
v, p = st(p)
|
||||
elif ty == 9: # array
|
||||
et = u32(p); n = u64(p + 4); p += 12
|
||||
got = []
|
||||
for _i in range(n):
|
||||
if et != 8:
|
||||
raise SystemExit("kv array element type %d" % et)
|
||||
s, p = st(p)
|
||||
got.append(s)
|
||||
v = ",".join(got)
|
||||
else:
|
||||
raise SystemExit("kv type %d is not steppable" % ty)
|
||||
out.append("KV %s %s" % (key, v))
|
||||
tens = []
|
||||
for _ in range(n_t):
|
||||
name, p = st(p)
|
||||
nd = u32(p); p += 4
|
||||
dims = [u64(p + 8 * i) for i in range(nd)]; p += 8 * nd
|
||||
ty = u32(p); off = u64(p + 4); p += 12
|
||||
tens.append((name, dims, ty, off))
|
||||
ds = (p + align - 1) // align * align
|
||||
out.append("HDR version=%d n_tensors=%d n_kv=%d alignment=%d data_start=%d"
|
||||
% (ver, n_t, n_kv, align, ds))
|
||||
for name, dims, ty, off in tens:
|
||||
out.append("TENSOR %s %s %d %d" % (name, "x".join(str(x) for x in dims), ty, off))
|
||||
if len(sys.argv) > 2 and sys.argv[2] == "vals":
|
||||
name, i0, n = sys.argv[3], int(sys.argv[4]), int(sys.argv[5])
|
||||
for nm, dims, ty, off in tens:
|
||||
if nm == name:
|
||||
base = ds + off
|
||||
vals = [("%d" % v) if float(v).is_integer() else repr(v)
|
||||
for v in (struct.unpack_from("<f", d, base + 4 * (i0 + i))[0]
|
||||
for i in range(n))]
|
||||
out.append("VALS %s %s" % (name, " ".join(vals)))
|
||||
print("\n".join(out))
|
||||
PY
|
||||
}
|
||||
|
||||
# aligned_offsets <file> — exit 0 iff the data section starts aligned AND every
|
||||
# tensor's absolute data offset (data_start + t.off) is a multiple of
|
||||
# general.alignment. This is the writer bug that no value assertion can see: an
|
||||
# unaligned offset makes the reader's extent arithmetic land mid-float in the
|
||||
# previous tensor, and it reads the result without complaining.
|
||||
aligned_offsets() {
|
||||
python3 - "$1" <<'PY'
|
||||
import struct, sys
|
||||
d = open(sys.argv[1], "rb").read()
|
||||
def u32(o): return struct.unpack_from("<I", d, o)[0]
|
||||
def u64(o): return struct.unpack_from("<Q", d, o)[0]
|
||||
def st(o):
|
||||
n = u64(o)
|
||||
return d[o + 8:o + 8 + n].decode(), o + 8 + n
|
||||
ver, n_t, n_kv = u32(4), u64(8), u64(16)
|
||||
p, align = 24, 32
|
||||
for _ in range(n_kv):
|
||||
key, p = st(p)
|
||||
ty = u32(p); p += 4
|
||||
if ty == 4:
|
||||
v = u32(p); p += 4
|
||||
if key == "general.alignment":
|
||||
align = v
|
||||
if ty == 8:
|
||||
_v, p = st(p)
|
||||
elif ty == 9:
|
||||
et = u32(p); n = u64(p + 4); p += 12
|
||||
for _i in range(n):
|
||||
if et == 8:
|
||||
_v, p = st(p)
|
||||
elif et == 4:
|
||||
p += 4
|
||||
else:
|
||||
raise SystemExit("kv array element type %d" % et)
|
||||
elif ty in (0, 1):
|
||||
p += 1
|
||||
elif ty in (2, 3):
|
||||
p += 2
|
||||
elif ty in (5, 6):
|
||||
p += 4
|
||||
elif ty in (10, 11, 12):
|
||||
p += 8
|
||||
tens = []
|
||||
for _ in range(n_t):
|
||||
name, p = st(p)
|
||||
nd = u32(p); p += 4
|
||||
p += 8 * nd
|
||||
ty = u32(p); off = u64(p + 4); p += 12
|
||||
tens.append((name, ty, off))
|
||||
ds = (p + align - 1) // align * align
|
||||
bad = [] if ds % align == 0 else ["data_start=%d" % ds]
|
||||
bad += ["%s=%d" % (nm, ds + off) for nm, _ty, off in tens if (ds + off) % align]
|
||||
if bad:
|
||||
print("UNALIGNED", ";".join(bad))
|
||||
sys.exit(1)
|
||||
print("ALIGNED %d tensor(s), data_start=%d, alignment=%d" % (len(tens), ds, align))
|
||||
PY
|
||||
}
|
||||
|
||||
# --- the artifact -----------------------------------------------------------
|
||||
|
||||
@test "a complete recording becomes a GGUF v3 file and the run reports itself" {
|
||||
C="$BATS_TEST_TMPDIR/full"
|
||||
full_recording "$C"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf" \
|
||||
--datasets calib,chat --chunk-size 16
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"IMATRIX_OK"* ]]
|
||||
head -c 4 "$BATS_TEST_TMPDIR/im.gguf" | grep -q "GGUF"
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
|
||||
[[ "$output" == *"HDR version=3"* ]]
|
||||
# The reader refuses anything but version 3, so this is the whole gate.
|
||||
[[ "$output" != *"version=2"* ]]
|
||||
# Everything the reader looks for by key, and the honesty of the chunk math:
|
||||
# 48 corpus rows over 16-row chunks is 3 chunks.
|
||||
[[ "$output" == *"KV general.alignment 32"* ]]
|
||||
[[ "$output" == *"KV imatrix.chunk_size 16"* ]]
|
||||
[[ "$output" == *"KV imatrix.chunk_count 3"* ]]
|
||||
[[ "$output" == *"KV imatrix.datasets calib,chat"* ]]
|
||||
[[ "$output" == *"n_tensors=12"* ]]
|
||||
}
|
||||
|
||||
@test "the reader finds the folded MoE entry: one per layer, expert-major, f32" {
|
||||
C="$BATS_TEST_TMPDIR/full"
|
||||
full_recording "$C"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
|
||||
[ "$status" -eq 0 ]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
|
||||
# dims (k, n_expert) with k fastest, ggml type 0 (f32) or the entry is
|
||||
# warned away, and 5 experts because the fixture's largest global id is 4.
|
||||
[[ "$output" == *"TENSOR blk.0.ffn_down_exps.in_sum2 4x5 0 "* ]]
|
||||
[[ "$output" == *"TENSOR blk.0.ffn_down_exps.counts 1x5 0 "* ]]
|
||||
# counts are the routing rows, not the energy trace: 100 3 64 0 100.
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 0 5
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
|
||||
# Expert 0 (floats 0..3) and expert 4 (floats 16..19) carry their own
|
||||
# matrices' diagonals; the 99.0 off-diagonal fill appears nowhere.
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 10 20 30 40"* ]]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 16 4
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 5 6 7 8"* ]]
|
||||
[[ "$output" != *"99"* ]]
|
||||
}
|
||||
|
||||
@test "an expert with no own gram carries the pooled diagonal, and the count of them is reported" {
|
||||
C="$BATS_TEST_TMPDIR/full"
|
||||
full_recording "$C"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"inherited=2"* ]]
|
||||
# Expert 1 has a gram (3 rows, under the 4*k floor) and expert 3 has none at
|
||||
# all; both slices must be the POOLED diag 1+0.5, 2+0.5, 3+0.5, 4+0.5.
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 4 4
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 12 4
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
|
||||
# Their counts stay their own routing: 3 for the under-threshold expert, 0
|
||||
# for the unrecorded one — popularity is count's job, never the gram's.
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 1 3
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 3 64 0"* ]]
|
||||
# The census is a report for the operator, so it belongs on stderr. `run`
|
||||
# folds both streams into $output, so a second run into a different -o (an
|
||||
# existing output is refused by design) captures stderr alone: without the
|
||||
# stderr write there is no file to grep and this assertion goes red.
|
||||
python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im2.gguf" 2>"$BATS_TEST_TMPDIR/err"
|
||||
[ -s "$BATS_TEST_TMPDIR/err" ]
|
||||
grep -q "inherited=2" "$BATS_TEST_TMPDIR/err"
|
||||
}
|
||||
|
||||
@test "the hidden-width entries are the same bytes, and nothing else is emitted" {
|
||||
C="$BATS_TEST_TMPDIR/full"
|
||||
full_recording "$C"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
|
||||
[ "$status" -eq 0 ]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
|
||||
# gate_up shares the down matrices (same hidden-state rows), so its slice 4
|
||||
# must read 5 6 7 8 at its own tensor info's offset.
|
||||
[[ "$output" == *"TENSOR blk.0.ffn_gate_up_exps.in_sum2 4x5 0 "* ]]
|
||||
[[ "$output" == *"TENSOR blk.0.ssm_inz.in_sum2 4 0 "* ]]
|
||||
[[ "$output" == *"TENSOR blk.0.ssm_inz.counts 1 0 "* ]]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_gate_up_exps.in_sum2 16 4
|
||||
[[ "$output" == *"VALS blk.0.ffn_gate_up_exps.in_sum2 5 6 7 8"* ]]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.in_sum2 0 4
|
||||
[[ "$output" == *"VALS blk.0.ssm_inz.in_sum2 1.5 2.5 3.5 4.5"* ]]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.counts 0 1
|
||||
[[ "$output" == *"VALS blk.0.ssm_inz.counts 48"* ]]
|
||||
# The stop rule: importance_for() false-on-a-width-mismatch means an entry
|
||||
# for a projection whose input is NOT the gram's width is a silent lie.
|
||||
[[ "$output" != *"ssm_out"* ]]
|
||||
[[ "$output" != *"attn"* ]]
|
||||
[[ "$output" != *"mtp."* ]]
|
||||
[[ "$output" != *"ple_ngram"* ]]
|
||||
[[ "$output" != *"output.weight"* ]]
|
||||
}
|
||||
|
||||
@test "rows is a u64: a corpus too big for u32 survives the header" {
|
||||
# The trap this fixture law exists for: pack rows as u32 at offset 16 and a
|
||||
# 100-row file reads back as 4294967296000. 2**32 is exactly representable
|
||||
# in f32, so the count round-trips only if the tool read 8 bytes.
|
||||
C="$BATS_TEST_TMPDIR/wide"
|
||||
mkdir -p "$C"
|
||||
make_gram "$C/gram.r0.blk.0.ffn_down_exps.bin" 4 0 4294967296 0 "1,1,1,1"
|
||||
make_gram "$C/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 4294967296 0 "3,3,3,3"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide.gguf" --chunk-size 4096
|
||||
[ "$status" -eq 0 ]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/wide.gguf" vals blk.0.ffn_down_exps.counts 0 1
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 4294967296"* ]]
|
||||
[[ "$output" != *"4294967296000"* ]]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/wide.gguf"
|
||||
[[ "$output" == *"KV imatrix.chunk_count 1048576"* ]]
|
||||
# imatrix.chunk_count is a u32 field, so 2**32 rows in 1-row chunks has no
|
||||
# legal encoding: wrapping it would put a number downstream believes.
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide2.gguf" --chunk-size 1
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"*"CHUNK_COUNT"* ]]
|
||||
[ ! -e "$BATS_TEST_TMPDIR/wide2.gguf" ]
|
||||
}
|
||||
|
||||
@test "every tensor's data offset is a multiple of general.alignment" {
|
||||
C="$BATS_TEST_TMPDIR/full"
|
||||
full_recording "$C"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
|
||||
[ "$status" -eq 0 ]
|
||||
run aligned_offsets "$BATS_TEST_TMPDIR/im.gguf"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == "ALIGNED 12 tensor(s),"*"alignment=32" ]]
|
||||
|
||||
# Red proof, in the same test so the assertion cannot rot: the same file
|
||||
# with one tensor's declared offset nudged by 4. A counts tensor is 4 bytes,
|
||||
# so a writer that forgets to pad between tensors produces exactly this
|
||||
# shape of file — and every value assertion above still passes on it.
|
||||
python3 - "$BATS_TEST_TMPDIR/im.gguf" "$BATS_TEST_TMPDIR/unaligned.gguf" <<'PY'
|
||||
import struct, sys
|
||||
src, dst = sys.argv[1], sys.argv[2]
|
||||
d = bytearray(open(src, "rb").read())
|
||||
def u32(o): return struct.unpack_from("<I", d, o)[0]
|
||||
def u64(o): return struct.unpack_from("<Q", d, o)[0]
|
||||
def st(o):
|
||||
n = u64(o)
|
||||
return d[o + 8:o + 8 + n].decode(), o + 8 + n
|
||||
p = 24
|
||||
for _ in range(u64(16)):
|
||||
key, p = st(p)
|
||||
ty = u32(p); p += 4
|
||||
if ty == 4:
|
||||
p += 4
|
||||
elif ty == 8:
|
||||
_v, p = st(p)
|
||||
elif ty == 9:
|
||||
et = u32(p); n = u64(p + 4); p += 12
|
||||
for _i in range(n):
|
||||
_v, p = st(p)
|
||||
for _ in range(u64(8)):
|
||||
name, p = st(p)
|
||||
nd = u32(p); p += 4
|
||||
p += 8 * nd
|
||||
off_at = p + 4 # the u64 offset field of this tensor
|
||||
if name == "blk.0.ssm_inz.counts":
|
||||
struct.pack_into("<Q", d, off_at, u64(off_at) + 4)
|
||||
p += 12
|
||||
open(dst, "wb").write(bytes(d))
|
||||
PY
|
||||
run aligned_offsets "$BATS_TEST_TMPDIR/unaligned.gguf"
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"UNALIGNED"*"blk.0.ssm_inz.counts"* ]]
|
||||
}
|
||||
|
||||
# --- the refusals -----------------------------------------------------------
|
||||
|
||||
@test "a gram short by one float is refused and the file is named" {
|
||||
C="$BATS_TEST_TMPDIR/short"
|
||||
full_recording "$C"
|
||||
# Cut the last 4 bytes of one expert gram: a torn write, which is what a
|
||||
# killed tap actually leaves behind.
|
||||
p="$C/gram.r0.blk.0.ffn_down_exps.0.bin"
|
||||
python3 -c 'import sys
|
||||
p = sys.argv[1]
|
||||
open(p, "wb").write(open(p, "rb").read()[:-4])' "$p"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"*"SHORT"* ]]
|
||||
[[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
|
||||
[ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
|
||||
}
|
||||
|
||||
@test "an empty recording directory is refused" {
|
||||
mkdir -p "$BATS_TEST_TMPDIR/empty"
|
||||
run python3 "$TOOL" "$BATS_TEST_TMPDIR/empty" -o "$BATS_TEST_TMPDIR/no.gguf"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"*"NO_GRAMS"* ]]
|
||||
[[ "$output" == *"$BATS_TEST_TMPDIR/empty"* ]]
|
||||
[ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
|
||||
}
|
||||
|
||||
@test "expert grams with no pooled gram for the layer are refused" {
|
||||
# Without the pooled gram there is nothing for an under-threshold expert to
|
||||
# inherit, and the tool would quietly invent a zero diagonal for it.
|
||||
C="$BATS_TEST_TMPDIR/nopooled"
|
||||
mkdir -p "$C"
|
||||
make_gram "$C/gram.r0.blk.3.ffn_down_exps.7.bin" 4 0 500 0 "1,2,3,4"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"*"NO_POOLED"*"layer 3"* ]]
|
||||
[[ "$output" == *"gram.r<rank>.blk.3.ffn_down_exps.bin"* ]]
|
||||
}
|
||||
|
||||
@test "the same global expert recorded twice with different numbers is refused" {
|
||||
# Rank files carry GLOBAL expert numbering and so describe disjoint experts;
|
||||
# a collision means the recording is corrupt, and whichever file sorted last
|
||||
# would win.
|
||||
C="$BATS_TEST_TMPDIR/conflict"
|
||||
full_recording "$C"
|
||||
make_gram "$C/gram.r1.blk.0.ffn_down_exps.0.bin" 4 1 77 0 "10,20,30,40"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"*"EXPERT_CONFLICT"*"expert 0"* ]]
|
||||
[[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
|
||||
[[ "$output" == *"gram.r1.blk.0.ffn_down_exps.0.bin"* ]]
|
||||
}
|
||||
|
||||
@test "an existing output is refused by name rather than overwritten" {
|
||||
# The trash law: a re-run must not destroy the imatrix a convert is reading.
|
||||
C="$BATS_TEST_TMPDIR/full"
|
||||
full_recording "$C"
|
||||
: > "$BATS_TEST_TMPDIR/keep.gguf"
|
||||
printf 'do not clobber me\n' > "$BATS_TEST_TMPDIR/keep.gguf"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/keep.gguf"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"*"EXISTS"*"keep.gguf"* ]]
|
||||
run cat "$BATS_TEST_TMPDIR/keep.gguf"
|
||||
[[ "$output" == *"do not clobber me"* ]]
|
||||
}
|
||||
|
||||
@test "--min-rows moves the threshold and the bytes follow it" {
|
||||
# The default threshold is the engine's own law (4*k rows, lq_hess.h), but
|
||||
# the operator has to be able to raise it when a corpus is thin. Asserted on
|
||||
# the slices, not on a report line: a flag that only changes prose is a
|
||||
# flag that changes nothing.
|
||||
C="$BATS_TEST_TMPDIR/floor"
|
||||
full_recording "$C"
|
||||
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/far.gguf" --min-rows 200
|
||||
[ "$status" -eq 0 ]
|
||||
# 200 is above every expert's routing (100, 3, 64, 0, 100 and layer 1's 48),
|
||||
# so all six slices now inherit.
|
||||
[[ "$output" == *"inherited=6"* ]]
|
||||
run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
|
||||
# The counts do not move with the threshold: routing is a measurement, not a
|
||||
# judgement about how much data is enough.
|
||||
run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.counts 0 5
|
||||
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
|
||||
}
|
||||
Executable
+337
@@ -0,0 +1,337 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the BF16 REFERENCE container of the Qwen3.8-Flash-Next five-bit
|
||||
# campaign (feat/qwen38-w5nl64) — scripts/build-qwen38-bf16-rad.sh and
|
||||
# scripts/run-qwen38-bf16.sh.
|
||||
#
|
||||
# The reference is what every later stage is judged or derived from (the KLD
|
||||
# record scores the arms, the Grams derive the protected experts), so the
|
||||
# deliverable here is the set of refusals: a half checkpoint, a second convert,
|
||||
# a volume that cannot hold the convert, an engine-refused plan, and any run —
|
||||
# including a dry one — that would take the GPUs from frank-archon. Each test
|
||||
# drives the script's own exit status and printed argv. Offline: no engine, no
|
||||
# GPU, no container, no systemctl, no network. Path knobs aim at
|
||||
# $BATS_TEST_TMPDIR.
|
||||
#
|
||||
# The third mode, score, carries the campaign's comparability law: its engine
|
||||
# argv must equal kldref's except for --model and the --kld-* pair, or the KL
|
||||
# numbers of the FP8-native anchor, the served payload and the arms are not one
|
||||
# measurement series. The seam proves that by diffing the two dry runs.
|
||||
|
||||
# The 1.8.1 law asserted in two places below: --vram-kv-mib was REMOVED at
|
||||
# 1.3.0, so naming it is a boot refusal — the reference serve must carry the
|
||||
# budget-resolver knobs (--host-pool-mib / --gpu-headroom-mib) instead.
|
||||
|
||||
load helpers
|
||||
|
||||
BUILD="$REPO_ROOT/scripts/build-qwen38-bf16-rad.sh"
|
||||
RUNNER="$REPO_ROOT/scripts/run-qwen38-bf16.sh"
|
||||
|
||||
# The reference shape, spelled exactly as run-qwen38-bf16.sh must pass it: this
|
||||
# is the contract the recorded reference is measured at, so a silent edit here
|
||||
# is a silent edit to every KLD number in the campaign.
|
||||
SHAPE="--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 49152 --gpu-headroom-mib 96 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --ngram-placement ram --override-chat-template /models/tpl.jinja"
|
||||
|
||||
# oneline — both dry runs print one shell-quoted argument a line; joining them
|
||||
# lets an option and its value be asserted as the pair they are.
|
||||
oneline() { local s="${output//$'\n'/ }"; printf '%s' "$s"; }
|
||||
|
||||
# engine_argv — the arguments AFTER the image line of a dry run, with --model
|
||||
# and every --kld-* flag and its value removed: what is left is the measuring
|
||||
# shape, and it must be identical between kldref and score.
|
||||
engine_argv() {
|
||||
local seen=0 skip=0 a
|
||||
while IFS= read -r a; do
|
||||
if [ "$seen" = 0 ]; then
|
||||
[[ "$a" == *"radiance@sha256:"* ]] && seen=1
|
||||
continue
|
||||
fi
|
||||
case "$a" in
|
||||
--model|--kld-record|--kld-corpus|--kld-ref|--kld-out) skip=1; continue ;;
|
||||
esac
|
||||
if [ "$skip" = 1 ]; then skip=0; continue; fi
|
||||
printf '%s\n' "$a"
|
||||
done <<< "$1"
|
||||
}
|
||||
|
||||
# flag_value FLAG — the value a dry run printed for FLAG (the argv is one
|
||||
# argument a line, so the value is the line after it). Lets a test ask what
|
||||
# number the script chose instead of re-declaring the literal it guards.
|
||||
flag_value() {
|
||||
local want="$1" prev="" a
|
||||
while IFS= read -r a; do
|
||||
if [ "$prev" = "$want" ]; then printf '%s' "$a"; return 0; fi
|
||||
prev="$a"
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
@test "the build dry run prints the rad-convert argv for the checkpoint and the bf16 output" {
|
||||
M="$BATS_TEST_TMPDIR/models"
|
||||
run env FRANK_QWEN38_MODELS="$M" bash "$BUILD" --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"rad-convert"* ]]
|
||||
# the argv is the container's view: the models dir rides the /models mount
|
||||
[[ "$output" == *"$M:/models"* ]]
|
||||
flat="$(oneline)"
|
||||
[[ "$flat" == *"/models/Qwen3.8-Flash-Next -o /models/rad/qwen38-bf16.rad"* ]]
|
||||
# a reference is the checkpoint re-headed, not re-encoded, and it declares a
|
||||
# window the long KLD corpus can be read inside of
|
||||
[[ "$output" != *"--recipe"* ]]
|
||||
[[ "$output" != *"--reuse"* ]]
|
||||
[[ "$flat" == *"--max-ctx 32768"* ]]
|
||||
# and a dry run looked at nothing: the fixture dirs were never created
|
||||
[ ! -e "$M" ]
|
||||
}
|
||||
|
||||
@test "the build refuses a checkpoint that is not on disk and names the fetch command" {
|
||||
M="$BATS_TEST_TMPDIR/models"
|
||||
run env FRANK_QWEN38_MODELS="$M" bash "$BUILD"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"$M/Qwen3.8-Flash-Next"* ]]
|
||||
[[ "$output" == *"scripts/download-qwen38-bf16.sh --fetch"* ]]
|
||||
[ ! -e "$M/rad/qwen38-bf16.rad" ]
|
||||
}
|
||||
|
||||
@test "the build refuses an existing reference container and names --force as the way past it" {
|
||||
M="$BATS_TEST_TMPDIR/models"
|
||||
mkdir -p "$M/Qwen3.8-Flash-Next" "$M/rad"
|
||||
echo "not a container" > "$M/rad/qwen38-bf16.rad"
|
||||
run env FRANK_QWEN38_MODELS="$M" QWEN38_MIN_FREE_MIB=1 \
|
||||
QWEN38_TRASH="$BATS_TEST_TMPDIR/trash" bash "$BUILD"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"$M/rad/qwen38-bf16.rad"* ]]
|
||||
[[ "$output" == *"--force"* ]]
|
||||
# refusing retires nothing: the trash law means the file survives a refusal
|
||||
[ -f "$M/rad/qwen38-bf16.rad" ]
|
||||
[ ! -e "$BATS_TEST_TMPDIR/trash" ]
|
||||
}
|
||||
|
||||
@test "the build refuses when the output filesystem cannot hold a second 335 GiB" {
|
||||
# The campaign's peak-disk law: 343,323 MiB of container written next to
|
||||
# 343,323 MiB of checkpoint. The wanted figure is set above whatever this
|
||||
# box really has, which is what the gate reads.
|
||||
M="$BATS_TEST_TMPDIR/models"
|
||||
mkdir -p "$M/Qwen3.8-Flash-Next" "$M/rad"
|
||||
run env FRANK_QWEN38_MODELS="$M" QWEN38_MIN_FREE_MIB=999999999 bash "$BUILD"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"* ]]
|
||||
[[ "$output" =~ [0-9]+" MiB free" ]]
|
||||
[[ "$output" == *"999999999 MiB wanted"* ]]
|
||||
# the disk gate is the cheap one: it refuses long before the Hub sweep runs
|
||||
[[ "$output" != *"VERIFY_OK"* ]]
|
||||
[ ! -e "$M/rad/qwen38-bf16.rad" ]
|
||||
}
|
||||
|
||||
@test "the reference serve rejects an unknown mode and names every valid mode" {
|
||||
# `probe` is the swift15 era's third mode; it is retired and must not boot
|
||||
# a container here either.
|
||||
run bash "$RUNNER" probe
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"probe"* ]]
|
||||
[[ "$output" == *"kldref"* ]]
|
||||
[[ "$output" == *"calib"* ]]
|
||||
[[ "$output" == *"score"* ]]
|
||||
[[ "$output" != *"podman"* ]]
|
||||
run bash "$RUNNER"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"kldref"* ]]
|
||||
[[ "$output" == *"calib"* ]]
|
||||
[[ "$output" == *"score"* ]]
|
||||
# score without its two positionals is the same usage refusal
|
||||
run bash "$RUNNER" score
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"score RAD OUT"* ]]
|
||||
}
|
||||
|
||||
@test "the kldref dry run carries the record flags at the reference shape and no --vram-kv-mib" {
|
||||
run bash "$RUNNER" kldref --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
flat="$(oneline)"
|
||||
[[ "$flat" == *"--kld-record /data/kld-ref"* ]]
|
||||
[[ "$flat" == *"--kld-corpus /data/kld.jsonl"* ]]
|
||||
# the corpus arrives read-only, the record dir writable
|
||||
[[ "$flat" == *":/data/kld.jsonl:ro"* ]]
|
||||
[[ "$flat" == *":/data/kld-ref "* ]]
|
||||
[[ "$flat" == *"$SHAPE"* ]]
|
||||
# 1.8.1: the flag does not exist — naming it is a boot refusal
|
||||
[[ "$output" != *"--vram-kv-mib"* ]]
|
||||
}
|
||||
|
||||
@test "the calib dry run opens the tap at 12M tokens and serves on 7770" {
|
||||
run bash "$RUNNER" calib --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
flat="$(oneline)"
|
||||
[[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]]
|
||||
[[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]]
|
||||
[[ "$flat" == *"--port 7770"* ]]
|
||||
[[ "$flat" == *":/data/calib "* ]]
|
||||
# calib serves; it does not record
|
||||
[[ "$output" != *"--kld-record"* ]]
|
||||
}
|
||||
|
||||
@test "a dry run prints argv only: no drain, no container, nothing written" {
|
||||
L="$BATS_TEST_TMPDIR/logs"
|
||||
run env FRANK_QWEN38_LOG_DIR="$L" \
|
||||
bash "$RUNNER" kldref --dry-run --image example.invalid/radiance@sha256:0000000000000000000000000000000000000000000000000000000000000000
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" != *"systemctl"* ]]
|
||||
[[ "$output" != *"frank-archon stopped"* ]]
|
||||
[[ "$output" != *"podman rm"* ]]
|
||||
# not vacuous: the podman argv WAS printed, and --image replaced the pin
|
||||
[[ "$output" == *"podman"* ]]
|
||||
[[ "$output" == *"example.invalid/radiance@sha256:000"* ]]
|
||||
[[ "$output" != *"stilldeadcode/radiance"* ]]
|
||||
[ ! -d "$L" ]
|
||||
}
|
||||
|
||||
@test "the score dry run is the kldref argv with only --model and the --kld-* pair changed" {
|
||||
M="$BATS_TEST_TMPDIR/models"
|
||||
C="$BATS_TEST_TMPDIR/campaign"
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
|
||||
bash "$RUNNER" kldref --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
ref_argv="$(engine_argv "$output")"
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
|
||||
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
score_argv="$(engine_argv "$output")"
|
||||
# The whole value of the campaign's three KL numbers is that they rode this
|
||||
# one shape: same pin, --tp 2, fp8 KV, step size, --deterministic, no spec.
|
||||
if [[ "$score_argv" != "$ref_argv" ]]; then
|
||||
echo "--- kldref engine argv ---"; echo "$ref_argv"
|
||||
echo "--- score engine argv ---"; echo "$score_argv"
|
||||
return 1
|
||||
fi
|
||||
flat="$(oneline)"
|
||||
[[ "$flat" == *"--model /models/rad/arm-w5nl64.rad"* ]]
|
||||
[[ "$flat" == *"--kld-ref /data/kld-ref --kld-out /data/out/arm1.json"* ]]
|
||||
# the zero point mounts read-only; the report needs a writable bind because
|
||||
# --kld-out F also writes F.rows beside it
|
||||
[[ "$flat" == *":/data/kld-ref:ro "* ]]
|
||||
[[ "$flat" == *":/data/out "* ]]
|
||||
[[ "$output" != *"--kld-record"* ]]
|
||||
[[ "$output" != *"--kld-corpus"* ]]
|
||||
[[ "$output" != *"--vram-kv-mib"* ]]
|
||||
}
|
||||
|
||||
@test "score refuses a missing reference, a missing container and an existing report" {
|
||||
M="$BATS_TEST_TMPDIR/models"
|
||||
C="$BATS_TEST_TMPDIR/campaign"
|
||||
L="$BATS_TEST_TMPDIR/logs"
|
||||
mkdir -p "$M/rad"
|
||||
: > "$M/rad/arm-w5nl64.rad"
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
|
||||
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"$C/kld-ref"* ]]
|
||||
[[ "$output" == *"run-qwen38-bf16.sh kldref"* ]]
|
||||
# refused before the log dir, so before anything that could reach systemctl
|
||||
[ ! -d "$L" ]
|
||||
|
||||
mkdir -p "$C/kld-ref"
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
|
||||
bash "$RUNNER" score "$M/rad/gone.rad" "$C/scores/arm1.json"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"$M/rad/gone.rad"* ]]
|
||||
[ ! -d "$L" ]
|
||||
|
||||
mkdir -p "$C/scores"
|
||||
: > "$C/scores/arm1.json"
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
|
||||
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"$C/scores/arm1.json"* ]]
|
||||
[[ "$output" == *"trash"* ]]
|
||||
# the trash law: a refusal keeps the old report, it does not overwrite it
|
||||
[ -f "$C/scores/arm1.json" ]
|
||||
[ ! -d "$L" ]
|
||||
}
|
||||
|
||||
@test "the reference and a score pass ride ONE step size, legal on the file tier" {
|
||||
# The gate's claim is one shape; that shape's ceiling is set by the tightest
|
||||
# payload it boots, and the served iq4r and every arm are file-tier
|
||||
# containers, which stream only at <=2048. A reference recorded above that is
|
||||
# a KL between two shapes, not two weights.
|
||||
M="$BATS_TEST_TMPDIR/models"
|
||||
C="$BATS_TEST_TMPDIR/campaign"
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" bash "$RUNNER" kldref --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
ref_step="$(flag_value --max-num-batched-tokens <<< "$output")"
|
||||
[ -n "$ref_step" ]
|
||||
[ "$ref_step" -le 2048 ]
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
|
||||
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
score_step="$(flag_value --max-num-batched-tokens <<< "$output")"
|
||||
[ "$ref_step" = "$score_step" ]
|
||||
# one knob moves both, so the operator can lower it at 2 a.m. without an edit
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
|
||||
bash "$RUNNER" kldref --dry-run
|
||||
[ "$(flag_value --max-num-batched-tokens <<< "$output")" = "1024" ]
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
|
||||
bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
|
||||
[ "$(flag_value --max-num-batched-tokens <<< "$output")" = "1024" ]
|
||||
# calibration is exempt: grams are not logits, so the tap keeps the wide step
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
|
||||
bash "$RUNNER" calib --dry-run
|
||||
[ "$(flag_value --max-num-batched-tokens <<< "$output")" = "65536" ]
|
||||
# a step size that is not a token count is refused, never handed to the engine
|
||||
run env FRANK_QWEN38_MNBT=broad bash "$RUNNER" kldref --dry-run
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"FRANK_QWEN38_MNBT"* ]]
|
||||
[[ "$output" == *"must be a token count"* ]]
|
||||
}
|
||||
|
||||
@test "kldref refuses a corpus its window cannot hold, before the service is touched" {
|
||||
M="$BATS_TEST_TMPDIR/models" R="$M/rad"
|
||||
mkdir -p "$M/tpl" "$R" "$BATS_TEST_TMPDIR/out"
|
||||
: > "$M/tpl/qwen38-froggeric-v22.5.jinja"
|
||||
: > "$R/qwen38-bf16.rad"
|
||||
CORPUS="$BATS_TEST_TMPDIR/long.jsonl"
|
||||
{
|
||||
printf '{"prompt":"short one"}\n'
|
||||
printf '{"prompt":"%s"}\n' "$(head -c 120000 /dev/zero | tr '\0' 'x')"
|
||||
printf '{"prompt":"short three"}\n'
|
||||
} > "$CORPUS"
|
||||
# 32,768 tokens x 3 chars is the pessimistic limit; the 120,002-char line
|
||||
# that opens record 2 is well past it, so the campaign says no up front.
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
|
||||
bash "$RUNNER" kldref --dry-run
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"REFUSED"* ]]
|
||||
[[ "$output" == *"record 2"* ]]
|
||||
[[ "$output" == *"120013 chars"* ]]
|
||||
[[ "$output" == *"past the 98304 a 32768-token window"* ]]
|
||||
[[ "$output" == *"FRANK_QWEN38_MAX_MODEL_LEN"* ]]
|
||||
# the same corpus is fine once the window is raised past it, and the raised
|
||||
# window is what the boot is asked for
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
|
||||
FRANK_QWEN38_MAX_MODEL_LEN=65536 FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" \
|
||||
bash "$RUNNER" kldref --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"corpus fits"* ]]
|
||||
flat="$(oneline)"
|
||||
[[ "$flat" == *"--max-model-len 65536"* ]]
|
||||
# the shipped corpus of three short records fits the shipped window
|
||||
SHORT="$BATS_TEST_TMPDIR/short.jsonl"
|
||||
printf '{"prompt":"tiny"}\n{"prompt":"also tiny"}\n' > "$SHORT"
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$SHORT" \
|
||||
FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" bash "$RUNNER" kldref --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"corpus fits: longest record 22 chars (number 2)"* ]]
|
||||
# a real kldref run refuses at the same place: no log dir, no drain, and the
|
||||
# service would still be up when the script gave up
|
||||
L="$BATS_TEST_TMPDIR/logs-real"
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
|
||||
FRANK_QWEN38_LOG_DIR="$L" bash "$RUNNER" kldref
|
||||
[ "$status" -eq 2 ]
|
||||
[ ! -d "$L" ]
|
||||
# and it is kldref that reads the corpus: score and calib never consult it
|
||||
run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
|
||||
FRANK_QWEN38_KLD_REF_DIR="$BATS_TEST_TMPDIR/ref" \
|
||||
FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" \
|
||||
bash "$RUNNER" score "$R/qwen38-bf16.rad" "$BATS_TEST_TMPDIR/out/arm.json" --dry-run
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" != *"corpus fits"* ]]
|
||||
[[ "$output" != *"REFUSED"* ]]
|
||||
}
|
||||
@@ -20,8 +20,9 @@ SEERR_UNIT="$REPO_ROOT/dankclaw/seerr/quadlet/seerr.container"
|
||||
grep -q '^Restart=always$' "$SEERR_UNIT"
|
||||
}
|
||||
|
||||
@test "seerr image is pinned to an exact version tag" {
|
||||
grep -Eq '^Image=ghcr\.io/seerr-team/seerr:v[0-9]+\.[0-9]+\.[0-9]+$' "$SEERR_UNIT"
|
||||
@test "seerr image is pinned to an exact version tag and digest" {
|
||||
# 2026-10-21 fleet law: tag AND digest (tests/quadlet_image_pins.bats)
|
||||
grep -Eq '^Image=ghcr\.io/seerr-team/seerr:v[0-9]+\.[0-9]+\.[0-9]+@sha256:[0-9a-f]{64}$' "$SEERR_UNIT"
|
||||
}
|
||||
|
||||
@test "seerr publishes 5055 on loopback only (guard chain is the user path)" {
|
||||
|
||||
Reference in new issue
Block a user