Merge pull request 'fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches' (#66 ) from fix/dragoon-context-200k into main

fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches
The lane's child dlopens libxrt_core.so.2 from /opt/xilinx/lib/<multiarch>/ as well; our mounts filled only /opt/xilinx/xrt/lib and the install path. Since the host's libxrt2 was reconfigured 2026-10-20 the child dies naming the missing prefix ('No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2') and the engine supervisor takes the WHOLE container down on a dead lane — measured 2026-10-21: frank-dragoon boot-looped 88 times with chat dead while the GPU lines looked healthy. Each XRT library now mounts THREE times from the same real file (never a symlink); the multiarch name derives from FRANK_DRAGOON_XRT_DIR's basename so the bats fixture proves the derivation. Proven end-to-end before commit: production-shaped scratch boot + the three extra mounts answers the full boot ledger, npu_api buckets u0..u9, and /health ok at the 262144 ceiling. A dead lane is not survivable on this engine — arm it only with all three prefixes (page law updated).
2026-10-21 02:25:42 -07:00 · 2026-10-21 02:21:53 -07:00 · 2026-10-21 01:40:40 -07:00 · 2026-10-21 01:34:28 -07:00 · 2026-10-21 01:34:28 -07:00 · 2026-10-21 00:39:54 -07:00
27 changed files with 1221 additions and 290 deletions
@@ -419,16 +419,17 @@ disagrees with it is the bug.
  digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
  Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
  `/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
+- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
-  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
+  **per-request ceiling** (a longer prompt is a 400, never a truncation);
-  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
+  `HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
-  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
+  pool positions buy concurrency, the per-request window never moves;
-  drafter always armed (the adaptive default disarms it and halves decode);
+  `HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
-  server sampler 1.0/0.95/20.
+  default disarms it and halves decode); server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
+- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
-  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
+  ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
-  under-counts, so a harness window equal to the ceiling makes every long
+  estimate under-counts (trigger is 95 %), so a harness window at the ceiling
-  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
+  makes every long request a guaranteed 400.
  Reasoning arrives in `reasoning_content`; the
  template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
  OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
  `/completion`, `/ping`, `/props`, `/tokenize` are 404.
@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 [Container]
-Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
+Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0
@@ -40,6 +40,7 @@ SERVICE_ORDER = (
    "alloy",
    "node-exporter",
    "cadvisor",
    "gpu-exporter",
    "alertmanager",
    "authentik-postgres",
    "authentik-server",
@@ -94,6 +95,7 @@ DANKCLAW_SERVICES = {
    "alloy",
    "node-exporter",
    "cadvisor",
    "gpu-exporter",
    "alertmanager",
    "authentik-postgres",
    "authentik-server",
@@ -127,6 +129,7 @@ DANKSERVER_SERVICES = {
    "frank-archon",
    "node-exporter",
    "cadvisor",
    "gpu-exporter",
 }
 # FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -452,6 +455,12 @@ def services_for_path(path: str, target: str) -> set[str]:
            # change to it redeploying alertmanager is the only path that
            # reaches it on the host.
            _add(services, "alertmanager")
        if name in {"gpu-exporter.py", "gpu-exporter.service"}:
            # The host-process GPU exporter is installed directly by the
            # gpu-exporter deploy on BOTH hosts, so its script and unit select
            # that service (target_services then keeps it only on the host being
            # planned). Editing the collector restarts the running exporter.
            _add(services, "gpu-exporter")
        if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
            # llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
            # files here; each one selects frank-zealot only. The ik_llama.cpp
@@ -86,7 +86,7 @@ service_is_allowed() {
        dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
        dankclaw:comfyui|dankclaw:hermes|\
        dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|\
        dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
        dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
        dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
-        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|\
+        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
        frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
        frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
        frankbook-pro:readarr)
@@ -275,6 +275,36 @@ install_signal_relay() {
    return 1
 }
 # Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
 # it runs python3 from the checkout on the host so it can read the DRM/cgroup
 # sysfs and shell out to nvidia-smi. Same two-address bind posture as
 # node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
 deploy_gpu_exporter() {
    local unit_dir="$HOME/.config/systemd/user" lan dropin i
    case "$TARGET" in
        dankclaw) lan=192.168.0.143 ;;
        dankserver) lan=192.168.0.235 ;;
        *) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
    esac
    mkdir -p "$unit_dir"
    cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
    dropin="$unit_dir/gpu-exporter.service.d"
    mkdir -p "$dropin"
    printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
        > "$dropin/bind.conf"
    systemctl --user daemon-reload
    systemctl --user enable gpu-exporter.service
    systemctl --user restart gpu-exporter.service
    # The interpreter needs a moment to bind after restart (the same race that
    # once failed a healthy signal-relay deploy).
    for i in $(seq 1 15); do
        curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
        sleep 1
    done
    echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
    return 1
 }
 copy_loki_config() {
    run_privileged mkdir -p /opt/loki/config
    run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
@@ -1371,9 +1401,10 @@ deploy_anubis() {
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
 # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
 # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
-# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
+# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
-# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
+# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
-# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
+# run a 200,000 window with its ~45k fit gap; the pool still carries two
 # 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
 # ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
 # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
 # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -2004,6 +2035,7 @@ deploy_service() {
        tautulli) deploy_tautulli ;;
        alloy) deploy_alloy ;;
        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
        gpu-exporter) deploy_gpu_exporter ;;
        comfyui) deploy_comfyui ;;
        *) deploy_compose_service "$service" ;;
    esac
@@ -133,13 +133,15 @@ fi
 valid_service() {
    local service="$1"
    case "$TARGET:$service" in
        # dankclaw:searxng-network is client-only: a network quadlet with no
        # remote dispatch handler, so the remote gate does not (must not) list it.
        dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
        dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
        dankclaw:gitea-server|dankclaw:homepage|\
        dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
        dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
        dankclaw:alertmanager|\
-        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
+        dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
        dankclaw:open-notebook|\
        dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
        dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
-        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
+        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
-        dankserver:frank-archon|\
+        dankserver:prowlarr|dankserver:frank-archon|\
        frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
        frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
        frankbook-pro:readarr)
@@ -46,31 +46,35 @@ FRANK_ARCHON_TP=2
 # stays EXACT for single-sequence decode and any step of <= 25 rows — see
 # MAX_SEQS x (1 + SPEC_TOKENS) below.
 FRANK_ARCHON_TP_WIRE=wht6
-# Decode slots. A slot costs KV, never decode; the pool is sized to this
+# Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
-# contract and nothing more. At 1.8.1 the budget resolver caps the cache at
+# exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
-# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height
+# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
-# conversation answers 400 while /health and every shorter chat work fine
+# least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
-# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two
+# target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
-# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane,
+# the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
-# instead of advertising a third slot that only exists at shorter prompts.
+# keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
-FRANK_ARCHON_MAX_SEQS=2
+# 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
 # fourth conversations are backed by the cache instead of over-committing it.
 FRANK_ARCHON_MAX_SEQS=4
 # Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
 # limit, and models.yml contextWindow MUST quote this number (over-length is a
 # refusal, not a truncation). The pool is sized for seqs x maxlen, not one
 # sequence.
 FRANK_ARCHON_MAX_MODEL_LEN=200000
 FRANK_ARCHON_PLACEMENT=expert_tiered
-# Pinned host pool for expert rows that do not fit VRAM. Under the budget
+# Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
-# resolver this is ALSO the lever that decides which experts stay on the card:
+# host must pin tp x this number, and the refusal that names it is
-# the card's leftover goes to the expert plane first and the KV cache takes the
+# core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
-# rest, so growing the pool moves experts off the card and shrinking it spills
+# lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
-# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
+# MiB of experts off the card and into the cache. Derived from the boot ledger (a
-# refused a boot). The ledger's coverage line is the referee, not this file.
+# 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
-# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left
+# drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
-# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra
+# the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
-# host RAM above what the expert plane's footprint needs buys no cache. Back at
+# twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
-# 44,928, the size that boots clean.
+# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
-FRANK_ARCHON_HOST_POOL_MIB=44928
+# x8 link when the router picks it. The ledger's coverage line is the referee, not
 # this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
 FRANK_ARCHON_HOST_POOL_MIB=49152
 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
 # floor is built from it.
 FRANK_ARCHON_GPU_HEADROOM_MIB=96
@@ -93,16 +97,29 @@ FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
 # Step sharing. A prefill chunk that shares a step with decoding sequences holds
 # the device for the chunk's whole time, and every decode row in that step waits
-# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
+# that long for its next token: measured at this shape, decode-only steps 15.5 ms
-# `auto` leaves each decode row behind the chunk a row plus its draft window and
+# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
-# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
+# while another session prefills (law and census: [[frank-archon]]).
-# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
+# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
-# chunk took the step and admission never ran. Interleave inserts N decode-only
+# draft window per decoding sequence (3 tokens) and an eighth of the budget per
-# steps after a chunk step taken beside decoders: it protects their cadence and
+# waiting request or waiting prefill, and the source states it will not halve for
-# holds a waiting request out of those steps, so it stays 0 until the post-migration
+# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
-# decode-vs-prefill buckets say it is wanted.
+# steps. A number is the only value that applies whenever the chunk shares a step,
 # and it must not go under the prefill stager's smallest pass -- 1025 rows here,
 # named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
 # because below it every routed layer reads its pooled experts synchronously, so
 # 512 is a trap and the vendor measures the whole job slower there. Interleave
 # inserts N decode-only steps after a chunk step taken beside decoders: it protects
 # their cadence and holds a waiting request out of those steps. That is the decoder
 # dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
 # AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
 # price falls on the prefilling prompt (its chunks lose every held step) and on any
 # request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
 # ALONE, so it cannot buy this back while decoders run. Meter:
 # radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
 # before N rises.
 FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
-FRANK_ARCHON_PREFILL_INTERLEAVE=0
+FRANK_ARCHON_PREFILL_INTERLEAVE=1
 # Prefix cache: pinned host memory only. The disk tier was removed outright
 # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
 # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
@@ -118,9 +135,10 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # RESETS pages_limit to half of RAM on every boot; below the plan the load dies
 # mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
 # itself before refusing by name, so recovery is a boot, not a visit.
-# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
+# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
-# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB.
+# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
-FRANK_ARCHON_TTM_MIN_MIB=150000
+# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
 FRANK_ARCHON_TTM_MIN_MIB=162000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands
 # this unquoted on purpose, so keep multi-flag values as one string.
@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
 # stays at the image default, which is what was qualified) ------------------
 # Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
 HALOGEN_MODEL_ID=frank-dragoon
-# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
+# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
-# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
+# longer prompt is a 400 naming the limit, never a truncation; stretching
-# the page) — a window equal to the ceiling makes every long request a 400.
+# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
-HALOGEN_CTX=200704
+# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
-# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
+# page): with the window at 200,000 the gap is 62,144, which absorbs the
-# at once. Pool positions buy concurrency; the per-request window never moves
+# chars/4 under-count. A window at or near the ceiling makes every long
-# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
+# request a guaranteed 400.
-# RAM the pool takes is RAM the n-gram table's page cache loses.
+HALOGEN_CTX=262144
 # ONE pool behind both slots. The harness window is 200,000, so 401,408
 # still carries TWO maximum harness-length conversations at once
 # (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
 # ceiling-length requests would queue instead — buying those means raising
 # the pool, and RAM the pool takes is RAM the n-gram table's page cache
 # loses. Pool positions buy concurrency; the per-request window never moves
 # with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
 HALOGEN_KV_POOL_POSITIONS=401408
 # Two concurrent conversations: the MTP drafter is not slot-locked on this
 # engine, so the second slot is capacity, not a speculation tax.
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
 FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
 # Node paths and the XRT directory are variables so the bats seam can drive
 # these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
-# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
+# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
-# once at its own path, because those libraries find each other at their install
+# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
-# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
+# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
-# system dir, which dangle inside the container.
+# searches that third prefix; without it the lane dies naming it and the
 # supervisor takes the container down — 2026-10-21 boot loop). The basename of
 # this directory IS the multiarch name. The vendor forbids mounting an
 # /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
 FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
 FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
 # sysfs root the fabric-clock preflight reads (same knob the vendor's
@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
@@ -0,0 +1,212 @@
 #!/usr/bin/env python3
 """gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
 Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
    gpu_utilization_percent  gpu_memory_used_bytes  gpu_memory_total_bytes
    gpu_temperature_celsius  gpu_power_watts
 each labelled {gpu, vendor, model}. The scrape job adds `host`.
 Two sources, both host-native — no container, no GPU passthrough:
  * AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
    Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
    as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
    is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
    the R9700s, so those cards legitimately have no gpu_power_watts line; a
    missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
    would plot as a real reading.
  * NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
    Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
    fails; that is tolerated and simply yields no NVIDIA series.
 Default mode prints one scrape to stdout and exits (the test harness, and any
 `service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
 Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
 wildcard, matching node-exporter's posture on these boxes.
 """
 import argparse
 import glob
 import os
 import subprocess
 import sys
 import threading
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 FAMILIES = [
    ("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
    ("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
    ("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
    ("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
    ("gpu_power_watts", "gauge", "Board power draw in watts."),
 ]
 DEFAULT_NVIDIA_CMD = (
    "nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
    "memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
 )
 def _read(path):
    try:
        with open(path) as fh:
            return fh.read().strip()
    except OSError:
        return ""
 def _pci_slot(device_dir):
    uevent = _read(os.path.join(device_dir, "uevent"))
    for line in uevent.splitlines():
        if line.startswith("PCI_SLOT_NAME="):
            return line.split("=", 1)[1]
    return ""
 def _pci_id(device_dir):
    uevent = _read(os.path.join(device_dir, "uevent"))
    for line in uevent.splitlines():
        if line.startswith("PCI_ID="):
            return "amd:" + line.split("=", 1)[1].lower()
    return "amd"
 def collect_amd(sys_dir):
    for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
        device = os.path.join(card, "device")
        busy = _read(os.path.join(device, "gpu_busy_percent"))
        if busy == "":
            continue  # not an amdgpu compute node (e.g. a render node w/o metrics)
        labels = {"gpu": _pci_slot(device) or os.path.basename(card),
                  "vendor": "amd", "model": _pci_id(device)}
        yield "gpu_utilization_percent", labels, busy
        for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
                           ("gpu_memory_total_bytes", "mem_info_vram_total")):
            v = _read(os.path.join(device, leaf))
            if v != "":
                yield name, labels, v
        temp = ""
        for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
            temp = _read(hwmon)
            if temp != "":
                break
        if temp != "":
            yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
        power = _read(os.path.join(device, "power1_average"))
        if power != "":  # empty on the R9700s -> metric absent, not zero
            yield "gpu_power_watts", labels, str(int(power) / 1e6)
 def collect_nvidia(cmd):
    if not cmd:
        return
    try:
        out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
                             timeout=10, check=True).stdout
    except (subprocess.SubprocessError, OSError):
        return  # no NVIDIA GPU / no driver: tolerate, emit nothing
    for line in out.splitlines():
        parts = [p.strip() for p in line.split(",")]
        if len(parts) < 7:
            continue
        idx, name, util, mused, mtotal, temp, power = parts[:7]
        labels = {"gpu": idx, "vendor": "nvidia", "model": name}
        yield "gpu_utilization_percent", labels, util
        yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
        yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
        if temp not in ("", "[N/A]"):
            yield "gpu_temperature_celsius", labels, temp
        if power not in ("", "[N/A]"):
            yield "gpu_power_watts", labels, power
 def collect(sys_dir, nvidia_cmd):
    samples = {}
    for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
        for name, labels, value in source:
            try:
                value = str(float(value))  # normalise; drop trailing junk
            except ValueError:
                continue
            samples.setdefault(name, []).append((labels, value))
    return samples
 def _fmt(value):
    f = float(value)
    if f == int(f) and abs(f) < 1e15:
        return str(int(f))
    return repr(f)
 def render(samples):
    out = []
    for name, kind, help_text in FAMILIES:
        out.append("# HELP %s %s" % (name, help_text))
        out.append("# TYPE %s %s" % (name, kind))
        for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
            lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
            out.append("%s{%s} %s" % (name, lset, _fmt(value)))
    return "\n".join(out) + "\n"
 class Handler(BaseHTTPRequestHandler):
    sys_dir = "/sys/class/drm"
    nvidia_cmd = DEFAULT_NVIDIA_CMD
    def do_GET(self):  # noqa: N802 (http.server API)
        if self.path.split("?")[0] not in ("/metrics", "/"):
            self.send_error(404)
            return
        body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; version=0.0.4")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args):
        pass
 def main(argv):
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
    ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
    ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
    ap.add_argument("--port", type=int, default=9401)
    ap.add_argument("--bind", action="append", default=None,
                    help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
                         "comma-list if unset. NEVER a wildcard.")
    args = ap.parse_args(argv)
    if not args.serve:
        sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
        return 0
    binds = args.bind
    if not binds:
        env = os.environ.get("GPU_EXPORTER_BINDS", "")
        binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
    Handler.sys_dir = args.amdgpu_sysfs
    Handler.nvidia_cmd = args.nvidia_cmd
    servers = []
    for b in binds:
        if b in ("0.0.0.0", "::", "*"):
            sys.stderr.write("refusing wildcard bind %s\n" % b)
            return 2
        servers.append(ThreadingHTTPServer((b, args.port), Handler))
    sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
    # EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
    # healthy in `ss` yet times out every scrape once its SYN backlog fills —
    # and the loopback deploy gate cannot see it. One thread per extra bind.
    for s in servers[1:]:
        threading.Thread(target=s.serve_forever, daemon=True).start()
    servers[0].serve_forever()
    return 0
 if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,35 @@
 # Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
 #
 # Install path: ~/.config/systemd/user/gpu-exporter.service
 # Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
 #
 # Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
 # AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
 # user unit, not a quadlet: it runs python3 straight from the checkout (a pull
 # updates it with no copy step), a hand-written user unit can be enabled where
 # the podman v1 generator's units cannot, and with Linger=yes for dank it starts
 # at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
 # it must run on the host — a container would need GPU/DRM passthrough for
 # nothing gained.
 #
 # BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
 # drop-in adding THIS host's LAN address (the same two-address posture
 # node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
 # unauthenticated metrics on the LAN, identical to node-exporter :9100 and
 # cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
 # the Dankserver instance over the LAN from Dankclaw.
 [Unit]
 Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
 After=network-online.target
 Wants=network-online.target
 [Service]
 # Loopback by default; the deploy drop-in appends the host LAN address.
 Environment=GPU_EXPORTER_BINDS=127.0.0.1
 ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
 Restart=always
 RestartSec=5
 [Install]
 WantedBy=default.target
@@ -147,10 +147,21 @@ def main():
    deep = corpus(160_000)
    wide = corpus(1_100_000)
-    # 3.28 chars/token is the measured ratio of this corpus through the
+    # The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
-    # froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
+    # (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
-    # deploy); 98.5 % of the window leaves the response room under the ceiling.
+    # 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
-    chars = int(0.985 * maxlen * 3.28)
+    # stale constant sends leg B past --max-model-len, where the engine answers a
    # perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
    # context size (200000)") and the leg reads as a capacity failure. One small
    # request prices the ratio; 97 % of the window leaves the response room.
    sample = prompt(wide, 4000, "CAL", "cal-marker")
    cal, _, _ = post({"model": MODEL,
                      "messages": [{"role": "user", "content": sample}],
                      "max_tokens": 1, "temperature": 0.0,
                      "chat_template_kwargs": {"enable_thinking": False}})
    ratio = len(sample) / cal["usage"]["prompt_tokens"]
    print(f"[cal] corpus ratio {ratio:.2f} chars/token")
    chars = int(0.97 * maxlen * ratio)
    # Each marker's FIRST hyphen-separated word is unique, because show()
    # scores the hit on that word — a shared prefix would score every slot HIT
    # on any answer that says the word.
@@ -176,7 +187,7 @@ def main():
                print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
    print(f"[B] total {time.time() - t0:.1f}s")
    # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
-    dchars = int(0.25 * maxlen * 3.28)
+    dchars = int(0.25 * maxlen * ratio)
    dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
               for i in range(seqs)]
    print(f"[D] {seqs} concurrent streams x 512 tokens, "
@@ -15,18 +15,21 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000
+# Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a
-# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in
+# >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache
-# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab
+# in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
-# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair
+# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
-# and the budget resolver derives it from headroom, the cache bound and the host
+# resolver derives it from headroom, the cache bound and the host pool, so the slab
-# pool, so the slab size and the pool's token count are LEDGER lines: after every
+# size and the pool's token count are LEDGER lines. After every boot read them, plus
-# boot read them out of the log and check the pool still backs 3 x 200,000.
+# the two lines that size a prefill step — "activation arena: ... lends ... while
 # steps are at most N tokens" (the arena levels the expert slab borrows) and
 # "prefill staging: ... a step of more than N tokens ..." (the chunk size below which
 # pooled experts are read synchronously) — and the cache's session backing.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
+# LimitMEMLOCK=infinity) grants the pin budget: 2 x 48 GiB expert pools + the 47.7 GiB PLE
-# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
+# table + 8 GiB prefix host tier + the checkpoint store (slots x 112 MiB) — ~159 GiB of
-# pages, which is ALSO why this script refuses to start when ttm's
+# pinned pages, which is ALSO why this script refuses to start when ttm's
 # pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
 set -euo pipefail
@@ -136,10 +136,17 @@ fi
 #      The engine re-checks size+sha256 against that catalog at boot; it cannot
 #      fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
 #      stays unset — no outbound connection at boot),
-#   2. the distro XRT's three libraries, each mounted twice (where the image
+#   2. the distro XRT's three libraries, each mounted THREE times: where the
-#      looks for XRT, and at its own path — the libraries find each other there;
+#      image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
-#      an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
+#      libraries find each other there), and at /opt/xilinx/lib/<multiarch>
-#      container, which the vendor calls out explicitly),
+#      — the NPU child searches that third prefix too, and since the host's
 #      libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
 #      names: omitting it crashes the lane with `No such library
 #      '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
 #      supervisor takes the dead lane down on the whole container (measured
 #      2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
 #      of symlinks into the system dir dangles inside the container, which the
 #      vendor calls out explicitly),
 #   3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
 #      hang the machine while FCLK changes speed, and the engine refuses to
 #      start when it moves — it cannot read host sysfs from inside the
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
            exit 1
        fi
        REAL="$(readlink -f "$XRT_DIR/$L")"
-        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
+        MULTIARCH="${XRT_DIR##*/}"
        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
                   -v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
    done
    HELD=0
    for D in "$SYSFS"/class/drm/card*/device; do
@@ -293,6 +293,14 @@ for path in "${CHANGED[@]}"; do
            add_test tests/caddyfile.bats
            add_test tests/deployment_scoping.bats
            ;;
        # The GPU exporter is a host process; its seam pins the collector, the
        # user unit, the deployer function, the planner route, the Prometheus
        # scrape jobs and the host-system GPU panels — a cross-file contract, so
        # the whole seam rides on any of its own files.
        scripts/gpu-exporter.py|scripts/gpu-exporter.service)
            add_test tests/gpu_exporter.bats
            add_test tests/deployment_scoping.bats
            ;;
        dankclaw/udev/*)
            # Host-level udev rules are applied by manual sudo install + udevadm
            # trigger, not dispatched as a service; nothing here is bats-testable.
@@ -52,28 +52,31 @@ Where the contract lives:
 The arithmetic (the coupled contract every endpoint's maxTokens comment
 encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
 endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
-follows is a **worked example** recomputed from the live files on 2026-10-18 —
+follows is a **worked example** recomputed from the live files on 2026-10-21 —
-a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
+a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
 maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
-   95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
+   95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
   bundle clamps the percentage to 1–99 and converts it to a token threshold.
 2. **Output budget** = capped at the target model's `maxTokens` — every role
   here compacts to frank-zealot, so the budget is its 16,384.
-3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
-   window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
+   dragoon session the fitting window is the ENGINE ceiling, not a slot: the
-   200,704. The registry's own comments design around an 85 % trigger (archon:
+   190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
-   170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
+   error plus the 45,056 request budget lands inside — that headroom is the
-   does hold. While `config.yml` and those comments disagree, treat 85 % as the
+   reason the ceiling moved to native on 2026-10-21. For a session compacting
-   design point and re-run this arithmetic after ANY window change — a 400 from
+   INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
-   the compaction target mid-conversation is the symptom.
+   does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
   200,704. Treat 85 % as the design point until config.yml and the registry
   comments agree, and re-run this arithmetic after ANY window change — a 400
   from the compaction target mid-conversation is the symptom.
   (Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
   177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
   32,768 ≤ 200,704 zealot.)
 Procedure (recompute from the LIVE files every time — windows rotate with
-every swap; the block above is the 2026-10-18 state):
+every swap; the block above is the 2026-10-21 state):
 1. Read the `config.yml` compaction block + `models.yml` entries for source
   and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
 as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
 dankclaw — the comment/config disagreed. The comments now name the real
 target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
-truth; fix comments in the same change.** A fresh instance of the same trap is
+truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
-open 2026-10-18: every `maxTokens` comment still says "compaction fires at
+disagreement is not academic: it is why long dragoon sessions kept drawing
-85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
+400s at the old 200,704 ceiling — the trigger fires late AND the est
 under-counts, and a window parked near the ceiling has no room for either.
 `maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
 input. Boot-shape OOM laddering is a different problem
 ([[Llama Cpp Serving Sizing]]).
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
 tags:
  - dankserver
  - plex
-timestamp: 2026-09-15
+timestamp: 2026-10-21
 ---
 # Plex
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Item | Value |
 |------|-------|
 | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
-| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
+| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
 | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
 | Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
@@ -2,7 +2,7 @@
 type: reference
 title: "Observability on Dankclaw"
 description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
-tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
+tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
 timestamp: 2026-10-20
 ---
@@ -10,10 +10,10 @@ timestamp: 2026-10-20
 ## Summary
-Seven containers and one host-side process: Prometheus (metrics), Grafana
+Seven containers and two host-side processes: Prometheus (metrics), Grafana
 (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
-metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
+metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
-carrying every firing group to Signal. All of it is Quadlet-managed,
+plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
 boot-persistent and loopback-only, deployed by
 `scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
 end to end; the log half is configured, healthy, and carries no data.
@@ -35,6 +35,7 @@ that proves it today.
 | Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
 | node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
 | cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
 | GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
 | Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
 Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -45,10 +46,11 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
 the same day, and `homepage-status.timer` — the root unit that probes all of
 these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
 the probe registry `shared/homepage/services.json` to `/srv/homepage`).
-Dankserver runs two members of this family, node-exporter and cadvisor, as
+Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
-its own units (`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
+(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
 `generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
-cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`).
+cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
 `gpu-exporter` (a user unit installed the same way as `signal-relay`).
 ## Exposure
@@ -65,10 +67,18 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
 in the host network namespace, and it binds exactly two addresses rather than
 the wildcard — see [[Node-Exporter]].
-The relay's bind is the one address a bridge container can reach on the host:
+**Same-host reachability from a bridge container is measured, not inferred**
-`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
+(2026-10-21, from inside the Prometheus container): `host.containers.internal`
-and the podman bridge subnet `10.89.0.0/16` before the DROP
+(the podman gateway) reaches host services whether they bind host-net or are
-(`scripts/deploy-remote.sh:255-262`).
+published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
 — while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
 every same-host Prometheus target uses the gateway and only cross-host targets
 (dankserver) use plain LAN. Whether LAN works at all is network-specific: the
 relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
 iptables INPUT admitting loopback, the host itself and the podman bridge subnet
 `10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
 point a scrape target at a host LAN IP from its own host on the strength of
 one working example; dial the gateway and probe.
 ## Deployment
@@ -117,6 +127,18 @@ verify it through the APIs.
  `container` label. Network and filesystem are NOT exported for these containers
  (no netns/classid attribution under the host-PID layout), so the dashboard
  graphs only CPU, memory and OOM and does not fake the rest.
 * **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
  `deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
  host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
  own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
  without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
  `mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
  emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
  °C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
  job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
  `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
  the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
  node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
 * **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
  a rootless Alloy cannot read the `0640` journal files, and the container-log
  globs point at a root-storage layout that does not exist here. The fix is built
@@ -135,6 +157,7 @@ verify it through the APIs.
 - `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
 - `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
@@ -239,45 +239,92 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
 sequence stops producing entirely and then re-prefills from token 0, which is a
 stall, not sludge.
-Knobs, in the order worth trying them (radiance GUIDE §4.5):
+**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
 looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
 the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
 (steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
 decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
 with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
 14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
 a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
 worth ~10 % of one step — not a lever.
- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
+**Four traps in the PromQL above.** Learn them before trusting a bucket:
-  steps carry decode rows only and admit nothing. Decoders keep their cadence; the
+- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
-  prompt pays that in TTFT. Counts as `engine_steps_held_total`.
+  histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
+  is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
-  `auto` (default) reserves a row plus the draft window per decode row behind it and
+  *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
-  an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
+  endpoint never stalls anyone".
-  whole step (admission never runs while it does), `N` pins the share. Never under
+- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
-  the prefill stager's smallest pass — the startup line names it, and below it every
+  `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
-  routed layer reads its pooled experts synchronously, which costs the prefill far
+  scrape in eight — use step ≤ 30 s for a bucket table.
-  more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
+- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
+  one shape (`up`, or the first sample of a counter that only the current build exports) before
-  curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
+  ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
-  at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
+  turned out to belong entirely to the older one.
-  magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
+- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
-  expert residency to the arena.
+  above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
 - `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
  shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
  back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
  at short prompts, a large TTFT gain at the far end of the window.
-Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
+Three instruments the PromQL above cannot reach:
 expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
 decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
 And **re-check
 `prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
 pool are the same memory, and the failure looks like turn two taking as long as turn
 one.
-Availability is a version question, not a config question: `--busy-prefill-chunk` and
+- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
-`--prefill-interleave` are absent from older pins — prove it on the running container
+  `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
-(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
+  prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
-tree, and check the source clone's depth before dating a flag from `git log -S`
+  the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
-(a `--depth 1` clone reports its own tip for every search).
+  and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
  configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
 - **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
  (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
  cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
  prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
  prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` -
  exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner:
  [[frank-archon]].
 - **Bucket the arriving request too** — `histogram_quantile` over
  `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
  sequences that are already running; the one waiting for its first token pays the same steps and
  shows in no decode number.
-Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
+Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
-interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
+
-for this fleet's shape are that page's.
+- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
  rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
  cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
  Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
 - `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
  setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
  of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
  alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
  `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
  stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
  `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
  every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
  decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
  chunk blindly buys latency in both directions.
 - `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
  carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
  measurement says a span starts to win), while **every other step is planned at the shared
  budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
  knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
  pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
 - `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
  arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
  decode — [[frank-archon]]).
 Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
 reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
 quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
 slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
 arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
 taking exactly as long as turn one.
 Availability is a version question, not a config question: `--busy-prefill-chunk`,
 `--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
 container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
 `configuration` block, never from a vendored tree, and check a source clone's depth before dating a
 flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
 Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
 ## Related
@@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
 |---|---|
 | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
-| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
+| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
 | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
 | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
-| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
+| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
 | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
 | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
 | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
 | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
-| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
+| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
-| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
+| TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
 | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
-Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
+Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
-the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
+the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
-`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
+`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
-**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
+The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
-2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
+sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
-(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
+per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
-(≤ 4,194,304 px, 16,384 patches/pass).
+`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
 host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
 (= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
 (≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
 directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
 the bigger cache, paid over PCIe when those experts are hit.
 The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
 anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -145,6 +150,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
 | 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
 | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
 | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
 | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
@@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
  never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
  buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
  the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
+- **At 1.8.1 the split belongs to the resolver, and the source fixes its order.**
-  The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
+  `core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena →
-  `--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
+  staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a
-  sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
+  floor, not a share**, and divides the `elastic` remainder in three lines (146-156):
  hand. Three boots of evidence:
-  | shape | demand | cache given | backing |
+  ```cpp
  exp = std::min(b.expert_floor, b.elastic);
  kv  = std::min(kv_cap, std::max(b.elastic - exp, kv_one));
  exp = std::min(std::max(b.expert_ceiling, b.expert_floor), b.elastic - kv);
  ```
  `kv_cap` is `KVManager::ceiling()` = `state + Σ_paged max_seqs × ceil(max_ctx / block) × bytes`
  (`core/mem/kv.cpp:266`), so `--max-num-seqs` and `--max-model-len` set the cache's **demand**, never
  its funding. The funding is `elastic − expert_floor`, and the floor is
  `min(plane + spare, plane + spare + 2·unit_max − host_room)` (`core/place/planner.cpp:100-104`, with
  `host_room = host_pool − file_stage_reserve` at `core/engine_bringup.cpp:1418-1422`). **That is the
  cache's lever: every MiB added to `--host-pool-mib` drops the expert floor one-for-one and the cache
  takes it** — minus `stage`, the prefill stager's ring (two layers of the experts left off the card),
  which is charged back out of the cache whenever the cache lends (`vram_budget.cpp:193-203`). Boots of
  evidence, all on 1.8.1:
  | shape | demand | cache given | engine's `backs %` |
  |---|---|---|---|
-  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
+  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
  | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
-  | 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
+  | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
-  So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
+  The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
-  never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
+  `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
-  slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
+  ledger line `kv: the pool backs N % … the cache can grow to … N tokens`, never arithmetic.** To buy
-  grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
+  window: raise `--host-pool-mib` (cost: pinned host RAM, and experts evicted to host cross PCIe —
-  `--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
+  `experts-from-host` was 5.99 GiB a rank at 44,928 and is 8.99 at 49,152) or cut the per-token cost
-  hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
+  (`fp8` KV). Raising slots raises demand **and** costs 10 × 111.98 MiB of pinned checkpoint-store RAM
-  era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
+  per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in
-  the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
+  the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape:
-  `--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
+  `/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused,
  and the 400 class is a single request over `--max-model-len`.
 - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
  slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
  256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
@@ -280,41 +303,104 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
  in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
  GiB, refused).
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
+- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not
-  3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
+  protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s,
-  1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
+  `engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` /
-  at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
+  `prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
-  chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
+  chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
-  decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
+  step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the
-  samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
+  harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s
-  `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
+  scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light
-  bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
+  prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two
-  0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
+  sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
-  busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
+  else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
-  confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
+  17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from
-  resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
+  30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
-  [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
+  victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
-  and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
+  and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
+  median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
-  shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
+  `vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
-  the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
+  chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
-  step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
+  and `busy_cap` is the fourth law.
-  the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
+- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves
-  The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
+  `1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
-  knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
+  budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
-  edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
+  "not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
-  it is trusted.
+  the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
+  anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live
-  adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
+  proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
-  declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
+  with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies
-  (single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
+  "whenever the chunk shares the step".
-  today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
+- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
-  floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
+  ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
-  moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
+  `max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode
-  the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
+  rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
-  mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
+  prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
-  retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
+  the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
-  fp8-native start.
+  routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
  pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
  vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
  beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
 - **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
  lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
  it **synchronously before the step** — units revert to their host copies (an address change, not a
  copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big
  step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
  with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics`
  says the binding constraint in one line: "a prefill step streams every routed layer's pooled
  experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
  Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
 - **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
  N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
  costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
  two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
  ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
  measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
  substitute for it.
 - **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
  intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
  intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
  starts to win), while **every other step is planned at the shared budget of 2048** — which is a
  lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot
  express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
  cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
  span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
  second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at
  75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
  does not boot at 3 slots; this shape is 2.
 - **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
  the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
  replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
  **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
  `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
  `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as
  deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
  · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
  `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the
  cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly
  **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs
  retention only - the writes happen at every interval whatever the policy says (only `--prefill-span`
  skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
  alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB
  against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` -
  the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
  Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a
  chain that keeps resuming at one deep point stops paying for it. Shortening
  `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
  `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
  interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
 - **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance
  constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store
  RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over
  `seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so
  single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises
  the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic −
  expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured
  2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured
  2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of
  expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old
  "seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
  specialization, and radiance booted seqs 2 on its first fp8-native start.
 - **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
  of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
  pinned weights, and the doubling is invisible in the weight count. Read the engine log for
@@ -396,12 +482,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
  4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
  theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
  60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
-  software theory.
+  software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo
  dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
  between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
  before crediting any software revert with a fix — and before believing a clean run of the arm
  you just tested, because the confounder is still in the slot.
 - **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
  `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
  all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
  them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
  decode, `exact` is the bit-exact alternative.
 - **The link is already at the board's ceiling — do not chase the `link-below-max` warning.**
  Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
  the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states.
  `/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its
  x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
  side of the chain is x8 max, which the same endpoint reports two findings later as
  `link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
  the only lever on a link-bound prefill is **less traffic over the link** (more expert
  residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the
  driver, the two cards reach each other's memory), so the collectives do not hairpin through host
  RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
  all-reduce wins the link, the staged experts wait.
 - **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
  cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre
  and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured
  during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU
  contention is not the current decode culprit**, so do not spend an experiment window on it;
  re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
  different from the step-sharing one.
 - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
  the disk tier silently never opens. The tier does survive a payload swap when arch and
  tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
@@ -445,11 +554,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 ## Contract (the wire)
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
-  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
+  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health`
-  JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
+  answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while
-  (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
+  stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load
-  `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
+  (`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses
-  bare `curl -fsS …/health` gate is honest and needs no body grep.
+  connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep.
 - **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
  returns `configuration` (every flag, so drift from the env file is checkable without the host),
  `environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card
  `link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings`
  (level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses
  PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`,
  `tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is
  structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
 - **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
  only the alias; container, unit, scripts and env keep the service name). The env's contract
  line is that **radiance validates the requested model name, so every client — harness, bench
@@ -476,12 +593,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
  and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
  — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
 - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
-  longer than the KV pool's token count is accepted then refused as too large to fit; under the
+  longer than the KV pool's token count is accepted then refused as too large to fit; a request
-  3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
+  beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here)
  and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height
  sessions, so four conversations can be resident but a fourth near-full-height one can still
  **preempt a running sequence from the back** to free its blocks
  (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then
  re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`);
  `radiance:num_requests_preempted` is only the instant depth.
 - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
-  Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
+  Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
-  published-port quirk is same-host only). The boot ledger — the referee for every shape number
+  netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
-  above — is in the container log (`podman logs frank-archon`)
+  `radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
  running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only
  query misses every latency answer. The step-phase counters are all scraped:
  `engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`,
  `engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`,
  `prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the
  engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
  `token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
  prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
  `/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape
  number here — is in the container log (`podman logs frank-archon`);
  `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
 ## Files
@@ -10,14 +10,15 @@ tags:
  - gfx1151
  - halogen
  - endpoint
-timestamp: 2026-10-19
+timestamp: 2026-10-21
 ---
 # frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
 Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
-(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
+(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
 This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
 ## Current shape
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
 unauthenticated token port 8730 stays on container loopback. Engine stdout →
 `/home/dank/logs/frank_dragoon.log`, **not** the journal.
- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
+- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
-  `HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
+  2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
-  enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  `HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
-  exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
+  (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
  exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
  conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
 - Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
  omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
  pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
  (Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
  ~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
  Caddy/observability).
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
+- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
-  (= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
+  — sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
-  `reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
+  `tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
-  rides [[frank-zealot]].
+  together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
  `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
 - Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
  pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
  Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
@@ -124,14 +128,18 @@ created it — earlier topologies in .
  prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
  85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
  re-measure the warm ratio after any pool change.
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
+- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
-  155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
+  clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
-  its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
+  traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
-  3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
+  thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
-  over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
+  window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
-  target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
+  theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
-  is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
+  At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
-  under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
+  200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
  both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
  ≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
  `tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
  from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
 - **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
  503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
  together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
   (`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
   compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
   return as a crash-loop.
-2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
+2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
-   container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
+   distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
-   `/usr/lib/x86_64-linux-gnu`, where all three exist.
+   `libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
   symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
   searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
   2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
   engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
   A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
 3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
   while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
   engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**
@@ -18,6 +18,16 @@
        "includeAll": true,
        "multi": true,
        "current": { "text": "All", "value": "$__all" }
      },
      {
        "name": "gpuhost",
        "label": "gpu host",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "prometheus" },
        "query": { "query": "label_values(gpu_utilization_percent, host)", "refId": "A" },
        "includeAll": true,
        "multi": true,
        "current": { "text": "All", "value": "$__all" }
      }
    ]
  },
@@ -139,6 +149,56 @@
        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
      ]
    },
    {
      "id": 13, "type": "timeseries", "title": "GPU busy %",
      "description": "Per-GPU utilization from gpu-exporter (amdgpu gpu_busy_percent / nvidia-smi). The picker above selects the host; each series is one card.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 36 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
      "targets": [
        { "expr": "gpu_utilization_percent{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
      ]
    },
    {
      "id": 14, "type": "timeseries", "title": "GPU VRAM used",
      "description": "Device memory in use. On the R9700s (frank-archon) and the 3090 (frank-zealot) this is the loaded model's footprint.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 36 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] },
      "targets": [
        { "expr": "gpu_memory_used_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
      ]
    },
    {
      "id": 15, "type": "timeseries", "title": "GPU VRAM used %",
      "description": "Used / total device memory. Near 100% is the pre-OOM wall.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 44 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
      "targets": [
        { "expr": "100 * gpu_memory_used_bytes{host=~\"$gpuhost\"} / gpu_memory_total_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
      ]
    },
    {
      "id": 16, "type": "timeseries", "title": "GPU temperature",
      "description": "Package temperature. AMD edge temp (hwmon temp1_input); NVIDIA gpu temp.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 44 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "celsius", "min": 0 }, "overrides": [] },
      "targets": [
        { "expr": "gpu_temperature_celsius{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
      ]
    },
    {
      "id": 17, "type": "timeseries", "title": "GPU power draw",
      "description": "Board power draw. Emitted only where readable: NVIDIA and the Dankclaw iGPU. The R9700s have no sysfs power sensor, so they are simply absent here — never a fabricated 0.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 52 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "watt", "min": 0 }, "overrides": [] },
      "targets": [
        { "expr": "gpu_power_watts{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
      ]
    }
  ]
 }
@@ -140,12 +140,15 @@ scrape_configs:
  # system.slice daemons and machine.slice scopes carry no such name and are
  # dropped, as is the per-CPU-core split (only cpu="total" survives), so the
  # scrape stays small. cadvisor cannot bind per-interface, so it is reached at
-  # the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
+  # the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
-  # is why node-dankclaw scrapes via host.containers.internal), a published port
+  # DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
-  # IS reachable from this bridge container at the host LAN address.
+  # connection whether the port is a host-net bind or a published port (the
  # original note here claimed published ports were exempt; measured 2026-10-21
  # from inside this container: LAN refused, gateway answers). Cross-host
  # targets (dankserver) have no hairpin problem and stay plain LAN.
  - job_name: "cadvisor-dankclaw"
    static_configs:
-      - targets: ["192.168.0.143:8080"]
+      - targets: ["host.containers.internal:8080"]
        labels:
          host: dankclaw
    metric_relabel_configs:
@@ -191,3 +194,25 @@ scrape_configs:
        action: keep
      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
        action: labeldrop
  # Per-GPU metrics (gpu_utilization_percent / gpu_memory_used_bytes /
  # gpu_memory_total_bytes / gpu_temperature_celsius / gpu_power_watts) from the
  # host-process gpu-exporter (scripts/gpu-exporter.py). One job per host, same
  # reach rules as node/cadvisor: Dankclaw is scraped at the podman gateway
  # because the netavark hairpin quirk makes Prometheus fail to dial its own
  # host LAN IP; Dankserver is plain LAN. The gpu label (PCI BDF on AMD, index on
  # NVIDIA) + vendor/model come from the exporter, so the GPU row on the
  # host-system dashboard filters by host. Power is emitted only where readable
  # (NVIDIA + the Dankclaw iGPU); the R9700s have no sysfs power, so they simply
  # have no gpu_power_watts series — no fabricated zero.
  - job_name: "gpu-dankclaw"
    static_configs:
      - targets: ["host.containers.internal:9401"]
        labels:
          host: dankclaw
  - job_name: "gpu-dankserver"
    static_configs:
      - targets: ["192.168.0.235:9401"]
        labels:
          host: dankserver
@@ -125,3 +125,28 @@ load helpers
    [ ! -e "$sandbox/frank-default-sightsound" ]
    [ -e "$BATS_TEST_TMPDIR/.AGENTTRASH/$(date +%Y%m%d)-retired-deploy-state/frank-default-sightsound" ]
 }
@test "the client and remote allowlists agree per target" {
    # Two gates guard `deploy.sh --service X`: the client's valid_service and
    # the remote runner's service_is_allowed. When only one side gained a
    # service the other answered "is not deployed on <target>" and the only
    # repair path was a fleet-wide drain — cadvisor and gpu-exporter both fell
    # through this crack within a day of each other. The lists MUST agree;
    # searxng-network is the one sanctioned exception (client-only network
    # unit, no remote dispatch handler).
    run env OUT="$BATS_TEST_TMPDIR" bash -c '
        grep -oE "dank(claw|server):[a-z0-9-]+" \
            <(sed -n "/^valid_service()/,/^}/p" "$1/scripts/deploy.sh") \
            | grep -v "dankclaw:searxng-network" | sort -u > "$OUT/cli"
        grep -oE "dank(claw|server):[a-z0-9-]+" \
            <(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
            | sort -u > "$OUT/rem"
        diff "$OUT/cli" "$OUT/rem"
    ' _ "$REPO_ROOT"
    [ "$status" -eq 0 ] || { echo "deploy.sh vs deploy-remote.sh allowlist drift:"; echo "$output"; return 1; }
    # Drift aside, the gate must actually know the host-process exporters —
    # their first-ever deploy has to pass the client gate.
    for pair in dankclaw:cadvisor dankclaw:gpu-exporter dankserver:cadvisor dankserver:gpu-exporter; do
        grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
    done
 }
@@ -145,7 +145,7 @@ nosudo_bin() {
    # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
    # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
    # tied inside the +-3% noise floor — slots buy concurrency, not speed.
-    grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
@@ -158,7 +158,7 @@ nosudo_bin() {
    # so the pool's token count is a boot-ledger line — verified live by the
    # capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
    grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
-    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=49152$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
    # Prefix cache: pinned host RAM only (the disk tier was removed), and the
@@ -177,8 +177,11 @@ nosudo_bin() {
    [ "$ttmt" -gt "$ttmm" ]
    pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
    ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE")
-    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
+    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host + MNBT + the auto
-    ledger=$(( 2 * pool + 48824 + ph + 2048 ))
+    # checkpoint store (MAX_SEQS x 10 slots of 112 MiB -- the store is pinned host
    # memory too, and it grows with the slot count).
    seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
    ledger=$(( 2 * pool + 48824 + ph + 2048 + seqs * 10 * 112 ))
    [ "$ttmm" -ge "$ledger" ]
    # The .rad size floor is positive (a truncated convert must fail fast).
    grep -q '^FRANK_ARCHON_RAD_MIN_MIB=' "$ENV_FILE"
@@ -3,7 +3,7 @@
 # of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
 # Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
 # digest-pinned), driven by the root unit's serve script — 2 slots over ONE
-# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
+# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
 # and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
 # shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
 #
@@ -85,7 +85,7 @@ good_env() {
        "FRANK_DRAGOON_VISION_BYTES=0"
        "FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
        "HALOGEN_MODEL_ID=frank-dragoon"
-        "HALOGEN_CTX=200704"
+        "HALOGEN_CTX=262144"
        "HALOGEN_KV_POOL_POSITIONS=401408"
        "HALOGEN_KV_SLOTS=2"
        "HALOGEN_SPEC_ADAPT=0"
@@ -159,9 +159,11 @@ good_env() {
    reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
    # The pool is ONE budget behind the slots (boot ledger form: "2 slots over
    # ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
-    # a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
+    # a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
-    # (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
+    # native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
-    # resident at once — frank-zealot parity on total in-flight context. What
+    # MAXIMUM harness-length (200,000-token) conversations are resident at once;
    # two literal ceiling-length requests would queue — widening that is a pool
    # raise, not a window move.
    # still binds: the pool must back ONE full-length request (below that the
    # advertised per-request ceiling would be unreachable, hence pool >= ctx),
    # it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
    [[ "$output" == *"--name frank-dragoon-halogen"* ]]
    [[ "$output" == *"--publish 7770:8731"* ]]
    [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
-    [[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
+    [[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
    [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
    [[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
    [[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"
@test "the shipped env keeps the harness fit law inside the ceiling" {
    # HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
-    # naming the limit, never a silent truncation). The law is INVARIANT to
+    # naming the limit, never a silent truncation). The law: the harness
-    # the value: the harness window must sit ~45k UNDER it, because the
+    # window must sit ~45k UNDER the ceiling, because the harness clamps
-    # harness clamps max_tokens = min(45056, window − est − 4096), compacts
+    # max_tokens = min(45056, window − est − 4096), compacts at the trigger
-    # at 85 % of its declared window, and its chars/4 est UNDER-counts the
+    # percent of its declared window, and its chars/4 est UNDER-counts the
-    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
+    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
-    # est 152,938 on 2026-10-07 — with window == ceiling, every request over
+    # at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
-    # ~65 % of the window was a guaranteed 400 before compaction could fire).
+    # request is a guaranteed 400 before compaction can fire). The 2026-10-21
-    # The shipped pair is ctx 200704 / window 155648 (lowered from
+    # pair: ceiling 262144 (native) under a harness window of 200,000.
-    # 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
+    # The declared window is read from the endpoint page — the one owner —
-    # monotone maxima of est + budget (est ≤ window − 4096):
+    # so env, models.yml and page that disagree FAIL HERE, not as a 400
-    #   full budget at the compaction threshold: 1.03*132300 + 45056
+    # mid-session (the recurring failure this seam exists for).
-    #   clamped budget at the clamp's zero point: 1.03*151552 + 0
+    grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
-    # and the compaction request that rides on frank-zealot (its
+    ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
-    # 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
+    page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
-    grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
+    grep -qE 'contextWindow: 200000' "$page"
-    ctx=200704
+    hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
-    hw=155648; mt=45056
+    mt=45056
-    thr=$(( hw * 85 / 100 ))
+    # Invariant checked at the live 95 % trigger (config.yml
    # thresholdPercent) — strictly stronger than the 85 % design point.
    # Both monotone maxima of est + budget (est ≤ window − 4096):
    thr=$(( hw * 95 / 100 ))
    [ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
    zero=$(( hw - 4096 ))
    [ $(( zero * 103 / 100 )) -le "$ctx" ]
    # The compaction request rides on frank-zealot's 200,704-token slot:
    # input ≈ threshold − keepRecent(20k).
    [ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
 }
@@ -423,9 +430,12 @@ npu_fixture() {
    # the stock root:render 0660 mode, and the serve gate checks exactly that.
    touch "$TEST_TMP/accel0"
    chmod 666 "$TEST_TMP/accel0"
-    mkdir -p "$TEST_TMP/xrt"
+    # basename carries a multiarch shape: the third mount prefix is derived
    # from the XRT dir's basename (the NPU child searches
    # /opt/xilinx/lib/<multiarch>/, measured 2026-10-21).
    mkdir -p "$TEST_TMP/x86_64-linux-gnu"
    for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        touch "$TEST_TMP/xrt/$l"
+        touch "$TEST_TMP/x86_64-linux-gnu/$l"
    done
    mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
    mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
@@ -435,7 +445,7 @@ npu_fixture() {
        "HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
        "FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
        "FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
-        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/xrt"
+        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/x86_64-linux-gnu"
        "FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
    )
 }
@@ -465,11 +475,15 @@ $(cat "$TEST_TMP/serve.log")"
    ! grep -q 'accel0' "$TEST_TMP/serve.log"
 }
-@test "the lane arms the accel node, the staged model dir and each XRT library twice" {
+@test "the lane arms the accel node, the staged model dir and each XRT library three times" {
-    # Each library mounts at the image's XRT path AND at its own path: the
+    # Each library mounts at the image's XRT path, at its own path (the distro
-    # distro libraries find each other where they were installed, and an
+    # libraries find each other where they were installed), and at
-    # /opt/xilinx/xrt of symlinks into the system dir dangles inside the
+    # /opt/xilinx/lib/<multiarch>/ — the prefix the NPU child searches and
-    # container (the vendor calls that out by name).
+    # names when it is missing (`No such library
    # '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'`, measured
    # 2026-10-21: the lane died and the supervisor took the whole container
    # with it, 88-restart loop). An /opt/xilinx/xrt of symlinks into the
    # system dir dangles inside the container (the vendor calls that out).
    podman_stub
    good_env
    npu_fixture
@@ -484,8 +498,9 @@ $(cat "$TEST_TMP/serve.log")"
    [[ "$output" != *":/models/npu"* ]]
    [[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
    for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:$TEST_TMP/xrt/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:$TEST_TMP/x86_64-linux-gnu/$l:ro"* ]]
        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/lib/x86_64-linux-gnu/$l:ro"* ]]
    done
    # The Flash payload still rides the same container: the lane is additive.
    [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
@@ -560,7 +575,7 @@ $(cat "$TEST_TMP/serve.log")"
    podman_stub
    good_env
    npu_fixture
-    rm "$TEST_TMP/xrt/libxrt_driver_xdna.so.2"
+    rm "$TEST_TMP/x86_64-linux-gnu/libxrt_driver_xdna.so.2"
    run_with_log
    [ "$status" -ne 0 ]
    [[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]
@@ -0,0 +1,227 @@
 #!/usr/bin/env bats
 # Seam: the GPU metrics exporter's black-box contract.
 #
 # The exporter reads amdgpu sysfs (one card per /sys/class/drm/cardN that has a
 # device/gpu_busy_percent file) and, on the box that has a working NVIDIA stack,
 # one `nvidia-smi --query-gpu` CSV, and emits the unified `gpu_*` Prometheus
 # families the host-system dashboard graphs. Both sources are injectable so the
 # whole thing is testable with fixtures on a GPU-less dev box.
 #
 # Only the inputs a consumer sees are pinned: the metric families, their label
 # set, and that absent hardware yields ABSENT series (never a fabricated 0 — a
 # wrong number on a dashboard gets believed). The R9700s expose no sysfs
 # power1_average, so those cards legitimately carry no gpu_power_watts line.
 load helpers
 EXPORTER="$REPO_ROOT/scripts/gpu-exporter.py"
 # Build a fixture amdgpu sysfs tree. $BATS_TEST_TMPDIR/sys
 mk_amd() { # gpu busy vram_used vram_total power_uw_or_blank temp_mc
    local card=$1 busy=$2 used=$3 total=$4 power=$5 temp=$6
    local d="$SYS/card$card/device"
    mkdir -p "$d/hwmon/hwmon9"
    printf '%s\n' "$busy" > "$d/gpu_busy_percent"
    printf '%s\n' "$used" > "$d/mem_info_vram_used"
    printf '%s\n' "$total" > "$d/mem_info_vram_total"
    printf '%s\n' "$power" > "$d/power1_average"          # blank => absent on R9700
    [ -n "$temp" ] && printf '%s\n' "$temp" > "$d/hwmon/hwmon9/temp1_input"
    printf 'DRIVER=amdgpu\nPCI_ID=1002:7551\nPCI_SLOT_NAME=0000:03:00.0\n' > "$d/uevent"
 }
 setup() {
    SYS="$BATS_TEST_TMPDIR/sys"
    mkdir -p "$SYS"
 }
@test "exporter exists and compiles" {
    [ -f "$EXPORTER" ]
    python3 -m py_compile "$EXPORTER"
 }
@test "amd card with full sysfs yields util/vram/temp/power with the right labels and units" {
    mk_amd 1 42 1073741824 34359738368 250000000 71000   # 250 W, 71 C
    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
    # util is a raw percent, VRAM verbatim bytes, temp milli->C, power microW->W.
    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 42$' || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 1073741824$' || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_memory_total_bytes\{[^}]*\} 34359738368$' || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*\} 71$' || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*\} 250$' || { echo "$output"; return 1; }
    # identity labels: the PCI slot is a stable unique id, vendor marks the family.
    echo "$output" | grep -Eq 'gpu_utilization_percent\{.*gpu="0000:03:00\.0"' || { echo "$output"; return 1; }
 }
@test "card with an empty power file and no hwmon yields util/vram but omits power and temp" {
    mk_amd 1 99 34108338176 34208743424 "" ""           # the R9700 shape
    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*\} 99$' || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 34108338176$'
    # Absent means ABSENT, not a zero the dashboard would plot as a real reading.
    ! echo "$output" | grep -Eq '^gpu_power_watts\{' || { echo "fabricated power line: $output"; return 1; }
    ! echo "$output" | grep -Eq '^gpu_temperature_celsius\{' || { echo "fabricated temp line: $output"; return 1; }
 }
@test "nvidia csv yields util/vram/temp/power tagged vendor=nvidia" {
    nvidia="$BATS_TEST_TMPDIR/nvidia-smi"
    cat > "$nvidia" <<'STUB'
 echo '0, NVIDIA GeForce RTX 3090, 93, 23004, 24576, 76, 397.40'
 STUB
    chmod +x "$nvidia"
    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$nvidia"
    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="nvidia"[^}]*\} 93$' || { echo "$output"; return 1; }
    # memory.used MiB -> bytes (23004 * 1048576)
    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*vendor="nvidia"[^}]*\} 24121442304$' || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*vendor="nvidia"[^}]*\} 76$'
    echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*vendor="nvidia"[^}]*\} 397\.4$'
    echo "$output" | grep -Eq 'model="NVIDIA GeForce RTX 3090"'
 }
@test "a failing nvidia command is tolerated: amd still emits and exit is 0" {
    mk_amd 1 50 1000000 2000000 "" 40000
    failing="$BATS_TEST_TMPDIR/nvidia-fail"
    printf '#!/bin/sh\nexit 1\n' > "$failing"; chmod +x "$failing"
    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$failing"
    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 50$'
    ! echo "$output" | grep -q 'vendor="nvidia"'
 }
@test "no GPUs at all yields only HELP/TYPE headers and exits 0" {
    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
    # families are declared even when empty, so the dashboard/queries stay valid.
    echo "$output" | grep -q '# TYPE gpu_utilization_percent gauge'
    echo "$output" | grep -q '# TYPE gpu_memory_used_bytes gauge'
    ! echo "$output" | grep -Eq '^gpu_utilization_percent\{'
 }
@test "output is valid Prometheus text" {
    mk_amd 1 42 1000 2000 "" 50000
    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
    [ "$status" -eq 0 ]
    # every non-comment line is "<name>{labels} value" or "<name> value"
    printf '%s\n' "$output" | python3 -c '
 import sys, re
 pat = re.compile(r"^[a-z_]+(\{[^}]*\})? [-+0-9.eE+]+$")
 for i, l in enumerate(sys.stdin):
    l = l.rstrip("\n")
    if not l or l.startswith("#"):
        continue
    assert pat.match(l), "malformed line %d: %r" % (i, l)
 print("prometheus text ok")
 '
 }
 # ---- deployment + wiring contracts -------------------------------------
 # The exporter is a HOST-PROCESS user unit (not a quadlet): it must read the
 # host's own DRM sysfs and shell to nvidia-smi, which a container could not do
 # without GPU passthrough. These tests pin the cross-file wiring a reader
 # cannot eyeball — a lone change on one side leaves the metric silently absent.
@test "the base unit runs the exporter from the checkout and never binds a wildcard" {
    local u="$REPO_ROOT/scripts/gpu-exporter.service"
    [ -f "$u" ]
    grep -q '^ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve$' "$u" \
        || { echo "unit does not run the exporter --serve from the checkout"; return 1; }
    grep -q '^Restart=always$' "$u"
    grep -q '^WantedBy=default.target$' "$u"
    # Base unit binds loopback only; the per-host LAN address is added by a
    # deploy drop-in, so a checked-in unit is never a wildcard.
    grep -q '^Environment=GPU_EXPORTER_BINDS=127.0.0.1$' "$u"
    ! grep -qE '0\.0\.0\.0' "$u"
 }
@test "the deployer installs, drops in the LAN bind, and health-gates gpu-exporter" {
    local handler="$REPO_ROOT/scripts/deploy-remote.sh"
    grep -q '^deploy_gpu_exporter() {' "$handler"
    # Per-host LAN from $TARGET (never a hardcoded host, never a wildcard).
    grep -q 'dankclaw) lan=192.168.0.143' "$handler"
    grep -q 'dankserver) lan=192.168.0.235' "$handler"
    # Drop-in binds loopback + this host's LAN as one comma list.
    grep -q "GPU_EXPORTER_BINDS=127.0.0.1,%s" "$handler"
    grep -q 'systemctl --user enable gpu-exporter.service' "$handler"
    grep -q 'systemctl --user restart gpu-exporter.service' "$handler"
    # Health gate waits on the loopback metrics endpoint.
    grep -q 'http://127.0.0.1:9401/metrics' "$handler"
    # Dispatched from the central case.
    grep -q 'gpu-exporter) deploy_gpu_exporter ;;' "$handler"
    # Allowlisted on BOTH hosts.
    grep -q 'dankclaw:gpu-exporter' "$handler"
    grep -q 'dankserver:gpu-exporter' "$handler"
 }
@test "the planner selects gpu-exporter for its files on each target" {
    local t f
    for t in dankclaw dankserver; do
        for f in scripts/gpu-exporter.py scripts/gpu-exporter.service; do
            run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target "$t" --paths "$f"
            [ "$status" -eq 0 ]
            [ "$output" = "gpu-exporter" ] || { echo "$t/$f -> '$output'"; return 1; }
        done
    done
 }
@test "Prometheus scrapes gpu-exporter where node/cadvisor are reachable on each host" {
    # Cross-file contract, same posture as node-exporter: the local instance is
    # reached via the podman gateway (this Prometheus is a bridge container and
    # a host-process bind is refused at the host LAN IP); the Dankserver
    # instance is a different machine, so plain LAN routing reaches it. If either
    # address drifts from the exporter's bind, GPU panels go blank with no error.
    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
    grep -q 'job_name: "gpu-dankclaw"' "$cfg" || { echo "no gpu-dankclaw job"; return 1; }
    grep -q 'job_name: "gpu-dankserver"' "$cfg" || { echo "no gpu-dankserver job"; return 1; }
    grep -q '"host.containers.internal:9401"' "$cfg" \
        || { echo "gpu-dankclaw does not scrape the podman gateway"; return 1; }
    grep -q '"192.168.0.235:9401"' "$cfg" \
        || { echo "gpu-dankserver does not scrape the LAN address"; return 1; }
    # A vhost target would be red forever (Prometheus cannot solve Anubis).
    ! grep -qE '"[^"]*franksdank[^"]*:9401"' "$cfg"
 }
@test "the host-system dashboard graphs the gpu_* families the exporter emits" {
    local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/host-system.json"
    [ -f "$dash" ]
    python3 -c "
 import json
 d = json.load(open('$dash'))
 exprs = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
 for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
             'gpu_temperature_celsius', 'gpu_power_watts'):
    assert any(want in e for e in exprs), ('missing', want, exprs)
 print('gpu dashboard panels ok')
 "
 }
@test "--serve answers on EVERY declared bind, not just the first" {
    # The deployed unit serves loopback + the host LAN address, and Prometheus
    # reaches the host through the gateway — which dials the LAN-class bind.
    # A server that only runs serve_forever() on the first socket leaves the
    # second one listen()ing-but-never-accepting: the deploy health gate on
    # loopback passes, every scrape on the other bind dies on the SYN backlog,
    # and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
    # left both gpu-* targets down beside a green deploy.)
    mk_amd 0 42 1073741824 34359738368 "" 71000
    port=$((19400 + RANDOM % 500))
    python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
        --bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
        > "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
    local pid=$!
    trap 'kill "$pid" 2>/dev/null' EXIT
    local ok=0
    for _ in $(seq 1 20); do
        curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
        sleep 0.25
    done
    [ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
    # Both binds must ANSWER within 2 s each — the second one is the bug.
    for b in 127.0.0.1 127.0.0.2; do
        curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
            || { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
    done
    kill "$pid" 2>/dev/null
    trap - EXIT
 }
@@ -595,16 +595,22 @@ assert any('ALERTS' in e for e in exprs), exprs
 }
@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
-    # Cross-file contract: cadvisor publishes the LAN port; a cadvisor scrape
+    # Cross-file contract, measured 2026-10-21 from inside the Prometheus
-    # job pointing anywhere else reads nothing (unlike node-exporter's host-net
+    # container: a bridge container cannot dial its OWN host's LAN IP — the
-    # bind, a published port IS reachable from this bridge container at the LAN
+    # hairpin refuses published ports exactly like host-net binds, so
-    # address). And the whole dashboard depends on the id -> container relabel:
+    # cadvisor-dankclaw must scrape via the podman gateway (the original pin
-    # cadvisor names nothing itself, the quadlet unit name is only in the cgroup
+    # here demanded the LAN address and shipped a permanently-down target).
-    # path. If either half changes alone, the Containers dashboard is blank.
+    # dankserver is a different machine: plain LAN there. And the whole
    # dashboard depends on the id -> container relabel: cadvisor names nothing
    # itself, the quadlet unit name is only in the cgroup path. If either half
    # changes alone, the Containers dashboard is blank.
    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
    grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
    grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
-    grep -q '"192.168.0.143:8080"' "$cfg" || { echo "cadvisor-dankclaw does not scrape the published LAN port"; return 1; }
+    grep -q '"host.containers.internal:8080"' "$cfg" \
        || { echo "cadvisor-dankclaw does not scrape via the podman gateway"; return 1; }
    ! grep -q '"192.168.0.143:8080"' "$cfg" \
        || { echo "cadvisor-dankclaw dials its own host LAN IP — hairpin refuses it"; return 1; }
    grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
    # The relabel that turns …/app.slice/<name>.service… into a `container`.
    grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
Author	SHA1	Message	Date
dank	ccb805cf0b	Merge pull request 'fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches' (#66 ) from fix/dragoon-context-200k into main	2026-10-21 02:25:42 -07:00
Frank	c6e2887f94	fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches The lane's child dlopens libxrt_core.so.2 from /opt/xilinx/lib/<multiarch>/ as well; our mounts filled only /opt/xilinx/xrt/lib and the install path. Since the host's libxrt2 was reconfigured 2026-10-20 the child dies naming the missing prefix ('No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2') and the engine supervisor takes the WHOLE container down on a dead lane — measured 2026-10-21: frank-dragoon boot-looped 88 times with chat dead while the GPU lines looked healthy. Each XRT library now mounts THREE times from the same real file (never a symlink); the multiarch name derives from FRANK_DRAGOON_XRT_DIR's basename so the bats fixture proves the derivation. Proven end-to-end before commit: production-shaped scratch boot + the three extra mounts answers the full boot ledger, npu_api buckets u0..u9, and /health ok at the 262144 ceiling. A dead lane is not survivable on this engine — arm it only with all three prefixes (page law updated).	2026-10-21 02:21:53 -07:00
dank	9eb53ef937	Merge pull request 'feat(frank-dragoon): 200,000-token harness window via the native 262,144 ceiling' (#65 ) from fix/dragoon-context-200k into main	2026-10-21 01:40:40 -07:00
Frank	688928d489	docs(dragoon): the 262144 ceiling, the 200,000 window, and the 95 % trigger law Shape + harness contract on the endpoint page, AGENTS.md, and the OMP Harness worked example recomputed to the live pair; the 400-zone law now names the 85 %-comments vs 95 %-config disagreement as the recurring cause and points at the seam that enforces the arithmetic.	2026-10-21 01:34:28 -07:00
Frank	56503dbad1	feat(frank-dragoon): raise HALOGEN_CTX to the native 262144 ceiling The 200,704 ceiling squeezed the harness contextWindow down to 155,648 (the 400-zone fit law needs ~45k of gap under the ceiling), which is why long dragoon sessions kept hitting the 400 zone: the harness clamps max_tokens to window - est - 4096, its chars/4 est under-counts ~3 %, and config.yml compacts at 95 % while the pair was designed for 85 %. Ceiling -> checkpoint's native 262,144 (no RAM move: the KV pool stays 401,408 and is what the ledger bills; the band ctx <= pool <= slots*ctx still holds). The harness window becomes 200,000 with a 62,144 gap. The fit-law seam now reads the declared window from the vault page (the one owner) and checks the arithmetic at the live 95 % trigger, so a ceiling/window pair that drifts back together fails tests/run.sh instead of a live session.	2026-10-21 01:34:28 -07:00
dank	fbeb9701e4	Merge pull request 'fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway' (#64 ) from fix/gpu-serve-and-cadvisor-target into main	2026-10-21 00:39:54 -07:00
Frank	650b6e0281	fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and cadvisor-dankclaw DOWN beside a green deploy: - gpu-exporter.py built one ThreadingHTTPServer per bind but ran serve_forever() on servers[0] only: the LAN-class socket sat listen()ing but never accept()ed, loopback deploy gate passed, every scrape that reached the other bind timed out on the SYN backlog. Now one thread per extra bind. Regression test spins --serve on two loopback binds and requires BOTH to answer /metrics. - cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the Prometheus container: a bridge container cannot dial its own host's LAN address at all (hairpin refuses published ports too); the podman gateway answers. Target moved to host.containers.internal:8080; the seam that pinned the LAN address (asserting the wrong doctrine) is re-pinned to the gateway plus a negative pin against the own-host LAN dial. Observability page records the measured reachability law replacing the inferred one.	2026-10-21 00:34:17 -07:00
dank	b62a05837f	Merge pull request 'fix(deploy): client allowlist missing cadvisor/gpu-exporter/prowlarr' (#63 ) from fix/deploy-sh-allowlist-drift into main	2026-10-21 00:15:45 -07:00
Frank	0367244d2a	fix(deploy): client allowlist was missing cadvisor, gpu-exporter, prowlarr deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are two gates for the same door, and they drifted: the remote gate gained cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate never did, so `--service cadvisor\|gpu-exporter` died with "is not deployed on <target>" at the client — the first-ever deploy of a new service is exactly the case that needs the client gate to know it. dankserver:prowlarr (planner can emit it for dankserver) was remote-only too. Adds the four missing pairs to the client gate and a deployment_flow contract test that diffs the two extracted lists per target — the test the cadvisor PR should have carried. searxng-network stays the one sanctioned client-only exception (network unit, no remote handler), documented inline.	2026-10-21 00:15:21 -07:00
dank	656d9567d5	Merge pull request 'chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)' (#62 ) from chore/plex-1435 into main	2026-10-21 00:07:28 -07:00
Frank	0b4bc60a38	chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest) ls329 is the newest build on the linuxserver channel and Plex 1.43.5.11029 is their newest release; the live pin (ls325 / 1.43.4.10903) is two Plex releases behind. Digest-verified at pin time: the tag's manifest digest equals the registry's current 'latest' (sha256:8f655bad...e9e2334), the same law the page records. Quadlet restart runs PMS's usual one-minor schema migration on first boot; the deployer's /identity HTTPS gate tolerates the first-boot migration up to 600 s. Rollback: revert this commit (one line).	2026-10-21 00:06:53 -07:00
dank	a2011f4eb6	Merge pull request 'feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet' (#61 ) from feat/gpu-stats into main	2026-10-20 23:50:11 -07:00
Frank	b02647b992	feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet Add a host-process GPU exporter, scrape it on both servers, and graph it on the host-system dashboard. The fleet has four GPUs across three vendors of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw) and nothing exposed them until now. scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs (gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average) plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus families (utilization_percent, memory_used/total_bytes, temperature_celsius, power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series, never a fabricated 0: the R9700s expose no sysfs power, so they simply carry no gpu_power_watts line. Both sources are injectable, so the seam runs on a GPU-less dev box; live-verified on Dankserver's real sysfs. It runs as a plain user unit (not a quadlet): it must read the host's own DRM sysfs and shell to nvidia-smi, which a container cannot do without GPU passthrough. Same posture as signal-relay. The base unit binds loopback; the deploy drop-in appends the host LAN address, never a wildcard (matching node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it on both hosts and is wired into the planner (both targets), the test-changed router, and two Prometheus scrape jobs using the node/cadvisor reach rules (gateway on Dankclaw, LAN on Dankserver).	2026-10-20 23:42:26 -07:00
dank	b35269bfa5	Merge pull request 'docs(archon): four-slot ledger, resolver law from source, probe self-calibration' (#60 ) from wt/20261020-213526 into main	2026-10-20 23:30:27 -07:00
Frank	1aeb7af6df	docs(probe): price the corpus chars/token ratio at run time The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate 400 from the engine's own window check — which reads as a capacity-contract failure on a healthy endpoint. One ~4k-token request now prices the ratio, and the leg sizes to 97 % of the deployed window.	2026-10-20 23:26:53 -07:00
Frank	cd0be97b39	docs(vault): the four-slot ledger, and the resolver's funding law from the source The shape block and boot ledger now quote the live 1.8.1 boot: experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %) = 747,492 tokens, 3.7 full-length sessions, `--checkpoint-slots 40`, `queue_cap 32`, sampler 12 rows, `experts-from-host` 8.99 GiB a rank. The KV Capacity law is rewritten from the pinned source instead of inference: `core/mem/vram_budget.cpp:146-156` funds the cache as `elastic - expert_floor`, `core/mem/kv.cpp:266` sets its ceiling from `max_seqs x max_ctx` (demand, never funding), and `core/place/planner.cpp:100-104` makes the expert floor fall one-for-one with `--host-pool-mib` — which is why +4,224 MiB of pinned host pool bought the pool and cost 3.01 GiB of residency a card. The old "funds ~90 %, never 100 %" phrasing was inference; the byte-identical 45,568 row is the ceiling case, not a mystery. Slot-cost law carries the four-slot price (state + checkpoint store + rows, still under the 25-row exact-wire bound). The two-slot era is one row in Eras with its rollback commit. Date stamps in the trio comments corrected to 2026-10-20.	2026-10-20 23:26:53 -07:00
dank	7993a336bf	Merge pull request 'feat(frank-archon): four decode slots over a >=700k-token KV pool' (#59 ) from wt/20261020-213526 into main	2026-10-20 23:14:35 -07:00
Frank	1bb6399524	feat(frank-archon): four decode slots over a >=700k-token KV pool Frank's order: 4 slots, a 700k pool, round UP if the resolver cannot land it exactly. The cache target is MAX_SEQS x MAX_MODEL_LEN, so the demand is 800,000 fp8 tokens; the resolver funds ~91-93 % of demand (three boots of evidence on the endpoint page), which clears 700k, and the VRAM has to come from the expert plane -- which has to go somewhere, so HOST_POOL_MIB moves with it. expert_vram_floor (core/place/planner.cpp:100-104) is exactly plane + 2 units - host_room, and the live boot ledger pins that plane at 67.60 GiB a rank (23.73 GiB floor at 44,928 of room), so 49,152 drops the floor to 19.61 GiB and leaves the cache ~6.9 GiB: enough for the whole target, with the leftover returned to the experts. Costs, both real: pinned host RAM 87.75 -> 96.0 GiB (the pool is pinned once per rank) and ~3.2 GiB a card of expert residency, read over the x8 Gen5 link when the router picks a host-resident expert. The boot ledger's vram_kv line is the referee; if it comes back under 700,000 tokens HOST_POOL_MIB goes up 1024 and this redeploys. TTM_MIN_MIB 150,000 -> 162,000 for the new pin ledger (2 x pool + PLE 48,824 + prefix 8,192 + MNBT 2,048 + the auto checkpoint store, now 40 slots of 112 MiB = 161,848 MiB), and the bats ledger guard now counts the checkpoint store because it grows with the slot count. The two env-value assertions in that seam are bumped with the values they guard.	2026-10-20 23:10:05 -07:00
dank	31b71a16d6	Merge pull request 'feat(observability): per-container metrics via cadvisor + Containers dashboard' (#58 ) from feat/containers-observability into main	2026-10-20 23:08:23 -07:00
dank	a4edaa7d9d	Merge pull request 'feat(frank-archon): arm --prefill-interleave at 1' (#57 ) from wt/20261020-213526 into main	2026-10-20 22:55:58 -07:00
Frank	aa75b45543	feat(frank-archon): arm --prefill-interleave at 1 A prefill chunk that shares a step with decoding sequences holds the device for the chunk's whole time: measured on this shape, decode-only steps 15.5 ms against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for an arriving request goes 2.4 -> 18.0 s. BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source refuses to halve the chunk for decode rows alone -- which is why engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE is the knob that acts on the measured mechanism: N decode-only steps follow any chunk step taken beside decoders. N=1 is the conservative first arm -- a stream takes its token from the chunk step and one from the held step (~2 tokens per 182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's TTFT and on requests still queued, not on anything decoding. Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the serve script only passes it when > 0), engine_steps_held_total must leave zero, then the §9 bucket pair must move before N rises.	2026-10-20 22:52:31 -07:00
Frank	b1fc1113c6	docs(vault): replace an invented --checkpoint-policy name with the three that exist The replay law named `--checkpoint-policy floor`; config.cpp admits exactly geometric-backoff (default), tip-only, keep-all and refuses anything else at boot, so the recorded arm could never be deployed. Replaced with the real semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live ledger line -- so a turn diverging d behind the tip can resume at tip-3d: diverge at 120k, replay 21.7k); retention is not the write path, so keep-all buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4 GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room); and shortening --checkpoint-interval cannot fix the deep case -- the coarsest ladder scale is interval x 2^k under the depth, so the deepest gap stays about half the prompt at every interval. §9 keeps only the three-name locator.	2026-10-20 22:34:13 -07:00
Frank	9f7457e39e	docs(vault): answer "what are we doing wrong" on decode-vs-prefill frank-archon Laws: add the measured interference census (decode-only step 15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test that kills the residency theory, the busy_cap arithmetic, and the correction that --prefill-interleave is the decoder dial and auto is not. Facts that change how an operator reads this box: /diagnostics exists and reports link-below-max at 100% busy; latency metrics live under vllm:, cadence under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5 max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the 4600 corruption fix is recorded -- verify before blaming garbage tokens. §9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters, dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the chunk), instruments that cannot be reached from Prometheus, and the knob list priced from the pinned 1.8.1 source instead of the stale guide.	2026-10-20 22:08:00 -07:00
Frank	687dbfbabe	docs(scripts): correct frank-archon's step-sharing and slot-count comments The env comment claimed BUSY_PREFILL_CHUNK=auto protects decode rows; the 1.8.1 source (core/sched/scheduler.cpp busy_cap) reserves 1+spec rows per decoding sequence and 1/8 of the budget per waiting request, and refuses to shrink the chunk for decoders alone -- auto is a prefill-speed dial. The interleave comment carried a 512 that is not this build's auto value (0 until the held-step counter asks; 64 in help mode). The serve header still described three decode slots; MAX_SEQS has been 2 since the 2026-10-19 reslice. No behavior change.	2026-10-20 22:07:55 -07:00