Compare commits

...
Author SHA1 Message Date
Frank 688928d489 docs(dragoon): the 262144 ceiling, the 200,000 window, and the 95 % trigger law
Shape + harness contract on the endpoint page, AGENTS.md, and the OMP
Harness worked example recomputed to the live pair; the 400-zone law now
names the 85 %-comments vs 95 %-config disagreement as the recurring
cause and points at the seam that enforces the arithmetic.
2026-10-21 01:34:28 -07:00
Frank 56503dbad1 feat(frank-dragoon): raise HALOGEN_CTX to the native 262144 ceiling
The 200,704 ceiling squeezed the harness contextWindow down to 155,648
(the 400-zone fit law needs ~45k of gap under the ceiling), which is why
long dragoon sessions kept hitting the 400 zone: the harness clamps
max_tokens to window - est - 4096, its chars/4 est under-counts ~3 %,
and config.yml compacts at 95 % while the pair was designed for 85 %.

Ceiling -> checkpoint's native 262,144 (no RAM move: the KV pool stays
401,408 and is what the ledger bills; the band ctx <= pool <= slots*ctx
still holds). The harness window becomes 200,000 with a 62,144 gap.

The fit-law seam now reads the declared window from the vault page (the
one owner) and checks the arithmetic at the live 95 % trigger, so a
ceiling/window pair that drifts back together fails tests/run.sh
instead of a live session.
2026-10-21 01:34:28 -07:00
dank fbeb9701e4 Merge pull request 'fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway' (#64) from fix/gpu-serve-and-cadvisor-target into main 2026-10-21 00:39:54 -07:00
Frank 650b6e0281 fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:

- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
  serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
  but never accept()ed, loopback deploy gate passed, every scrape that
  reached the other bind timed out on the SYN backlog. Now one thread per
  extra bind. Regression test spins --serve on two loopback binds and
  requires BOTH to answer /metrics.

- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
  Prometheus container: a bridge container cannot dial its own host's LAN
  address at all (hairpin refuses published ports too); the podman gateway
  answers. Target moved to host.containers.internal:8080; the seam that
  pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
  gateway plus a negative pin against the own-host LAN dial. Observability
  page records the measured reachability law replacing the inferred one.
2026-10-21 00:34:17 -07:00
dank b62a05837f Merge pull request 'fix(deploy): client allowlist missing cadvisor/gpu-exporter/prowlarr' (#63) from fix/deploy-sh-allowlist-drift into main 2026-10-21 00:15:45 -07:00
Frank 0367244d2a fix(deploy): client allowlist was missing cadvisor, gpu-exporter, prowlarr
deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are
two gates for the same door, and they drifted: the remote gate gained
cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate
never did, so `--service cadvisor|gpu-exporter` died with "is not deployed
on <target>" at the client — the first-ever deploy of a new service is
exactly the case that needs the client gate to know it. dankserver:prowlarr
(planner can emit it for dankserver) was remote-only too.

Adds the four missing pairs to the client gate and a deployment_flow
contract test that diffs the two extracted lists per target — the test the
cadvisor PR should have carried. searxng-network stays the one sanctioned
client-only exception (network unit, no remote handler), documented inline.
2026-10-21 00:15:21 -07:00
dank 656d9567d5 Merge pull request 'chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)' (#62) from chore/plex-1435 into main 2026-10-21 00:07:28 -07:00
14 changed files with 204 additions and 96 deletions
+11 -10
View File
@@ -419,16 +419,17 @@ disagrees with it is the bug.
digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
`/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
`HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
drafter always armed (the adaptive default disarms it and halves decode);
server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
under-counts, so a harness window equal to the ceiling makes every long
request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
**per-request ceiling** (a longer prompt is a 400, never a truncation);
`HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
pool positions buy concurrency, the per-request window never moves;
`HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
default disarms it and halves decode); server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
estimate under-counts (trigger is 95 %), so a harness window at the ceiling
makes every long request a guaranteed 400.
Reasoning arrives in `reasoning_content`; the
template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
`/completion`, `/ping`, `/props`, `/tokenize` are 404.
+4 -3
View File
@@ -1401,9 +1401,10 @@ deploy_anubis() {
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
# qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
# + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
# run a 200,000 window with its ~45k fit gap; the pool still carries two
# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
# ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
# relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
# file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
+5 -3
View File
@@ -133,13 +133,15 @@ fi
valid_service() {
local service="$1"
case "$TARGET:$service" in
# dankclaw:searxng-network is client-only: a network quadlet with no
# remote dispatch handler, so the remote gate does not (must not) list it.
dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
dankclaw:gitea-server|dankclaw:homepage|\
dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
dankclaw:alertmanager|\
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
dankclaw:open-notebook|\
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
dankserver:frank-archon|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
dankserver:prowlarr|dankserver:frank-archon|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
+15 -8
View File
@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
# stays at the image default, which is what was qualified) ------------------
# Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
HALOGEN_MODEL_ID=frank-dragoon
# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
# the page) — a window equal to the ceiling makes every long request a 400.
HALOGEN_CTX=200704
# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
# at once. Pool positions buy concurrency; the per-request window never moves
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
# RAM the pool takes is RAM the n-gram table's page cache loses.
# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
# longer prompt is a 400 naming the limit, never a truncation; stretching
# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
# page): with the window at 200,000 the gap is 62,144, which absorbs the
# chars/4 under-count. A window at or near the ceiling makes every long
# request a guaranteed 400.
HALOGEN_CTX=262144
# ONE pool behind both slots. The harness window is 200,000, so 401,408
# still carries TWO maximum harness-length conversations at once
# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
# ceiling-length requests would queue instead — buying those means raising
# the pool, and RAM the pool takes is RAM the n-gram table's page cache
# loses. Pool positions buy concurrency; the per-request window never moves
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
HALOGEN_KV_POOL_POSITIONS=401408
# Two concurrent conversations: the MTP drafter is not slot-locked on this
# engine, so the second slot is capacity, not a speculation tax.
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
After=network-online.target
Wants=network-online.target
+6
View File
@@ -32,6 +32,7 @@ import glob
import os
import subprocess
import sys
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
FAMILIES = [
@@ -198,6 +199,11 @@ def main(argv):
return 2
servers.append(ThreadingHTTPServer((b, args.port), Handler))
sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
# EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
# healthy in `ss` yet times out every scrape once its SYN backlog fills —
# and the loopback deploy gate cannot see it. One thread per extra bind.
for s in servers[1:]:
threading.Thread(target=s.serve_forever, daemon=True).start()
servers[0].serve_forever()
return 0
+18 -14
View File
@@ -52,28 +52,31 @@ Where the contract lives:
The arithmetic (the coupled contract every endpoint's maxTokens comment
encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
follows is a **worked example** recomputed from the live files on 2026-10-18 —
a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
follows is a **worked example** recomputed from the live files on 2026-10-21 —
a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
bundle clamps the percentage to 1–99 and converts it to a token threshold.
2. **Output budget** = capped at the target model's `maxTokens` — every role
here compacts to frank-zealot, so the budget is its 16,384.
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
200,704. The registry's own comments design around an 85 % trigger (archon:
170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
does hold. While `config.yml` and those comments disagree, treat 85 % as the
design point and re-run this arithmetic after ANY window change — a 400 from
the compaction target mid-conversation is the symptom.
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
dragoon session the fitting window is the ENGINE ceiling, not a slot: the
190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
error plus the 45,056 request budget lands inside — that headroom is the
reason the ceiling moved to native on 2026-10-21. For a session compacting
INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
200,704. Treat 85 % as the design point until config.yml and the registry
comments agree, and re-run this arithmetic after ANY window change — a 400
from the compaction target mid-conversation is the symptom.
(Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
32,768 ≤ 200,704 zealot.)
Procedure (recompute from the LIVE files every time — windows rotate with
every swap; the block above is the 2026-10-18 state):
every swap; the block above is the 2026-10-21 state):
1. Read the `config.yml` compaction block + `models.yml` entries for source
and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
dankclaw — the comment/config disagreed. The comments now name the real
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
truth; fix comments in the same change.** A fresh instance of the same trap is
open 2026-10-18: every `maxTokens` comment still says "compaction fires at
85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
disagreement is not academic: it is why long dragoon sessions kept drawing
400s at the old 200,704 ceiling — the trigger fires late AND the est
under-counts, and a window parked near the ceiling has no room for either.
`maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
input. Boot-shape OOM laddering is a different problem
([[Llama Cpp Serving Sizing]]).
@@ -67,10 +67,18 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
the wildcard — see [[Node-Exporter]].
The relay's bind is the one address a bridge container can reach on the host:
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
and the podman bridge subnet `10.89.0.0/16` before the DROP
(`scripts/deploy-remote.sh:255-262`).
**Same-host reachability from a bridge container is measured, not inferred**
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
(the podman gateway) reaches host services whether they bind host-net or are
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
every same-host Prometheus target uses the gateway and only cross-host targets
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
iptables INPUT admitting loopback, the host itself and the podman bridge subnet
`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
point a scrape target at a host LAN IP from its own host on the strength of
one working example; dial the gateway and probe.
## Deployment
+26 -18
View File
@@ -10,14 +10,15 @@ tags:
- gfx1151
- halogen
- endpoint
timestamp: 2026-10-19
timestamp: 2026-10-21
---
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
## Current shape
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
unauthenticated token port 8730 stays on container loopback. Engine stdout →
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
`HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
`HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
(`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
- Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
(Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
Caddy/observability).
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
(= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
`reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
rides [[frank-zealot]].
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
— sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
`tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
`high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
- Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
@@ -124,14 +128,18 @@ created it — earlier topologies in .
prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
re-measure the warm ratio after any pool change.
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
`tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
- **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
+7 -4
View File
@@ -140,12 +140,15 @@ scrape_configs:
# system.slice daemons and machine.slice scopes carry no such name and are
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
# the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
# is why node-dankclaw scrapes via host.containers.internal), a published port
# IS reachable from this bridge container at the host LAN address.
# the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
# DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
# connection whether the port is a host-net bind or a published port (the
# original note here claimed published ports were exempt; measured 2026-10-21
# from inside this container: LAN refused, gateway answers). Cross-host
# targets (dankserver) have no hairpin problem and stay plain LAN.
- job_name: "cadvisor-dankclaw"
static_configs:
- targets: ["192.168.0.143:8080"]
- targets: ["host.containers.internal:8080"]
labels:
host: dankclaw
metric_relabel_configs:
+25
View File
@@ -125,3 +125,28 @@ load helpers
[ ! -e "$sandbox/frank-default-sightsound" ]
[ -e "$BATS_TEST_TMPDIR/.AGENTTRASH/$(date +%Y%m%d)-retired-deploy-state/frank-default-sightsound" ]
}
@test "the client and remote allowlists agree per target" {
# Two gates guard `deploy.sh --service X`: the client's valid_service and
# the remote runner's service_is_allowed. When only one side gained a
# service the other answered "is not deployed on <target>" and the only
# repair path was a fleet-wide drain — cadvisor and gpu-exporter both fell
# through this crack within a day of each other. The lists MUST agree;
# searxng-network is the one sanctioned exception (client-only network
# unit, no remote dispatch handler).
run env OUT="$BATS_TEST_TMPDIR" bash -c '
grep -oE "dank(claw|server):[a-z0-9-]+" \
<(sed -n "/^valid_service()/,/^}/p" "$1/scripts/deploy.sh") \
| grep -v "dankclaw:searxng-network" | sort -u > "$OUT/cli"
grep -oE "dank(claw|server):[a-z0-9-]+" \
<(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
| sort -u > "$OUT/rem"
diff "$OUT/cli" "$OUT/rem"
' _ "$REPO_ROOT"
[ "$status" -eq 0 ] || { echo "deploy.sh vs deploy-remote.sh allowlist drift:"; echo "$output"; return 1; }
# Drift aside, the gate must actually know the host-process exporters —
# their first-ever deploy has to pass the client gate.
for pair in dankclaw:cadvisor dankclaw:gpu-exporter dankserver:cadvisor dankserver:gpu-exporter; do
grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
done
}
+31 -24
View File
@@ -3,7 +3,7 @@
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE
# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
# shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
#
@@ -85,7 +85,7 @@ good_env() {
"FRANK_DRAGOON_VISION_BYTES=0"
"FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
"HALOGEN_MODEL_ID=frank-dragoon"
"HALOGEN_CTX=200704"
"HALOGEN_CTX=262144"
"HALOGEN_KV_POOL_POSITIONS=401408"
"HALOGEN_KV_SLOTS=2"
"HALOGEN_SPEC_ADAPT=0"
@@ -159,9 +159,11 @@ good_env() {
reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
# The pool is ONE budget behind the slots (boot ledger form: "2 slots over
# ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
# a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
# (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
# resident at once — frank-zealot parity on total in-flight context. What
# a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
# native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
# MAXIMUM harness-length (200,000-token) conversations are resident at once;
# two literal ceiling-length requests would queue — widening that is a pool
# raise, not a window move.
# still binds: the pool must back ONE full-length request (below that the
# advertised per-request ceiling would be unreachable, hence pool >= ctx),
# it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
[[ "$output" == *"--name frank-dragoon-halogen"* ]]
[[ "$output" == *"--publish 7770:8731"* ]]
[[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
[[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
[[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
[[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
[[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
[[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"
@test "the shipped env keeps the harness fit law inside the ceiling" {
# HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
# naming the limit, never a silent truncation). The law is INVARIANT to
# the value: the harness window must sit ~45k UNDER it, because the
# harness clamps max_tokens = min(45056, window − est − 4096), compacts
# at 85 % of its declared window, and its chars/4 est UNDER-counts the
# rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
# est 152,938 on 2026-10-07 — with window == ceiling, every request over
# ~65 % of the window was a guaranteed 400 before compaction could fire).
# The shipped pair is ctx 200704 / window 155648 (lowered from
# 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
# monotone maxima of est + budget (est ≤ window − 4096):
# full budget at the compaction threshold: 1.03*132300 + 45056
# clamped budget at the clamp's zero point: 1.03*151552 + 0
# and the compaction request that rides on frank-zealot (its
# 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
ctx=200704
hw=155648; mt=45056
thr=$(( hw * 85 / 100 ))
# naming the limit, never a silent truncation). The law: the harness
# window must sit ~45k UNDER the ceiling, because the harness clamps
# max_tokens = min(45056, window − est − 4096), compacts at the trigger
# percent of its declared window, and its chars/4 est UNDER-counts the
# rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
# at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
# request is a guaranteed 400 before compaction can fire). The 2026-10-21
# pair: ceiling 262144 (native) under a harness window of 200,000.
# The declared window is read from the endpoint page — the one owner —
# so env, models.yml and page that disagree FAIL HERE, not as a 400
# mid-session (the recurring failure this seam exists for).
grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
grep -qE 'contextWindow: 200000' "$page"
hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
mt=45056
# Invariant checked at the live 95 % trigger (config.yml
# thresholdPercent) — strictly stronger than the 85 % design point.
# Both monotone maxima of est + budget (est ≤ window − 4096):
thr=$(( hw * 95 / 100 ))
[ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
zero=$(( hw - 4096 ))
[ $(( zero * 103 / 100 )) -le "$ctx" ]
# The compaction request rides on frank-zealot's 200,704-token slot:
# input ≈ threshold − keepRecent(20k).
[ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
}
+30
View File
@@ -195,3 +195,33 @@ for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
print('gpu dashboard panels ok')
"
}
@test "--serve answers on EVERY declared bind, not just the first" {
# The deployed unit serves loopback + the host LAN address, and Prometheus
# reaches the host through the gateway — which dials the LAN-class bind.
# A server that only runs serve_forever() on the first socket leaves the
# second one listen()ing-but-never-accepting: the deploy health gate on
# loopback passes, every scrape on the other bind dies on the SYN backlog,
# and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
# left both gpu-* targets down beside a green deploy.)
mk_amd 0 42 1073741824 34359738368 "" 71000
port=$((19400 + RANDOM % 500))
python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
--bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
> "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
local pid=$!
trap 'kill "$pid" 2>/dev/null' EXIT
local ok=0
for _ in $(seq 1 20); do
curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
sleep 0.25
done
[ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
# Both binds must ANSWER within 2 s each — the second one is the bug.
for b in 127.0.0.1 127.0.0.2; do
curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
|| { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
done
kill "$pid" 2>/dev/null
trap - EXIT
}
+13 -7
View File
@@ -595,16 +595,22 @@ assert any('ALERTS' in e for e in exprs), exprs
}
@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
# Cross-file contract: cadvisor publishes the LAN port; a cadvisor scrape
# job pointing anywhere else reads nothing (unlike node-exporter's host-net
# bind, a published port IS reachable from this bridge container at the LAN
# address). And the whole dashboard depends on the id -> container relabel:
# cadvisor names nothing itself, the quadlet unit name is only in the cgroup
# path. If either half changes alone, the Containers dashboard is blank.
# Cross-file contract, measured 2026-10-21 from inside the Prometheus
# container: a bridge container cannot dial its OWN host's LAN IP — the
# hairpin refuses published ports exactly like host-net binds, so
# cadvisor-dankclaw must scrape via the podman gateway (the original pin
# here demanded the LAN address and shipped a permanently-down target).
# dankserver is a different machine: plain LAN there. And the whole
# dashboard depends on the id -> container relabel: cadvisor names nothing
# itself, the quadlet unit name is only in the cgroup path. If either half
# changes alone, the Containers dashboard is blank.
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
grep -q '"192.168.0.143:8080"' "$cfg" || { echo "cadvisor-dankclaw does not scrape the published LAN port"; return 1; }
grep -q '"host.containers.internal:8080"' "$cfg" \
|| { echo "cadvisor-dankclaw does not scrape via the podman gateway"; return 1; }
! grep -q '"192.168.0.143:8080"' "$cfg" \
|| { echo "cadvisor-dankclaw dials its own host LAN IP — hairpin refuses it"; return 1; }
grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
# The relabel that turns …/app.slice/<name>.service… into a `container`.
grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }