Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ccb805cf0b | ||
|
|
c6e2887f94 | ||
|
|
9eb53ef937 | ||
|
|
688928d489 | ||
|
|
56503dbad1 | ||
|
|
fbeb9701e4 | ||
|
|
650b6e0281 | ||
|
|
b62a05837f |
@@ -419,16 +419,17 @@ disagrees with it is the bug.
|
|||||||
digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
|
digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
|
||||||
Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
|
Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
|
||||||
`/home/dank/logs/frank_dragoon.log`, NOT the journal.
|
`/home/dank/logs/frank_dragoon.log`, NOT the journal.
|
||||||
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
|
- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
|
||||||
is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
|
**per-request ceiling** (a longer prompt is a 400, never a truncation);
|
||||||
`HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
|
`HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
|
||||||
the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
|
pool positions buy concurrency, the per-request window never moves;
|
||||||
drafter always armed (the adaptive default disarms it and halves decode);
|
`HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
|
||||||
server sampler 1.0/0.95/20.
|
default disarms it and halves decode); server sampler 1.0/0.95/20.
|
||||||
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
|
- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
|
||||||
The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
|
ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
|
||||||
under-counts, so a harness window equal to the ceiling makes every long
|
estimate under-counts (trigger is 95 %), so a harness window at the ceiling
|
||||||
request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
|
makes every long request a guaranteed 400.
|
||||||
|
Reasoning arrives in `reasoning_content`; the
|
||||||
template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
|
template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
|
||||||
OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
|
OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
|
||||||
`/completion`, `/ping`, `/props`, `/tokenize` are 404.
|
`/completion`, `/ping`, `/props`, `/tokenize` are 404.
|
||||||
|
|||||||
@@ -1401,9 +1401,10 @@ deploy_anubis() {
|
|||||||
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
|
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
|
||||||
# qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
|
# qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
|
||||||
# + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
|
# + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
|
||||||
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
|
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
|
||||||
# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
|
# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
|
||||||
# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
|
# run a 200,000 window with its ~45k fit gap; the pool still carries two
|
||||||
|
# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
|
||||||
# ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
|
# ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
|
||||||
# relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
|
# relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
|
||||||
# file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
|
# file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
|
||||||
|
|||||||
+22
-12
@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
|
|||||||
# stays at the image default, which is what was qualified) ------------------
|
# stays at the image default, which is what was qualified) ------------------
|
||||||
# Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
|
# Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
|
||||||
HALOGEN_MODEL_ID=frank-dragoon
|
HALOGEN_MODEL_ID=frank-dragoon
|
||||||
# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
|
# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
|
||||||
# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
|
# longer prompt is a 400 naming the limit, never a truncation; stretching
|
||||||
# the page) — a window equal to the ceiling makes every long request a 400.
|
# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
|
||||||
HALOGEN_CTX=200704
|
# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
|
||||||
# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
|
# page): with the window at 200,000 the gap is 62,144, which absorbs the
|
||||||
# at once. Pool positions buy concurrency; the per-request window never moves
|
# chars/4 under-count. A window at or near the ceiling makes every long
|
||||||
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
|
# request a guaranteed 400.
|
||||||
# RAM the pool takes is RAM the n-gram table's page cache loses.
|
HALOGEN_CTX=262144
|
||||||
|
# ONE pool behind both slots. The harness window is 200,000, so 401,408
|
||||||
|
# still carries TWO maximum harness-length conversations at once
|
||||||
|
# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
|
||||||
|
# ceiling-length requests would queue instead — buying those means raising
|
||||||
|
# the pool, and RAM the pool takes is RAM the n-gram table's page cache
|
||||||
|
# loses. Pool positions buy concurrency; the per-request window never moves
|
||||||
|
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
|
||||||
HALOGEN_KV_POOL_POSITIONS=401408
|
HALOGEN_KV_POOL_POSITIONS=401408
|
||||||
# Two concurrent conversations: the MTP drafter is not slot-locked on this
|
# Two concurrent conversations: the MTP drafter is not slot-locked on this
|
||||||
# engine, so the second slot is capacity, not a speculation tax.
|
# engine, so the second slot is capacity, not a speculation tax.
|
||||||
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
|
|||||||
FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
|
FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
|
||||||
# Node paths and the XRT directory are variables so the bats seam can drive
|
# Node paths and the XRT directory are variables so the bats seam can drive
|
||||||
# these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
|
# these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
|
||||||
# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
|
# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
|
||||||
# once at its own path, because those libraries find each other at their install
|
# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
|
||||||
# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
|
# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
|
||||||
# system dir, which dangle inside the container.
|
# searches that third prefix; without it the lane dies naming it and the
|
||||||
|
# supervisor takes the container down — 2026-10-21 boot loop). The basename of
|
||||||
|
# this directory IS the multiarch name. The vendor forbids mounting an
|
||||||
|
# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
|
||||||
FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
|
FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
|
||||||
FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
|
FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
|
||||||
# sysfs root the fabric-clock preflight reads (same knob the vendor's
|
# sysfs root the fabric-clock preflight reads (same knob the vendor's
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
|
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
Wants=network-online.target
|
Wants=network-online.target
|
||||||
|
|
||||||
|
|||||||
@@ -32,6 +32,7 @@ import glob
|
|||||||
import os
|
import os
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
import threading
|
||||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
|
||||||
FAMILIES = [
|
FAMILIES = [
|
||||||
@@ -198,6 +199,11 @@ def main(argv):
|
|||||||
return 2
|
return 2
|
||||||
servers.append(ThreadingHTTPServer((b, args.port), Handler))
|
servers.append(ThreadingHTTPServer((b, args.port), Handler))
|
||||||
sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
|
sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
|
||||||
|
# EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
|
||||||
|
# healthy in `ss` yet times out every scrape once its SYN backlog fills —
|
||||||
|
# and the loopback deploy gate cannot see it. One thread per extra bind.
|
||||||
|
for s in servers[1:]:
|
||||||
|
threading.Thread(target=s.serve_forever, daemon=True).start()
|
||||||
servers[0].serve_forever()
|
servers[0].serve_forever()
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|||||||
@@ -136,10 +136,17 @@ fi
|
|||||||
# The engine re-checks size+sha256 against that catalog at boot; it cannot
|
# The engine re-checks size+sha256 against that catalog at boot; it cannot
|
||||||
# fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
|
# fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
|
||||||
# stays unset — no outbound connection at boot),
|
# stays unset — no outbound connection at boot),
|
||||||
# 2. the distro XRT's three libraries, each mounted twice (where the image
|
# 2. the distro XRT's three libraries, each mounted THREE times: where the
|
||||||
# looks for XRT, and at its own path — the libraries find each other there;
|
# image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
|
||||||
# an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
|
# libraries find each other there), and at /opt/xilinx/lib/<multiarch>
|
||||||
# container, which the vendor calls out explicitly),
|
# — the NPU child searches that third prefix too, and since the host's
|
||||||
|
# libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
|
||||||
|
# names: omitting it crashes the lane with `No such library
|
||||||
|
# '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
|
||||||
|
# supervisor takes the dead lane down on the whole container (measured
|
||||||
|
# 2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
|
||||||
|
# of symlinks into the system dir dangles inside the container, which the
|
||||||
|
# vendor calls out explicitly),
|
||||||
# 3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
|
# 3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
|
||||||
# hang the machine while FCLK changes speed, and the engine refuses to
|
# hang the machine while FCLK changes speed, and the engine refuses to
|
||||||
# start when it moves — it cannot read host sysfs from inside the
|
# start when it moves — it cannot read host sysfs from inside the
|
||||||
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
REAL="$(readlink -f "$XRT_DIR/$L")"
|
REAL="$(readlink -f "$XRT_DIR/$L")"
|
||||||
NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
|
MULTIARCH="${XRT_DIR##*/}"
|
||||||
|
NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
|
||||||
|
-v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
|
||||||
done
|
done
|
||||||
HELD=0
|
HELD=0
|
||||||
for D in "$SYSFS"/class/drm/card*/device; do
|
for D in "$SYSFS"/class/drm/card*/device; do
|
||||||
|
|||||||
@@ -52,28 +52,31 @@ Where the contract lives:
|
|||||||
The arithmetic (the coupled contract every endpoint's maxTokens comment
|
The arithmetic (the coupled contract every endpoint's maxTokens comment
|
||||||
encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
|
encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
|
||||||
endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
|
endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
|
||||||
follows is a **worked example** recomputed from the live files on 2026-10-18 —
|
follows is a **worked example** recomputed from the live files on 2026-10-21 —
|
||||||
a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
|
a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
|
||||||
maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
|
maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
|
||||||
|
|
||||||
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
|
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
|
||||||
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
|
95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
|
||||||
bundle clamps the percentage to 1–99 and converts it to a token threshold.
|
bundle clamps the percentage to 1–99 and converts it to a token threshold.
|
||||||
2. **Output budget** = capped at the target model's `maxTokens` — every role
|
2. **Output budget** = capped at the target model's `maxTokens` — every role
|
||||||
here compacts to frank-zealot, so the budget is its 16,384.
|
here compacts to frank-zealot, so the budget is its 16,384.
|
||||||
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
|
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
|
||||||
window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
|
dragoon session the fitting window is the ENGINE ceiling, not a slot: the
|
||||||
200,704. The registry's own comments design around an 85 % trigger (archon:
|
190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
|
||||||
170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
|
error plus the 45,056 request budget lands inside — that headroom is the
|
||||||
does hold. While `config.yml` and those comments disagree, treat 85 % as the
|
reason the ceiling moved to native on 2026-10-21. For a session compacting
|
||||||
design point and re-run this arithmetic after ANY window change — a 400 from
|
INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
|
||||||
the compaction target mid-conversation is the symptom.
|
does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
|
||||||
|
200,704. Treat 85 % as the design point until config.yml and the registry
|
||||||
|
comments agree, and re-run this arithmetic after ANY window change — a 400
|
||||||
|
from the compaction target mid-conversation is the symptom.
|
||||||
(Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
|
(Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
|
||||||
177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
|
177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
|
||||||
32,768 ≤ 200,704 zealot.)
|
32,768 ≤ 200,704 zealot.)
|
||||||
|
|
||||||
Procedure (recompute from the LIVE files every time — windows rotate with
|
Procedure (recompute from the LIVE files every time — windows rotate with
|
||||||
every swap; the block above is the 2026-10-18 state):
|
every swap; the block above is the 2026-10-21 state):
|
||||||
|
|
||||||
1. Read the `config.yml` compaction block + `models.yml` entries for source
|
1. Read the `config.yml` compaction block + `models.yml` entries for source
|
||||||
and target. Every endpoint's `contextWindow` comment carries its own
|
and target. Every endpoint's `contextWindow` comment carries its own
|
||||||
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
|
|||||||
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
|
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
|
||||||
dankclaw — the comment/config disagreed. The comments now name the real
|
dankclaw — the comment/config disagreed. The comments now name the real
|
||||||
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
|
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
|
||||||
truth; fix comments in the same change.** A fresh instance of the same trap is
|
truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
|
||||||
open 2026-10-18: every `maxTokens` comment still says "compaction fires at
|
disagreement is not academic: it is why long dragoon sessions kept drawing
|
||||||
85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
|
400s at the old 200,704 ceiling — the trigger fires late AND the est
|
||||||
|
under-counts, and a window parked near the ceiling has no room for either.
|
||||||
`maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
|
`maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
|
||||||
input. Boot-shape OOM laddering is a different problem
|
input. Boot-shape OOM laddering is a different problem
|
||||||
([[Llama Cpp Serving Sizing]]).
|
([[Llama Cpp Serving Sizing]]).
|
||||||
|
|||||||
@@ -67,10 +67,18 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
|
|||||||
in the host network namespace, and it binds exactly two addresses rather than
|
in the host network namespace, and it binds exactly two addresses rather than
|
||||||
the wildcard — see [[Node-Exporter]].
|
the wildcard — see [[Node-Exporter]].
|
||||||
|
|
||||||
The relay's bind is the one address a bridge container can reach on the host:
|
**Same-host reachability from a bridge container is measured, not inferred**
|
||||||
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
|
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
|
||||||
and the podman bridge subnet `10.89.0.0/16` before the DROP
|
(the podman gateway) reaches host services whether they bind host-net or are
|
||||||
(`scripts/deploy-remote.sh:255-262`).
|
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
|
||||||
|
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
|
||||||
|
every same-host Prometheus target uses the gateway and only cross-host targets
|
||||||
|
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
|
||||||
|
relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
|
||||||
|
iptables INPUT admitting loopback, the host itself and the podman bridge subnet
|
||||||
|
`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
|
||||||
|
point a scrape target at a host LAN IP from its own host on the strength of
|
||||||
|
one working example; dial the gateway and probe.
|
||||||
|
|
||||||
## Deployment
|
## Deployment
|
||||||
|
|
||||||
|
|||||||
@@ -10,14 +10,15 @@ tags:
|
|||||||
- gfx1151
|
- gfx1151
|
||||||
- halogen
|
- halogen
|
||||||
- endpoint
|
- endpoint
|
||||||
timestamp: 2026-10-19
|
timestamp: 2026-10-21
|
||||||
---
|
---
|
||||||
|
|
||||||
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
|
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
|
||||||
|
|
||||||
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
|
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
|
||||||
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
|
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
|
||||||
(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
|
(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
|
||||||
|
This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
|
||||||
|
|
||||||
## Current shape
|
## Current shape
|
||||||
|
|
||||||
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
|
|||||||
unauthenticated token port 8730 stays on container loopback. Engine stdout →
|
unauthenticated token port 8730 stays on container loopback. Engine stdout →
|
||||||
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
|
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
|
||||||
|
|
||||||
- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
|
- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
|
||||||
`HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
|
2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
|
||||||
enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
|
`HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
|
||||||
exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
|
(`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
|
||||||
|
exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
|
||||||
|
conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
|
||||||
- Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
|
- Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
|
||||||
omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
|
omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
|
||||||
pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
|
pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
|
||||||
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
|
|||||||
(Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
|
(Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
|
||||||
~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
|
~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
|
||||||
Caddy/observability).
|
Caddy/observability).
|
||||||
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
|
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
|
||||||
(= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
|
— sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
|
||||||
`reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
|
`tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
|
||||||
rides [[frank-zealot]].
|
together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
|
||||||
|
`high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
|
||||||
- Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
|
- Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
|
||||||
pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
|
pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
|
||||||
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
|
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
|
||||||
@@ -124,14 +128,18 @@ created it — earlier topologies in .
|
|||||||
prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
|
prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
|
||||||
85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
|
85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
|
||||||
re-measure the warm ratio after any pool change.
|
re-measure the warm ratio after any pool change.
|
||||||
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
|
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
|
||||||
155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
|
clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
|
||||||
its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
|
traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
|
||||||
3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
|
thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
|
||||||
over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
|
window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
|
||||||
target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
|
theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
|
||||||
is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
|
At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
|
||||||
under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
|
200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
|
||||||
|
both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
|
||||||
|
≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
|
||||||
|
`tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
|
||||||
|
from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
|
||||||
- **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
|
- **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
|
||||||
503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
|
503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
|
||||||
together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
|
together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
|
||||||
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
|
|||||||
(`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
|
(`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
|
||||||
compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
|
compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
|
||||||
return as a crash-loop.
|
return as a crash-loop.
|
||||||
2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
|
2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
|
||||||
container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
|
distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
|
||||||
`/usr/lib/x86_64-linux-gnu`, where all three exist.
|
`libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
|
||||||
|
symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
|
||||||
|
searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
|
||||||
|
2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
|
||||||
|
engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
|
||||||
|
A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
|
||||||
3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
|
3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
|
||||||
while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
|
while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
|
||||||
engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**
|
engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**
|
||||||
|
|||||||
@@ -140,12 +140,15 @@ scrape_configs:
|
|||||||
# system.slice daemons and machine.slice scopes carry no such name and are
|
# system.slice daemons and machine.slice scopes carry no such name and are
|
||||||
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
|
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
|
||||||
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
|
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
|
||||||
# the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
|
# the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
|
||||||
# is why node-dankclaw scrapes via host.containers.internal), a published port
|
# DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
|
||||||
# IS reachable from this bridge container at the host LAN address.
|
# connection whether the port is a host-net bind or a published port (the
|
||||||
|
# original note here claimed published ports were exempt; measured 2026-10-21
|
||||||
|
# from inside this container: LAN refused, gateway answers). Cross-host
|
||||||
|
# targets (dankserver) have no hairpin problem and stay plain LAN.
|
||||||
- job_name: "cadvisor-dankclaw"
|
- job_name: "cadvisor-dankclaw"
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: ["192.168.0.143:8080"]
|
- targets: ["host.containers.internal:8080"]
|
||||||
labels:
|
labels:
|
||||||
host: dankclaw
|
host: dankclaw
|
||||||
metric_relabel_configs:
|
metric_relabel_configs:
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
|
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
|
||||||
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
|
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
|
||||||
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE
|
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE
|
||||||
# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
|
# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
|
||||||
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
|
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
|
||||||
# shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
|
# shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
|
||||||
#
|
#
|
||||||
@@ -85,7 +85,7 @@ good_env() {
|
|||||||
"FRANK_DRAGOON_VISION_BYTES=0"
|
"FRANK_DRAGOON_VISION_BYTES=0"
|
||||||
"FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
|
"FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
|
||||||
"HALOGEN_MODEL_ID=frank-dragoon"
|
"HALOGEN_MODEL_ID=frank-dragoon"
|
||||||
"HALOGEN_CTX=200704"
|
"HALOGEN_CTX=262144"
|
||||||
"HALOGEN_KV_POOL_POSITIONS=401408"
|
"HALOGEN_KV_POOL_POSITIONS=401408"
|
||||||
"HALOGEN_KV_SLOTS=2"
|
"HALOGEN_KV_SLOTS=2"
|
||||||
"HALOGEN_SPEC_ADAPT=0"
|
"HALOGEN_SPEC_ADAPT=0"
|
||||||
@@ -159,9 +159,11 @@ good_env() {
|
|||||||
reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
|
reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||||
# The pool is ONE budget behind the slots (boot ledger form: "2 slots over
|
# The pool is ONE budget behind the slots (boot ledger form: "2 slots over
|
||||||
# ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
|
# ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
|
||||||
# a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
|
# a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
|
||||||
# (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
|
# native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
|
||||||
# resident at once — frank-zealot parity on total in-flight context. What
|
# MAXIMUM harness-length (200,000-token) conversations are resident at once;
|
||||||
|
# two literal ceiling-length requests would queue — widening that is a pool
|
||||||
|
# raise, not a window move.
|
||||||
# still binds: the pool must back ONE full-length request (below that the
|
# still binds: the pool must back ONE full-length request (below that the
|
||||||
# advertised per-request ceiling would be unreachable, hence pool >= ctx),
|
# advertised per-request ceiling would be unreachable, hence pool >= ctx),
|
||||||
# it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
|
# it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
|
||||||
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
|
|||||||
[[ "$output" == *"--name frank-dragoon-halogen"* ]]
|
[[ "$output" == *"--name frank-dragoon-halogen"* ]]
|
||||||
[[ "$output" == *"--publish 7770:8731"* ]]
|
[[ "$output" == *"--publish 7770:8731"* ]]
|
||||||
[[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
|
[[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
|
||||||
[[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
|
[[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
|
||||||
[[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
|
[[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
|
||||||
[[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
|
[[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
|
||||||
[[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
|
[[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
|
||||||
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"
|
|||||||
|
|
||||||
@test "the shipped env keeps the harness fit law inside the ceiling" {
|
@test "the shipped env keeps the harness fit law inside the ceiling" {
|
||||||
# HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
|
# HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
|
||||||
# naming the limit, never a silent truncation). The law is INVARIANT to
|
# naming the limit, never a silent truncation). The law: the harness
|
||||||
# the value: the harness window must sit ~45k UNDER it, because the
|
# window must sit ~45k UNDER the ceiling, because the harness clamps
|
||||||
# harness clamps max_tokens = min(45056, window − est − 4096), compacts
|
# max_tokens = min(45056, window − est − 4096), compacts at the trigger
|
||||||
# at 85 % of its declared window, and its chars/4 est UNDER-counts the
|
# percent of its declared window, and its chars/4 est UNDER-counts the
|
||||||
# rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
|
# rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
|
||||||
# est 152,938 on 2026-10-07 — with window == ceiling, every request over
|
# at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
|
||||||
# ~65 % of the window was a guaranteed 400 before compaction could fire).
|
# request is a guaranteed 400 before compaction can fire). The 2026-10-21
|
||||||
# The shipped pair is ctx 200704 / window 155648 (lowered from
|
# pair: ceiling 262144 (native) under a harness window of 200,000.
|
||||||
# 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
|
# The declared window is read from the endpoint page — the one owner —
|
||||||
# monotone maxima of est + budget (est ≤ window − 4096):
|
# so env, models.yml and page that disagree FAIL HERE, not as a 400
|
||||||
# full budget at the compaction threshold: 1.03*132300 + 45056
|
# mid-session (the recurring failure this seam exists for).
|
||||||
# clamped budget at the clamp's zero point: 1.03*151552 + 0
|
grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
|
||||||
# and the compaction request that rides on frank-zealot (its
|
ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||||
# 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
|
page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
|
||||||
grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
|
grep -qE 'contextWindow: 200000' "$page"
|
||||||
ctx=200704
|
hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
|
||||||
hw=155648; mt=45056
|
mt=45056
|
||||||
thr=$(( hw * 85 / 100 ))
|
# Invariant checked at the live 95 % trigger (config.yml
|
||||||
|
# thresholdPercent) — strictly stronger than the 85 % design point.
|
||||||
|
# Both monotone maxima of est + budget (est ≤ window − 4096):
|
||||||
|
thr=$(( hw * 95 / 100 ))
|
||||||
[ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
|
[ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
|
||||||
zero=$(( hw - 4096 ))
|
zero=$(( hw - 4096 ))
|
||||||
[ $(( zero * 103 / 100 )) -le "$ctx" ]
|
[ $(( zero * 103 / 100 )) -le "$ctx" ]
|
||||||
|
# The compaction request rides on frank-zealot's 200,704-token slot:
|
||||||
|
# input ≈ threshold − keepRecent(20k).
|
||||||
[ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
|
[ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -423,9 +430,12 @@ npu_fixture() {
|
|||||||
# the stock root:render 0660 mode, and the serve gate checks exactly that.
|
# the stock root:render 0660 mode, and the serve gate checks exactly that.
|
||||||
touch "$TEST_TMP/accel0"
|
touch "$TEST_TMP/accel0"
|
||||||
chmod 666 "$TEST_TMP/accel0"
|
chmod 666 "$TEST_TMP/accel0"
|
||||||
mkdir -p "$TEST_TMP/xrt"
|
# basename carries a multiarch shape: the third mount prefix is derived
|
||||||
|
# from the XRT dir's basename (the NPU child searches
|
||||||
|
# /opt/xilinx/lib/<multiarch>/, measured 2026-10-21).
|
||||||
|
mkdir -p "$TEST_TMP/x86_64-linux-gnu"
|
||||||
for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
|
for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
|
||||||
touch "$TEST_TMP/xrt/$l"
|
touch "$TEST_TMP/x86_64-linux-gnu/$l"
|
||||||
done
|
done
|
||||||
mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
|
mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
|
||||||
mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
|
mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
|
||||||
@@ -435,7 +445,7 @@ npu_fixture() {
|
|||||||
"HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
|
"HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
|
||||||
"FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
|
"FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
|
||||||
"FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
|
"FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
|
||||||
"FRANK_DRAGOON_XRT_DIR=$TEST_TMP/xrt"
|
"FRANK_DRAGOON_XRT_DIR=$TEST_TMP/x86_64-linux-gnu"
|
||||||
"FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
|
"FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -465,11 +475,15 @@ $(cat "$TEST_TMP/serve.log")"
|
|||||||
! grep -q 'accel0' "$TEST_TMP/serve.log"
|
! grep -q 'accel0' "$TEST_TMP/serve.log"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "the lane arms the accel node, the staged model dir and each XRT library twice" {
|
@test "the lane arms the accel node, the staged model dir and each XRT library three times" {
|
||||||
# Each library mounts at the image's XRT path AND at its own path: the
|
# Each library mounts at the image's XRT path, at its own path (the distro
|
||||||
# distro libraries find each other where they were installed, and an
|
# libraries find each other where they were installed), and at
|
||||||
# /opt/xilinx/xrt of symlinks into the system dir dangles inside the
|
# /opt/xilinx/lib/<multiarch>/ — the prefix the NPU child searches and
|
||||||
# container (the vendor calls that out by name).
|
# names when it is missing (`No such library
|
||||||
|
# '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'`, measured
|
||||||
|
# 2026-10-21: the lane died and the supervisor took the whole container
|
||||||
|
# with it, 88-restart loop). An /opt/xilinx/xrt of symlinks into the
|
||||||
|
# system dir dangles inside the container (the vendor calls that out).
|
||||||
podman_stub
|
podman_stub
|
||||||
good_env
|
good_env
|
||||||
npu_fixture
|
npu_fixture
|
||||||
@@ -484,8 +498,9 @@ $(cat "$TEST_TMP/serve.log")"
|
|||||||
[[ "$output" != *":/models/npu"* ]]
|
[[ "$output" != *":/models/npu"* ]]
|
||||||
[[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
|
[[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
|
||||||
for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
|
for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
|
||||||
[[ "$output" == *"-v $TEST_TMP/xrt/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
|
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
|
||||||
[[ "$output" == *"-v $TEST_TMP/xrt/$l:$TEST_TMP/xrt/$l:ro"* ]]
|
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:$TEST_TMP/x86_64-linux-gnu/$l:ro"* ]]
|
||||||
|
[[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/lib/x86_64-linux-gnu/$l:ro"* ]]
|
||||||
done
|
done
|
||||||
# The Flash payload still rides the same container: the lane is additive.
|
# The Flash payload still rides the same container: the lane is additive.
|
||||||
[[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
|
[[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
|
||||||
@@ -560,7 +575,7 @@ $(cat "$TEST_TMP/serve.log")"
|
|||||||
podman_stub
|
podman_stub
|
||||||
good_env
|
good_env
|
||||||
npu_fixture
|
npu_fixture
|
||||||
rm "$TEST_TMP/xrt/libxrt_driver_xdna.so.2"
|
rm "$TEST_TMP/x86_64-linux-gnu/libxrt_driver_xdna.so.2"
|
||||||
run_with_log
|
run_with_log
|
||||||
[ "$status" -ne 0 ]
|
[ "$status" -ne 0 ]
|
||||||
[[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]
|
[[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]
|
||||||
|
|||||||
@@ -195,3 +195,33 @@ for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
|
|||||||
print('gpu dashboard panels ok')
|
print('gpu dashboard panels ok')
|
||||||
"
|
"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "--serve answers on EVERY declared bind, not just the first" {
|
||||||
|
# The deployed unit serves loopback + the host LAN address, and Prometheus
|
||||||
|
# reaches the host through the gateway — which dials the LAN-class bind.
|
||||||
|
# A server that only runs serve_forever() on the first socket leaves the
|
||||||
|
# second one listen()ing-but-never-accepting: the deploy health gate on
|
||||||
|
# loopback passes, every scrape on the other bind dies on the SYN backlog,
|
||||||
|
# and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
|
||||||
|
# left both gpu-* targets down beside a green deploy.)
|
||||||
|
mk_amd 0 42 1073741824 34359738368 "" 71000
|
||||||
|
port=$((19400 + RANDOM % 500))
|
||||||
|
python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
|
||||||
|
--bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
|
||||||
|
> "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
|
||||||
|
local pid=$!
|
||||||
|
trap 'kill "$pid" 2>/dev/null' EXIT
|
||||||
|
local ok=0
|
||||||
|
for _ in $(seq 1 20); do
|
||||||
|
curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
|
||||||
|
sleep 0.25
|
||||||
|
done
|
||||||
|
[ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
|
||||||
|
# Both binds must ANSWER within 2 s each — the second one is the bug.
|
||||||
|
for b in 127.0.0.1 127.0.0.2; do
|
||||||
|
curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
|
||||||
|
|| { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
|
||||||
|
done
|
||||||
|
kill "$pid" 2>/dev/null
|
||||||
|
trap - EXIT
|
||||||
|
}
|
||||||
|
|||||||
@@ -595,16 +595,22 @@ assert any('ALERTS' in e for e in exprs), exprs
|
|||||||
}
|
}
|
||||||
|
|
||||||
@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
|
@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
|
||||||
# Cross-file contract: cadvisor publishes the LAN port; a cadvisor scrape
|
# Cross-file contract, measured 2026-10-21 from inside the Prometheus
|
||||||
# job pointing anywhere else reads nothing (unlike node-exporter's host-net
|
# container: a bridge container cannot dial its OWN host's LAN IP — the
|
||||||
# bind, a published port IS reachable from this bridge container at the LAN
|
# hairpin refuses published ports exactly like host-net binds, so
|
||||||
# address). And the whole dashboard depends on the id -> container relabel:
|
# cadvisor-dankclaw must scrape via the podman gateway (the original pin
|
||||||
# cadvisor names nothing itself, the quadlet unit name is only in the cgroup
|
# here demanded the LAN address and shipped a permanently-down target).
|
||||||
# path. If either half changes alone, the Containers dashboard is blank.
|
# dankserver is a different machine: plain LAN there. And the whole
|
||||||
|
# dashboard depends on the id -> container relabel: cadvisor names nothing
|
||||||
|
# itself, the quadlet unit name is only in the cgroup path. If either half
|
||||||
|
# changes alone, the Containers dashboard is blank.
|
||||||
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
|
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
|
||||||
grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
|
grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
|
||||||
grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
|
grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
|
||||||
grep -q '"192.168.0.143:8080"' "$cfg" || { echo "cadvisor-dankclaw does not scrape the published LAN port"; return 1; }
|
grep -q '"host.containers.internal:8080"' "$cfg" \
|
||||||
|
|| { echo "cadvisor-dankclaw does not scrape via the podman gateway"; return 1; }
|
||||||
|
! grep -q '"192.168.0.143:8080"' "$cfg" \
|
||||||
|
|| { echo "cadvisor-dankclaw dials its own host LAN IP — hairpin refuses it"; return 1; }
|
||||||
grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
|
grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
|
||||||
# The relabel that turns …/app.slice/<name>.service… into a `container`.
|
# The relabel that turns …/app.slice/<name>.service… into a `container`.
|
||||||
grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
|
grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
|
||||||
|
|||||||
Reference in New Issue
Block a user