Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d4ce76b68b | ||
|
|
65bd5ccea7 | ||
|
|
5976ad0550 | ||
|
|
5ae6b7b928 | ||
|
|
cc0bc06288 | ||
|
|
bdead570e2 | ||
|
|
156ea71879 | ||
|
|
ab80c1366d | ||
|
|
c5a816e6a8 | ||
|
|
cb25796a6b | ||
|
|
f31e7c5ed8 | ||
|
|
028afdeb58 | ||
|
|
acf2a5cee5 |
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: revert the swap commit (w4dn8 `.rad` on disk); the fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
|
||||
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
|
||||
|
||||
@@ -35,11 +35,16 @@ the [runbook index](#runbook-index). The full swap history lives in
|
||||
|
||||
## Repository Layout
|
||||
|
||||
- `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack
|
||||
- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack
|
||||
- `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs
|
||||
- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
|
||||
scripts, systemd units, env tunables, bench harnesses
|
||||
- `dankserver/` — quadlet units + Caddyfile for the media stack (the
|
||||
compose.yaml inventory retired 2026-10-13; its mounts had diverged from
|
||||
the quadlets)
|
||||
- `dankclaw/` — quadlet units + Caddyfile for the AI stack (compose retired 2026-09-16)
|
||||
- `shared/` — the service inventory (`services.yml`), homepage, Prometheus,
|
||||
Loki, Alloy, SearXNG configs
|
||||
- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh` and
|
||||
`service-map.py` (the inventory's only reader), LLM serve scripts + the
|
||||
shared artifact gates (`scripts/lib/llm-gates.sh`), systemd units, env
|
||||
tunables, bench harnesses
|
||||
- `old/` — retired LLM scripts (reference only)
|
||||
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
|
||||
The managed skill library is distilled there one page per skill-cluster:
|
||||
@@ -70,8 +75,13 @@ All edits flow through git. Never edit files on a remote machine directly.
|
||||
./scripts/deploy.sh dankclaw --dry-run
|
||||
```
|
||||
|
||||
The planner maps changed paths to services; the remote runner restarts only
|
||||
the selected service and records its revision outside the checkout.
|
||||
The inventory (`shared/services.yml`, read only through
|
||||
`scripts/service-map.py`) maps changed paths to services; the remote runner
|
||||
restarts only the selected service and records its revision outside the
|
||||
checkout. Both `--service` allow-lists, the planner, and the changed-test
|
||||
seam mapper ARE the inventory — adding, moving, or retiring a service is a
|
||||
data edit, not a script edit
|
||||
(`serverAppConfigs-docs/Architecture/Service Inventory.md`).
|
||||
Documentation-only changes are a no-op. LLM services are selected only by
|
||||
their own files — every endpoint (`frank-dragoon`/`frank-zealot`/`frank-archon`)
|
||||
by its unit/env/serve trio in `scripts/` — so
|
||||
@@ -164,12 +174,17 @@ Tests are mandatory before commit (`bats` required on the dev machine):
|
||||
./tests/run.sh
|
||||
```
|
||||
|
||||
Seams: compose validity, env consistency (no leaked secrets), Caddyfile
|
||||
Seams: env consistency (no leaked secrets), Caddyfile
|
||||
validity, quadlet units, docs completeness **including vault link integrity**,
|
||||
script syntax (shell + python), LLM unit integrity, LLM secret hygiene,
|
||||
deployment flow, service-scoped planning, homepage config/pages/deploy/checker,
|
||||
and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in
|
||||
no project, because such a spec silently never runs.
|
||||
LLM artifact gates (`llm_gates`/`llm_deploy_gates` — the shared
|
||||
`scripts/lib/llm-gates.sh` every endpoint's serve and deploy gates run on),
|
||||
deployment flow, service-scoped planning, the **service inventory**
|
||||
(`service_inventory.bats`: inventory soundness, the pinned path->service
|
||||
plan fixture, render drift for the Pi-hole hosts file), homepage
|
||||
config/pages/deploy/checker, and `e2e_scaffold` — which fails if a
|
||||
Playwright spec file's number prefix is in no project, because such a spec
|
||||
silently never runs.
|
||||
See `serverAppConfigs-docs/Architecture/Testing Policy.md`.
|
||||
|
||||
## Secret Rules
|
||||
@@ -498,71 +513,48 @@ Rollback: revert the swap commit — the Gemma4 payload
|
||||
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
|
||||
|
||||
|
||||
### frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, TP=2 (Dankserver, ACTIVE)
|
||||
### frank-archon — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
|
||||
|
||||
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
||||
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
|
||||
for serve; `libr4d` gfx1201 kernels, plain `podman run`, root
|
||||
systemd unit, container name = unit name) serving the
|
||||
**publisher's container of the official Qwen/Qwen3.8-Flash-Next**:
|
||||
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` →
|
||||
`qwen3.8-next-flash-fp8-iq4r-moe.rad` (116,322 MiB, Hub-LFS sha256
|
||||
verified at fetch by `scripts/download-frank-archon-iq4r.sh`; recipe
|
||||
`qwen4exp-w4nl64-i8-hc8m`): **routed experts as 4-bit codes into a
|
||||
16-level non-uniform codebook** after a 128-point Walsh–Hadamard rotation
|
||||
(E4M3 scale per 64 weights; codes by GPTQ on a 10M-token calibration with
|
||||
per-expert down-proj Hessians, refined by 3 coordinate-descent sweeps),
|
||||
**10 whole bf16-protected experts/layer**, **int8 W8A8 trunk/lm_head**
|
||||
(scale per 128 columns searched), **fp8 HC mixtures**, PLE n-gram table at
|
||||
fp8 fixed scale (47.7 GiB), **MTP head and bf16 vision tower in the
|
||||
container**. Card quality (full-vocab teacher-forced KL vs bf16, ~75K
|
||||
positions): mean 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %.
|
||||
**MODEL_FILE is the only serving argument that moved** (2026-10-15); every
|
||||
number of the 2026-10-13 shape below carries verbatim. The plugin's
|
||||
**one-table-one-encoding law** (fleet-measured at rad-convert declare):
|
||||
one expert table takes ONE encoding; only whole bf16 experts are legal
|
||||
beside it. Rollback: revert the swap commit — the in-house w4dn8 `.rad`
|
||||
(KLD-gated Swift1.5 conversion) stays on disk; the fp8-native `.rad` was
|
||||
deleted 2026-10-15 (rebuild: `scripts/download-frank-archon.sh` +
|
||||
`scripts/build-frank-archon-rad.sh`).
|
||||
for serve/convert/KLD; `libr4d` gfx1201 kernels, plain `podman run`, root
|
||||
systemd unit, container name = unit name) serving
|
||||
**ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB, 131 shards,
|
||||
397/397 sha256-verified)** converted by `scripts/convert-swift15-w4dn8.sh`
|
||||
with recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` to
|
||||
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ
|
||||
w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine
|
||||
**1.2.0** via `RADIANCE_CALIB_DIR`/`RADIANCE_CALIB_TOKENS` — the only pin
|
||||
with the token knob; the RADGRAM2 grams are format-identical across pins and
|
||||
cross the boundary by design), **24 whole experts per layer cast bf16** (the
|
||||
PROTECTED set, derived from the calibration grams' energy share),
|
||||
**int8 trunk/lm_head**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE n-gram
|
||||
table at fp8 fixed scale (47.7 GiB in the container, from 95.4 GiB bf16).
|
||||
The plugin's **one-table-one-encoding law** (measured at rad-convert
|
||||
declare): one expert table takes ONE encoding; only whole bf16 experts are
|
||||
legal beside it — no w4/fp8 split of the down/gate_up pair (24-value
|
||||
`rad_arch_quant_desc` array, one per layer). KLD gate vs the BF16
|
||||
reference (recorded on the same 1.0.13 pin) is the precision referee;
|
||||
the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
|
||||
(122–173 t/s at 44.5k depth). Measured decode/prefill in the runbook.
|
||||
Rollback: revert the swap commit — the fp8-native `.rad`
|
||||
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
|
||||
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 2`,
|
||||
`--max-model-len 200000` (the harness contextWindow must quote this),
|
||||
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
|
||||
partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
|
||||
partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair
|
||||
replaced `--expert-vs-cache-ratio`; the two are ONE budget —
|
||||
`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
|
||||
32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
|
||||
209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
|
||||
claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
|
||||
hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
|
||||
because it is the only dtype that funds the slot count at this window**,
|
||||
`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
|
||||
8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
|
||||
**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
|
||||
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
|
||||
ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
|
||||
all at 200000 tokens at once, which is the worst case` — `kv_attn` page
|
||||
26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
|
||||
`kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
|
||||
(3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
|
||||
(3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
|
||||
563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
|
||||
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
|
||||
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
|
||||
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
|
||||
4.22 GiB to the expert cache. Prefill tracks this slab only up to a
|
||||
**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
|
||||
(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
|
||||
(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
|
||||
(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
|
||||
not a slope.
|
||||
|
||||
**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
|
||||
single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
|
||||
4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
|
||||
aggregate**, so the slot count is paid in decode concurrency (a fused-step
|
||||
lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
|
||||
`weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710
|
||||
consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype`
|
||||
**`bf16` — unquantized KV since 2026-10-11**, `--num-speculative-tokens 2`,
|
||||
`--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB
|
||||
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
|
||||
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
|
||||
`--override-chat-template`). Boot ledger: KV pool 6.75 GiB = **462,752 bf16
|
||||
tokens, "backs 2 sequences all at 200000 tokens at once"** (the fp8 pool's
|
||||
386,904 backed 1.93), weights slab filled to 147 KiB of slack, host pool
|
||||
10,375 slots, cards settled 1.85 GiB free.
|
||||
|
||||
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
|
||||
@@ -574,52 +566,35 @@ on the fp8 shape (512-token streamed probe). The campaign also
|
||||
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
|
||||
accept 0.7246).
|
||||
|
||||
**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
|
||||
`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
|
||||
llama.cpp's exact-K/quantized-V pair (what frank-zealot runs) has NO radiance
|
||||
equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
|
||||
the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
|
||||
KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
|
||||
block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
|
||||
and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
|
||||
dtype-independent terms that live in the same pool (~110 MiB a sequence of
|
||||
recurrent state + 448 MiB of prefix checkpoints):
|
||||
`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
|
||||
engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
|
||||
gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
|
||||
the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
|
||||
bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
|
||||
where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
|
||||
half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
|
||||
pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
|
||||
`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
|
||||
25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
|
||||
the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
|
||||
law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
|
||||
weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
|
||||
pair, never `--kv-cache-size-mib`.
|
||||
|
||||
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
|
||||
prefix cache, and the rule that a prompt over `--max-model-len` is
|
||||
refused, never truncated. What changed: the fp8-native era's NVMe expert
|
||||
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
|
||||
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
|
||||
on the fp8 shape (512-token streamed probe). The campaign also
|
||||
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
|
||||
accept 0.7246).
|
||||
**The bf16 KV law (2026-10-11)**: `--kv-cache-dtype` takes ONLY `bf16|fp8` —
|
||||
one dtype for the whole cache, so llama.cpp's exact-K/quantized-V pair (what
|
||||
frank-zealot runs) has NO radiance equivalent; the rung under bf16 is fp8 on
|
||||
both halves. Measured per rank from the two boot ledgers: `kv_attn` (13
|
||||
layers, 4-token blocks) pages 26.00 → 52.00 KiB, while `kv_qsa_bkey` (13
|
||||
layers) stays 3.25 KiB at BOTH dtypes (the QSA block key is an index, not
|
||||
attention state) — so a token costs 7.3125 KiB at fp8 and 13.8125 KiB at
|
||||
bf16: **1.89×, not 2×**. Pool sizing must also pay the dtype-independent terms
|
||||
that live in the same pool (~110 MiB a sequence of recurrent state + 448 MiB
|
||||
of prefix checkpoints): `(vram_kv − 110·seqs − 448) MiB / 13.8125 KiB` =
|
||||
462,946 where the engine reported 462,752 (0.04 % page rounding). The cost is
|
||||
paid twice on this engine, and half-refunded: idle KV VRAM is also the expert
|
||||
cache's spillover, and the bigger pool *lends* 6.08 GiB to the slab where the
|
||||
fp8 pool lent 2.69 GiB. `--max-num-seqs 3 → 2` came with the change (two slots
|
||||
× the window is the capacity contract; the "seqs 2 cannot boot" law belongs to
|
||||
the retired tcclaviger fork's torch.compile, not radiance) and it also drops
|
||||
the fused all-reduce to 2 × 3 = 6 rows, under the 25-row bound where wht6 is
|
||||
bit-exact. Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
|
||||
buffers + arena onto the weights side, so `weights + kv` is capped near
|
||||
29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
|
||||
|
||||
**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
|
||||
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
|
||||
the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
|
||||
`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
|
||||
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
|
||||
runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
|
||||
by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
|
||||
before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
|
||||
and the preflight re-refused 6,919 times. A manual sysfs write in the stop
|
||||
window is a convenience, not the repair; kernel cmdline persistence
|
||||
(`ttm.pages_limit=52428800`) is the belt to it.
|
||||
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
|
||||
script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=150000`, cap on this host
|
||||
204,800) refuses to start until both knobs are raised in the stop window
|
||||
(sysfs write first, kernel cmdline + reboot only if read-only).
|
||||
|
||||
**Contract facts** (source-verified): the HTTP listener binds AFTER the
|
||||
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
|
||||
@@ -631,9 +606,8 @@ chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
|
||||
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
|
||||
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md`.
|
||||
Retired Swift1.5 w4dn8 era (payload on disk = rollback): `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
|
||||
Retired fp8-native era (payload deleted 2026-10-15, rebuildable): `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
|
||||
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
|
||||
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
|
||||
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md`.
|
||||
@@ -802,17 +776,6 @@ linked runbooks.
|
||||
the KV pool is worse than the OOM it prevents.
|
||||
- Host swap state is invisible to `deploy.sh` (no repo file) — check
|
||||
`swapon --show` before reasoning about headroom.
|
||||
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
|
||||
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
|
||||
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
|
||||
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
|
||||
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
|
||||
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
|
||||
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
|
||||
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
|
||||
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
|
||||
after a write the kernel rejected. Never prescribe a manual sysfs write as the
|
||||
recovery procedure for a service that can boot before a human reaches it.
|
||||
- **Silent RAM corruption outranks every software theory.** frank-archon's
|
||||
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
|
||||
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
|
||||
@@ -1093,7 +1056,7 @@ Active:
|
||||
|---------|---------|
|
||||
| frank-dragoon (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md` |
|
||||
| frank-zealot (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
|
||||
| frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` |
|
||||
| frank-archon (radiance StillDeadcode, Swift1.5-Qwen3.8-Flash-Next GPTQ w4 + bf16-protected experts, int8 trunk, bf16 KV, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
@@ -1101,8 +1064,8 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (`frank-archon`) | Dankserver | 2026-10-08 → 10-15 (rollback target — `.rad` on disk; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim) | `frank-archon-radiance-swift15-w4dn8.md` |
|
||||
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` |
|
||||
| radiance StillDeadcode Qwen3.8-Flash-Next native-FP8, 2×R9700 TP=2 (`frank-archon`) | Dankserver | 2026-10-08 → w4dn8 swap (rollback target — `.rad` payload on disk) | `frank-archon-radiance-qwen38-native-fp8.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-archon`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-archon-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-archon`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-archon-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-archon-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-archon-freetoken-pr587.md` |
|
||||
|
||||
+24
-27
@@ -56,6 +56,21 @@
|
||||
}
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# Named snippet: the pre-auth guard chain. Anubis (:8923) challenges,
|
||||
# the embedded Authentik outpost authenticates, and authenticated
|
||||
# traffic returns to the http://:8017 listener's post-auth handlers.
|
||||
# Every guarded vhost is `import guarded` — the chain is EDITED HERE,
|
||||
# never inlined per site (rank 6: it used to be nine verbatim copies).
|
||||
# Exemptions live in their own vhosts (auth. is the IdP itself; git
|
||||
# smart-HTTP and self-authenticating services document their reason).
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
(guarded) {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
}
|
||||
|
||||
# --- Bare-IP access (health checks, convenience) ---
|
||||
:80 {
|
||||
redir https://www.franksdank.cloud/ 308
|
||||
@@ -72,9 +87,7 @@ franksdank.cloud {
|
||||
}
|
||||
|
||||
www.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- Legacy name: the old homepage name redirects straight onto www ---
|
||||
@@ -100,9 +113,7 @@ auth.franksdank.cloud {
|
||||
|
||||
# --- SearXNG (quadlet; behind the Gluetun tunnel, published 8081) ---
|
||||
search.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- OpenWebUI (quadlet on slirp4netns; published 3005) ---
|
||||
@@ -126,30 +137,22 @@ gitea.franksdank.cloud {
|
||||
# never reach Grafana — RP mode needs none and works regardless of topology.
|
||||
# The basicauth edge this replaces (CADDY_BASICAUTH_HASH) is retired.
|
||||
grafana.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- Prometheus ---
|
||||
prometheus.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- Loki (guarded: Anubis → Authentik; loki-proxy provider, internal_host caddy:8017) ---
|
||||
loki.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- Seerr (guarded: Anubis → Authentik; seerr provider, internal_host caddy:8017) ---
|
||||
seerr.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- Marreta (guarded: Anubis → Authentik; marreta provider, internal_host caddy:8017) ---
|
||||
@@ -157,9 +160,7 @@ seerr.franksdank.cloud {
|
||||
# group gate). Egress rides the searxng gluetun tunnel; reached via the
|
||||
# host-published port 8082 (gluetun netns), not by container name.
|
||||
marreta.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- Pi-hole (quadlet on slirp4netns; published 8083) ---
|
||||
@@ -233,16 +234,12 @@ books.franksdank.cloud {
|
||||
|
||||
# --- RomM ---
|
||||
romm.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- Copyparty ---
|
||||
files.franksdank.cloud {
|
||||
reverse_proxy anubis:8923 {
|
||||
header_up X-Real-Ip {remote_host}
|
||||
}
|
||||
import guarded
|
||||
}
|
||||
|
||||
# --- Jellyfin ---
|
||||
|
||||
@@ -1,30 +1,32 @@
|
||||
# Quadlet unit: node-exporter (rootless, boot-persistent).
|
||||
# Quadlet unit: node-exporter (rootless, boot-persistent) — RENDERED ARTIFACT.
|
||||
# Regenerate: python3 scripts/service-map.py render node-exporter-unit --host dankclaw
|
||||
# Source: shared/services.yml (entry node-exporter) + this template.
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/node-exporter.container
|
||||
# Unit name: node-exporter.service (container name: node-exporter)
|
||||
#
|
||||
# Host metrics source for Prometheus. This replaces the compose service of the
|
||||
# same name in dankclaw/compose.yaml ("Observability" block).
|
||||
# Host metrics source for Prometheus. Replaces the compose service of the same
|
||||
# name (the "Observability" block): Dankclaw moved to this quadlet on
|
||||
# 2026-09-11, Dankserver on 2026-09-16 — its compose-only copy died at a reboot
|
||||
# and stayed dead (rootless Podman does not restart bare containers after the
|
||||
# explicit stop shutdown performs), dead for a day and a half before anyone
|
||||
# noticed. One owner per container name; no compose file remains anywhere.
|
||||
#
|
||||
# HOST NAMESPACE. It must run in the host network namespace to report the
|
||||
# host's own interfaces/connections, and --pid=host for the processes and
|
||||
# cgroups collectors. "localhost" inside the container is therefore the host's
|
||||
# loopback, which is what the homepage status timer probes.
|
||||
#
|
||||
# LISTEN ADDRESSES. Until 2026-09-11 this bound :9100 on every interface. It
|
||||
# now binds exactly two: loopback for the homepage status probe, and the LAN
|
||||
# address because Prometheus — which runs in a container on dankclaw_default —
|
||||
# scrapes http://192.168.0.143:9100 (shared/prometheus/prometheus.yml,
|
||||
# job "node-dankclaw"). node_exporter itself has no authentication, so keeping
|
||||
# the wildcard bind off the table matters: this is the second half of the
|
||||
# pairing that removes the unauthenticated observability surface, and both
|
||||
# consumers are accounted for. Dankserver runs its own copy of this container
|
||||
# (scraped by the "node-dankserver" job) — since 2026-09-16 as a Quadlet unit in
|
||||
# the same shape (dankserver/observability/quadlet/), because the compose-only
|
||||
# version it replaced had been dead for a day and a half before anyone noticed.
|
||||
# LISTEN ADDRESSES. Exactly two: loopback for the homepage status probe, and
|
||||
# the LAN address, because Prometheus — which runs in a container on
|
||||
# dankclaw_default — scrapes http://192.168.0.143:9100/metrics
|
||||
# (shared/prometheus/prometheus.yml, job "node-dankclaw"). node_exporter has no
|
||||
# authentication, so the wildcard bind stays off the table; this is the
|
||||
# pairing that removes the unauthenticated observability surface on every
|
||||
# host, and both consumers are accounted for.
|
||||
#
|
||||
# MOUNTS. The container reads the host's /proc, /sys and / as /host/* read-only
|
||||
# and is pointed at them with --path.*; it writes nothing anywhere.
|
||||
# MOUNTS. The container reads the host's /proc, /sys and / as /host/*
|
||||
# read-only and is pointed at them with --path.*; it writes nothing anywhere.
|
||||
|
||||
[Unit]
|
||||
Description=Prometheus node exporter (Dankclaw host metrics)
|
||||
|
||||
+30
-34
@@ -1,47 +1,43 @@
|
||||
# A records for the flat franksdank.name scheme (dnsmasq addn-hosts format:
|
||||
# <IP> <name> [<name> ...]). Served by the Pi-hole quadlet (FTL is a
|
||||
# dnsmasq fork): the quadlet mounts this file into /etc/pihole/hosts/ read-only.
|
||||
# A records for the flat franksdank.cloud scheme (dnsmasq addn-hosts
|
||||
# format: <IP> <name>). RENDERED — do not edit by hand:
|
||||
#
|
||||
# Caddy is consolidated on dankclaw (192.168.0.143): it fronts every service
|
||||
# on both hosts, so every name resolves to that one IP. Keep in sync with the
|
||||
# Caddyfile (tests/pihole.bats cross-checks this).
|
||||
# scripts/service-map.py render pihole-hosts > dankclaw/pihole/addn-hosts
|
||||
#
|
||||
# 192.168.0.143 = dankclaw (runs the Caddy container + the AI/dev stack)
|
||||
# 192.168.0.235 = dankserver (media stack; reached by Caddy via published
|
||||
# LAN ports — no DNS records point here)
|
||||
# The source of truth is the `dns:` field of shared/services.yml; the
|
||||
# drift gate in tests/service_inventory.bats fails when this file and
|
||||
# the inventory disagree. Caddy is consolidated on Dankclaw, so every
|
||||
# name — including the services running on dankserver and
|
||||
# frankbook-pro — resolves at the Dankclaw edge.
|
||||
#
|
||||
# dnsmasq strips ONLY whole-line comments in addn-hosts: a trailing
|
||||
# comment on a data line becomes a HOSTNAME token. No inline notes
|
||||
# here; the service attribution lives in the inventory.
|
||||
|
||||
# --- Homepage (naked domain + www) + legacy names ---
|
||||
192.168.0.143 search.franksdank.cloud
|
||||
192.168.0.143 marreta.franksdank.cloud
|
||||
192.168.0.143 pihole.franksdank.cloud
|
||||
192.168.0.143 ai.franksdank.cloud
|
||||
192.168.0.143 gitea.franksdank.cloud
|
||||
192.168.0.143 comfyui.franksdank.cloud
|
||||
192.168.0.143 seerr.franksdank.cloud
|
||||
192.168.0.143 hermes.franksdank.cloud
|
||||
192.168.0.143 hermesadmin.franksdank.cloud
|
||||
192.168.0.143 prometheus.franksdank.cloud
|
||||
192.168.0.143 grafana.franksdank.cloud
|
||||
192.168.0.143 loki.franksdank.cloud
|
||||
192.168.0.143 auth.franksdank.cloud
|
||||
192.168.0.143 franksdank.cloud
|
||||
192.168.0.143 www.franksdank.cloud
|
||||
192.168.0.143 dankclaw.franksdank.cloud
|
||||
192.168.0.143 dankserver.franksdank.cloud
|
||||
|
||||
# --- Authentik (SSO / OIDC provider) ---
|
||||
192.168.0.143 auth.franksdank.cloud
|
||||
|
||||
# --- Media services (dankserver, proxied) ---
|
||||
192.168.0.143 plex.franksdank.cloud
|
||||
192.168.0.143 plex.franksdank.casa
|
||||
192.168.0.143 books.franksdank.cloud
|
||||
192.168.0.143 romm.franksdank.cloud
|
||||
192.168.0.143 files.franksdank.cloud
|
||||
192.168.0.143 jelly.franksdank.cloud
|
||||
192.168.0.143 books.franksdank.cloud
|
||||
192.168.0.143 plex.franksdank.cloud
|
||||
192.168.0.143 plex.franksdank.casa
|
||||
192.168.0.143 prowlarr.franksdank.cloud
|
||||
192.168.0.143 qbittorrent.franksdank.cloud
|
||||
192.168.0.143 radarr.franksdank.cloud
|
||||
192.168.0.143 lidarr.franksdank.cloud
|
||||
192.168.0.143 readarr.franksdank.cloud
|
||||
192.168.0.143 prowlarr.franksdank.cloud
|
||||
192.168.0.143 qbittorrent.franksdank.cloud
|
||||
|
||||
# --- AI / search / dev services (dankclaw, local) ---
|
||||
192.168.0.143 search.franksdank.cloud
|
||||
192.168.0.143 comfyui.franksdank.cloud
|
||||
192.168.0.143 ai.franksdank.cloud
|
||||
192.168.0.143 hermes.franksdank.cloud
|
||||
192.168.0.143 hermesadmin.franksdank.cloud
|
||||
192.168.0.143 gitea.franksdank.cloud
|
||||
192.168.0.143 grafana.franksdank.cloud
|
||||
192.168.0.143 prometheus.franksdank.cloud
|
||||
192.168.0.143 loki.franksdank.cloud
|
||||
192.168.0.143 pihole.franksdank.cloud
|
||||
192.168.0.143 seerr.franksdank.cloud
|
||||
192.168.0.143 marreta.franksdank.cloud
|
||||
|
||||
@@ -1,34 +1,32 @@
|
||||
# Quadlet unit: node-exporter (rootless, boot-persistent) — DANKSERVER.
|
||||
# Quadlet unit: node-exporter (rootless, boot-persistent) — RENDERED ARTIFACT.
|
||||
# Regenerate: python3 scripts/service-map.py render node-exporter-unit --host dankserver
|
||||
# Source: shared/services.yml (entry node-exporter) + this template.
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/node-exporter.container
|
||||
# Unit name: node-exporter.service (container name: node-exporter)
|
||||
#
|
||||
# WHY THIS FILE EXISTS. Until 2026-09-16 the Dankserver node-exporter ran only
|
||||
# from `dankserver/compose.yaml` (the "Observability" block, `restart:
|
||||
# unless-stopped`). Rootless Podman does not restart bare containers after the
|
||||
# explicit stop that shutdown performs, so it died at a reboot and stayed dead:
|
||||
# `podman ps -a` showed it `Exited (143)` and Prometheus' `node-dankserver` job
|
||||
# had been scraping nothing since. The compose entry also bound the default
|
||||
# wildcard address, which is the unauthenticated-metrics-API-on-every-interface
|
||||
# posture that was already removed on Dankclaw. This unit replaces both: one
|
||||
# owner per container name (the compose service is deleted), explicit listen
|
||||
# addresses, and a generator-provided boot path.
|
||||
# Host metrics source for Prometheus. Replaces the compose service of the same
|
||||
# name (the "Observability" block): Dankclaw moved to this quadlet on
|
||||
# 2026-09-11, Dankserver on 2026-09-16 — its compose-only copy died at a reboot
|
||||
# and stayed dead (rootless Podman does not restart bare containers after the
|
||||
# explicit stop shutdown performs), dead for a day and a half before anyone
|
||||
# noticed. One owner per container name; no compose file remains anywhere.
|
||||
#
|
||||
# HOST NAMESPACE. Networking and PID in the host's namespaces: the collectors
|
||||
# report the host's own interfaces, connections, processes and cgroups. The
|
||||
# host's /proc, /sys and / are mounted read-only under /host and /rootfs, and
|
||||
# node_exporter writes nothing.
|
||||
# HOST NAMESPACE. It must run in the host network namespace to report the
|
||||
# host's own interfaces/connections, and --pid=host for the processes and
|
||||
# cgroups collectors. "localhost" inside the container is therefore the host's
|
||||
# loopback, which is what the homepage status timer probes.
|
||||
#
|
||||
# LISTEN ADDRESSES. Exactly two:
|
||||
# - 127.0.0.1:9100 for host-local probes
|
||||
# - 192.168.0.235:9100 for Prometheus, which scrapes this host from Dankclaw
|
||||
# (shared/prometheus/prometheus.yml, job "node-dankserver"). It cannot use
|
||||
# loopback across hosts, and node_exporter has no authentication, so the
|
||||
# wildcard bind stays off the table. This is the same deliberate LAN
|
||||
# listener posture the observability runbook documents.
|
||||
# LISTEN ADDRESSES. Exactly two: loopback for the homepage status probe, and
|
||||
# the LAN address, because Prometheus — which runs in a container on
|
||||
# dankclaw_default — scrapes http://192.168.0.235:9100/metrics
|
||||
# (shared/prometheus/prometheus.yml, job "node-dankserver"). node_exporter has no
|
||||
# authentication, so the wildcard bind stays off the table; this is the
|
||||
# pairing that removes the unauthenticated observability surface on every
|
||||
# host, and both consumers are accounted for.
|
||||
#
|
||||
# MIRROR. Keep this file in step with dankclaw/observability/quadlet/
|
||||
# node-exporter.container; only the Description and the LAN address differ.
|
||||
# MOUNTS. The container reads the host's /proc, /sys and / as /host/*
|
||||
# read-only and is pointed at them with --path.*; it writes nothing anywhere.
|
||||
|
||||
[Unit]
|
||||
Description=Prometheus node exporter (Dankserver host metrics)
|
||||
|
||||
@@ -26,8 +26,8 @@ RECIPE=/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn
|
||||
CALIB=/home/dank/corpora/calib
|
||||
OUT=/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||
LOG=/home/dank/logs/swift15_convert.log
|
||||
# gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert table;
|
||||
# the plugin's one-encoding law dropped the block-fp8 down tier -- see the recipe header).
|
||||
# gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert
|
||||
# table; one encoding per expert table rules out a block-fp8 down tier -- see the recipe header).
|
||||
MIN_MIB=${FRANK_SWIFT15_RAD_MIN_MIB:-108000}
|
||||
|
||||
# Gate 1: inputs. A missing Gram directory converts UNCALIBRATED with a
|
||||
|
||||
+54
-461
@@ -1,475 +1,68 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Plan runtime services affected by repository changes.
|
||||
"""deploy-plan.py — map changed paths to deployable services.
|
||||
|
||||
The planner is intentionally side-effect free. It accepts either an explicit
|
||||
path list or two Git revisions. The latter lets compose-file changes resolve
|
||||
to only the service definitions that actually changed instead of restarting an
|
||||
entire compose project.
|
||||
Thin wrapper over scripts/service-map.py, which owns the path->service
|
||||
semantics in the shared inventory (shared/services.yml): per-service path
|
||||
rules, the quadlet netns/network coupling graph, host scoping (a host's
|
||||
directory can only select that host's services), and canonical restart
|
||||
order. This file keeps the deployer's CLI contract only.
|
||||
|
||||
Usage:
|
||||
deploy-plan.py --target NAME --repo DIR --base REV --head REV
|
||||
deploy-plan.py --target NAME [paths...] (paths also on stdin)
|
||||
|
||||
Prints one service name per line (canonical restart order); exit 0 with no
|
||||
output when nothing relevant changed.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
# Dependency-sensitive services come before their consumers. Independent
|
||||
# compose services follow. LLM services are deliberately last and are never
|
||||
# implied by a generic host, stack, documentation, or config change.
|
||||
SERVICE_ORDER = (
|
||||
"gluetun",
|
||||
"searxng-valkey",
|
||||
"searxng",
|
||||
"marreta",
|
||||
"pihole",
|
||||
"open-webui",
|
||||
"gitea-postgres",
|
||||
"gitea-server",
|
||||
"comfyui",
|
||||
"seerr",
|
||||
"hermes",
|
||||
"hermes-dashboard",
|
||||
"hermes-webui",
|
||||
"prometheus",
|
||||
"grafana",
|
||||
"loki",
|
||||
"alloy",
|
||||
"node-exporter",
|
||||
"alertmanager",
|
||||
"authentik-postgres",
|
||||
"authentik-server",
|
||||
"authentik-worker",
|
||||
"authentik-outpost",
|
||||
"authentik-outpost-ldap",
|
||||
"caddy",
|
||||
"homepage",
|
||||
"romm-mariadb",
|
||||
"romm-redis",
|
||||
"romm",
|
||||
"copyparty",
|
||||
"jellyfin",
|
||||
"calibre-web",
|
||||
"calibre-server",
|
||||
"plex",
|
||||
"tautulli",
|
||||
"prowlarr",
|
||||
"qbittorrent",
|
||||
"radarr",
|
||||
"sonarr",
|
||||
"lidarr",
|
||||
"readarr",
|
||||
"frank-dragoon",
|
||||
"frank-zealot",
|
||||
"frank-archon",
|
||||
)
|
||||
|
||||
DANKCLAW_SERVICES = {
|
||||
"hermes-webui",
|
||||
"gluetun",
|
||||
"searxng-valkey",
|
||||
"searxng",
|
||||
"marreta",
|
||||
"pihole",
|
||||
"open-webui",
|
||||
"gitea-postgres",
|
||||
"gitea-server",
|
||||
"comfyui",
|
||||
"seerr",
|
||||
"hermes",
|
||||
"hermes-dashboard",
|
||||
"prometheus",
|
||||
"grafana",
|
||||
"loki",
|
||||
"alloy",
|
||||
"node-exporter",
|
||||
"alertmanager",
|
||||
"authentik-postgres",
|
||||
"authentik-server",
|
||||
"authentik-worker",
|
||||
"authentik-outpost",
|
||||
"authentik-outpost-ldap",
|
||||
"caddy",
|
||||
"homepage",
|
||||
"frank-dragoon",
|
||||
"frank-zealot",
|
||||
}
|
||||
|
||||
DANKSERVER_SERVICES = {
|
||||
"romm-mariadb",
|
||||
"romm-redis",
|
||||
"romm",
|
||||
"copyparty",
|
||||
"jellyfin",
|
||||
"calibre-web",
|
||||
"calibre-server",
|
||||
"plex",
|
||||
"tautulli",
|
||||
"prowlarr",
|
||||
"qbittorrent",
|
||||
"radarr",
|
||||
"sonarr",
|
||||
"lidarr",
|
||||
"readarr",
|
||||
"frank-archon",
|
||||
"node-exporter",
|
||||
}
|
||||
|
||||
# FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
|
||||
# macOS. Service names overlap the Linux hosts (gluetun, qbittorrent, ...);
|
||||
# the per-target sets and the frankbook-pro intercept in deploy-remote.sh
|
||||
# keep dispatch unambiguous.
|
||||
FRANKBOOK_SERVICES = {
|
||||
"gluetun",
|
||||
"qbittorrent",
|
||||
"prowlarr",
|
||||
"radarr",
|
||||
"sonarr",
|
||||
"lidarr",
|
||||
"readarr",
|
||||
}
|
||||
|
||||
DANKCLAW_COMPOSE_SERVICES = set()
|
||||
# Dankserver's compose file is still the source of truth for its media stack;
|
||||
# Dankclaw's is gone (deleted 2026-09-16). Every service on Dankclaw is a
|
||||
# Quadlet unit now — the last compose services were the hermes trio, whose units
|
||||
# have owned their containers since 2026-09-12 and whose compose blocks were
|
||||
# dead weight that could only create a duplicate container.
|
||||
|
||||
DANKSERVER_COMPOSE_SERVICES = {
|
||||
"mariadb",
|
||||
"redis",
|
||||
"romm",
|
||||
"copyparty",
|
||||
"jellyfin",
|
||||
"calibre-web",
|
||||
}
|
||||
COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
|
||||
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
SERVICE_MAP = os.path.join(SCRIPT_DIR, "service-map.py")
|
||||
|
||||
|
||||
|
||||
def _add(services: set[str], *names: str) -> None:
|
||||
services.update(names)
|
||||
|
||||
|
||||
def target_services(target: str) -> set[str]:
|
||||
if target == "dankclaw":
|
||||
return DANKCLAW_SERVICES
|
||||
if target == "dankserver":
|
||||
return DANKSERVER_SERVICES
|
||||
if target == "frankbook-pro":
|
||||
return FRANKBOOK_SERVICES
|
||||
raise ValueError(f"unknown target: {target}")
|
||||
|
||||
|
||||
def _compose_services(target: str) -> set[str]:
|
||||
return DANKCLAW_COMPOSE_SERVICES if target == "dankclaw" else DANKSERVER_COMPOSE_SERVICES
|
||||
|
||||
|
||||
def _git_show(repo: str, revision: str, path: str) -> str | None:
|
||||
result = subprocess.run(
|
||||
["git", "-C", repo, "show", f"{revision}:{path}"],
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL,
|
||||
check=False,
|
||||
)
|
||||
return result.stdout if result.returncode == 0 else None
|
||||
|
||||
|
||||
def _yaml_document(contents: str | None) -> dict[str, Any]:
|
||||
if contents is None:
|
||||
return {}
|
||||
try:
|
||||
data = yaml.safe_load(contents) or {}
|
||||
except yaml.YAMLError:
|
||||
return {}
|
||||
return data if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def _yaml_services(contents: str | None) -> dict[str, Any]:
|
||||
services = _yaml_document(contents).get("services", {})
|
||||
return services if isinstance(services, dict) else {}
|
||||
|
||||
|
||||
def compose_changes(repo: str, target: str, base: str, head: str) -> set[str]:
|
||||
"""Return only compose services whose YAML definitions or dependencies changed.
|
||||
|
||||
A target with no compose file (Dankclaw since 2026-09-16) plans nothing:
|
||||
its services are Quadlet units, selected by their own paths.
|
||||
"""
|
||||
path = f"{target}/compose.yaml"
|
||||
if not _git_show(repo, head, path) and not _git_show(repo, base, path):
|
||||
return set()
|
||||
old_doc = _yaml_document(_git_show(repo, base, path))
|
||||
new_doc = _yaml_document(_git_show(repo, head, path))
|
||||
old = old_doc.get("services", {}) if isinstance(old_doc.get("services", {}), dict) else {}
|
||||
new = new_doc.get("services", {}) if isinstance(new_doc.get("services", {}), dict) else {}
|
||||
names = set(old) | set(new)
|
||||
allowed = _compose_services(target)
|
||||
|
||||
# An unavailable or malformed revision is safer as a broad compose plan
|
||||
# than as a silent no-op. It still cannot include LLM units.
|
||||
if (old_doc and not new_doc) or (new_doc and not old_doc and names):
|
||||
return {COMPOSE_NAME_MAP.get(n, n) for n in allowed}
|
||||
|
||||
# Top-level networks/configuration can affect every compose service even
|
||||
# when no individual service mapping changed. Ignore the services key while
|
||||
# comparing the rest of the Compose document.
|
||||
old_globals = {key: value for key, value in old_doc.items() if key != "services"}
|
||||
new_globals = {key: value for key, value in new_doc.items() if key != "services"}
|
||||
if old_globals != new_globals:
|
||||
return {COMPOSE_NAME_MAP.get(n, n) for n in allowed}
|
||||
|
||||
changed = {name for name in names if name in allowed and old.get(name) != new.get(name)}
|
||||
return {COMPOSE_NAME_MAP.get(n, n) for n in changed}
|
||||
|
||||
|
||||
def services_for_path(path: str, target: str) -> set[str]:
|
||||
"""Return services affected by one repository-relative path."""
|
||||
path = path.replace("\\", "/")
|
||||
p = PurePosixPath(path)
|
||||
parts = p.parts
|
||||
services: set[str] = set()
|
||||
if not parts:
|
||||
return services
|
||||
|
||||
# Non-runtime material is deliberately a no-op.
|
||||
if parts[0] in {"docs", "tests", "old", ".githooks"}:
|
||||
return services
|
||||
if path in {"README.md", "AGENTS.md"} or parts[0] == ".obsidian":
|
||||
return services
|
||||
|
||||
if parts[0] == "shared":
|
||||
if len(parts) < 2:
|
||||
return services
|
||||
group = parts[1]
|
||||
if group == "searxng":
|
||||
_add(services, "searxng")
|
||||
elif group == "prometheus":
|
||||
_add(services, "prometheus")
|
||||
if len(parts) >= 3 and parts[2] == "alertmanager.yml":
|
||||
# The Alertmanager config ships from this directory: a change
|
||||
# to it must redeploy the service that installs it.
|
||||
_add(services, "alertmanager")
|
||||
elif group == "loki":
|
||||
_add(services, "loki")
|
||||
elif group == "alloy":
|
||||
_add(services, "alloy")
|
||||
elif group == "grafana":
|
||||
# Provisioned datasources and dashboards: git is the source of
|
||||
# truth, so a change here must restart the service that reads them.
|
||||
_add(services, "grafana")
|
||||
elif group == "homepage":
|
||||
# Homepage is a Dankclaw-owned deployment. Shared homepage files
|
||||
# are intentionally not copied during a Dankserver-only run.
|
||||
if target == "dankclaw":
|
||||
_add(services, "homepage")
|
||||
return services
|
||||
|
||||
if parts[0] == "dankclaw":
|
||||
if len(parts) >= 2 and parts[1] == "compose.yaml":
|
||||
# Dankclaw has no compose file since 2026-09-16: every service is a
|
||||
# Quadlet unit, each selected by its own path. A change (or a
|
||||
# deletion) of this path therefore restarts nothing.
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "Caddyfile":
|
||||
_add(services, "caddy")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "Containerfile.comfyui":
|
||||
_add(services, "comfyui")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] in {"caddy", "comfyui", "loki", "alloy", "hermes", "anubis", "seerr"}:
|
||||
_add(services, parts[1])
|
||||
return services
|
||||
if len(parts) >= 4 and parts[1] == "authentik" and parts[2] == "quadlet":
|
||||
name = parts[3].removesuffix(".container")
|
||||
if name in {"authentik-postgres", "authentik-server", "authentik-worker", "authentik-outpost", "authentik-outpost-ldap"}:
|
||||
_add(services, name)
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "searxng":
|
||||
if len(parts) >= 4 and parts[2] == "quadlet":
|
||||
name = parts[3]
|
||||
if name == "gluetun.container":
|
||||
# SearXNG and Marreta (+ its browser companion) share
|
||||
# Gluetun's network namespace. A Gluetun recreation
|
||||
# tears the netns down, so every sharer must restart.
|
||||
_add(services, "gluetun", "searxng", "marreta")
|
||||
elif name == "searxng.container":
|
||||
_add(services, "searxng")
|
||||
elif name == "searxng-valkey.container":
|
||||
_add(services, "searxng-valkey", "searxng")
|
||||
elif name == "searxng.network":
|
||||
_add(services, "gluetun", "searxng-valkey", "searxng", "marreta")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "marreta":
|
||||
# One deployable owns both the app and its browser companion;
|
||||
# both live in gluetun's netns. A change to either unit selects
|
||||
# the marreta service, which restarts browser-then-app.
|
||||
_add(services, "marreta")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "pihole":
|
||||
_add(services, "pihole")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "open-webui":
|
||||
_add(services, "open-webui")
|
||||
return services
|
||||
# frank-dragoon left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host
|
||||
# build, root unit): its files live in scripts/ and map there.
|
||||
if len(parts) >= 2 and parts[1] == "observability":
|
||||
# The five observability containers became Quadlet units on
|
||||
# 2026-09-11. A unit change selects exactly its own service;
|
||||
# SERVICE_ORDER already puts each dependency ahead of its
|
||||
# consumer (prometheus before grafana, loki before alloy).
|
||||
if len(parts) >= 4 and parts[2] == "quadlet":
|
||||
name = parts[3].removesuffix(".container")
|
||||
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
|
||||
_add(services, name)
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "gitea":
|
||||
if len(parts) >= 4 and parts[2] == "quadlet":
|
||||
name = parts[3]
|
||||
if name == "gitea-postgres.container":
|
||||
_add(services, "gitea-postgres")
|
||||
elif name == "gitea-server.container":
|
||||
_add(services, "gitea-server")
|
||||
elif name == "gitea-caddy.container":
|
||||
# Removing the old split proxy requires the canonical
|
||||
# Caddy handler to retire its installed unit/container
|
||||
# before it claims ports 80/443.
|
||||
_add(services, "caddy")
|
||||
elif name == "gitea.network":
|
||||
_add(services, "gitea-postgres", "gitea-server")
|
||||
return services
|
||||
return services
|
||||
|
||||
if parts[0] == "dankserver":
|
||||
if len(parts) >= 2 and parts[1] == "compose.yaml":
|
||||
# Compose service keys map onto the deployed unit names.
|
||||
_add(services, *(COMPOSE_NAME_MAP.get(n, n) for n in DANKSERVER_COMPOSE_SERVICES))
|
||||
return services
|
||||
if len(parts) >= 3 and parts[2] == "quadlet":
|
||||
quadlet_services = {
|
||||
"romm": {"romm", "romm-mariadb", "romm-redis"},
|
||||
"jellyfin": {"jellyfin"},
|
||||
"calibre-web": {"calibre-web"},
|
||||
"calibre-server": {"calibre-server"},
|
||||
"copyparty": {"copyparty"},
|
||||
"plex": {"plex"},
|
||||
"tautulli": {"tautulli"},
|
||||
}
|
||||
_add(services, *quadlet_services.get(parts[1], set()))
|
||||
if len(parts) >= 2 and parts[1] == "observability":
|
||||
# Dankserver's node-exporter became a Quadlet unit on 2026-09-16
|
||||
# (serverAppConfigs-docs/Infrastructure/Observability.md): its
|
||||
# compose service died at a reboot and stayed dead, so Prometheus'
|
||||
# node-dankserver job had nothing to scrape.
|
||||
if len(parts) >= 4 and parts[2] == "quadlet":
|
||||
name = parts[3].removesuffix(".container")
|
||||
if name == "node-exporter":
|
||||
_add(services, "node-exporter")
|
||||
return services
|
||||
# frank-archon's quadlet-era directory left the tree on 2026-09-26
|
||||
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
|
||||
# the trio in scripts/ now maps it). Nothing lives under
|
||||
# dankserver/frank-archon/ except dated trash, and trash plans
|
||||
# nothing (the retire rule above the service blocks).
|
||||
return services
|
||||
|
||||
if parts[0] == "frankbook-pro":
|
||||
if len(parts) >= 3 and parts[2] == "quadlet":
|
||||
name = parts[3].removesuffix(".container")
|
||||
if name == "gluetun":
|
||||
# Every *arr + qBittorrent container shares gluetun's
|
||||
# network namespace; a gluetun recreation kills all of
|
||||
# them, so the plan restarts the whole netns family.
|
||||
_add(services, "gluetun", "qbittorrent",
|
||||
"prowlarr", "sonarr", "radarr", "lidarr", "readarr")
|
||||
elif name in FRANKBOOK_SERVICES:
|
||||
_add(services, name)
|
||||
return services
|
||||
|
||||
if parts[0] == "scripts":
|
||||
name = p.name
|
||||
if name in {"homepage-status.py", "homepage-status.service", "homepage-status.timer"}:
|
||||
_add(services, "homepage")
|
||||
if name in {"signal-relay.py", "signal-relay.service"}:
|
||||
# The Signal relay is installed by the alertmanager deploy, so a
|
||||
# change to it redeploying alertmanager is the only path that
|
||||
# reaches it on the host.
|
||||
_add(services, "alertmanager")
|
||||
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
|
||||
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-zealot only. The ik_llama.cpp
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
_add(services, "frank-zealot")
|
||||
if name in {"frank-dragoon.env", "frank-dragoon.service", "serve-frank-dragoon.sh"}:
|
||||
# CIRU v4.4.1 frank-dragoon (2026-09-19) keeps its unit, env and
|
||||
# serve files here the same way: a root unit on a host-built
|
||||
# runtime. The download script fetches the payload, which is
|
||||
# host-side state, so it selects nothing.
|
||||
_add(services, "frank-dragoon")
|
||||
if name == "qwen-sharp-v22.4.0.jinja":
|
||||
# The Qwen-Sharp chat template frank-dragoon loads via
|
||||
# --chat-template-file. It lived under dankserver/frank-medium/ until
|
||||
# the 2026-09-26 cutover retired that directory; editing it changes
|
||||
# what the live frank-dragoon endpoint renders, so it plans
|
||||
# frank-dragoon.
|
||||
_add(services, "frank-dragoon")
|
||||
if name in {"frank-archon.env", "frank-archon.service", "serve-frank-archon.sh"}:
|
||||
# llama.cpp frank-archon (2026-09-26, replacing the Lucebox
|
||||
# DS4.1 quadlet) keeps its unit, env and serve files here the
|
||||
# same way: a root unit on the unsloth b11030-mix prebuilt
|
||||
# serving Qwen3.8-Flash-Next Q8_0. The download script fetches
|
||||
# the payload, which is host-side state, so it selects nothing.
|
||||
_add(services, "frank-archon")
|
||||
# Deployment/test/benchmark scripts and other repo tooling do not
|
||||
# restart a runtime service.
|
||||
return services
|
||||
|
||||
# Examples and env templates are not deployed runtime state.
|
||||
if p.name in {".env.example", "env.example"}:
|
||||
return services
|
||||
return services
|
||||
|
||||
def changed_paths(repo: str, base: str, head: str) -> list[str]:
|
||||
result = subprocess.run(
|
||||
["git", "-C", repo, "diff", "--name-only", "--diff-filter=ACDMRTUXB", base, head],
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
check=True,
|
||||
)
|
||||
return [line for line in result.stdout.splitlines() if line]
|
||||
|
||||
|
||||
def plan(paths: list[str], target: str, repo: str | None = None,
|
||||
base: str | None = None, head: str | None = None) -> list[str]:
|
||||
services: set[str] = set()
|
||||
for path in paths:
|
||||
if path == f"{target}/compose.yaml" and repo and base and head:
|
||||
services.update(compose_changes(repo, target, base, head))
|
||||
else:
|
||||
services.update(services_for_path(path, target))
|
||||
allowed = target_services(target)
|
||||
return [name for name in SERVICE_ORDER if name in services and name in allowed]
|
||||
def git_paths(repo: str, base: str, head: str) -> list[str]:
|
||||
proc = subprocess.run(
|
||||
["git", "-C", repo, "diff", "--name-only", base, head],
|
||||
text=True, capture_output=True)
|
||||
if proc.returncode != 0:
|
||||
sys.stderr.write(f"deploy-plan: git diff failed: {proc.stderr}")
|
||||
raise SystemExit(1)
|
||||
return [line for line in proc.stdout.splitlines() if line]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--target", choices=("dankclaw", "dankserver", "frankbook-pro"), required=True)
|
||||
parser.add_argument("--paths", nargs="+")
|
||||
parser.add_argument("--repo", help="repository path for revision-aware planning")
|
||||
parser.add_argument("--base", help="base Git revision")
|
||||
parser.add_argument("--head", help="head Git revision")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.paths and not (args.repo and args.base and args.head):
|
||||
parser.error("provide --paths or all of --repo, --base, and --head")
|
||||
paths = args.paths or changed_paths(args.repo, args.base, args.head)
|
||||
print("\n".join(plan(paths, args.target, args.repo, args.base, args.head)))
|
||||
args = sys.argv[1:]
|
||||
if not args or args[0] != "--target":
|
||||
print(__doc__, file=sys.stderr)
|
||||
return 2
|
||||
if len(args) < 2:
|
||||
print("missing target name after --target", file=sys.stderr)
|
||||
return 2
|
||||
target = args[1]
|
||||
# `--paths` is the historic seam keyword: a flag, not an option — the
|
||||
# paths are the positional arguments that follow it.
|
||||
rest = [a for a in args[2:] if a != "--paths"]
|
||||
if "--repo" in rest:
|
||||
try:
|
||||
repo = rest[rest.index("--repo") + 1]
|
||||
base = rest[rest.index("--base") + 1]
|
||||
head = rest[rest.index("--head") + 1]
|
||||
except (ValueError, IndexError):
|
||||
print("--repo, --base and --head must appear together", file=sys.stderr)
|
||||
return 2
|
||||
paths = git_paths(repo, base, head)
|
||||
else:
|
||||
paths = rest or [line.strip() for line in sys.stdin if line.strip()]
|
||||
if not paths:
|
||||
return 0
|
||||
proc = subprocess.run([sys.executable, SERVICE_MAP, "plan", "--host", target] + paths,
|
||||
text=True, capture_output=True)
|
||||
if proc.returncode != 0:
|
||||
sys.stderr.write(proc.stderr)
|
||||
return proc.returncode
|
||||
sys.stdout.write(proc.stdout)
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
+167
-240
@@ -9,6 +9,13 @@
|
||||
# pending until its service is explicitly deployed.
|
||||
set -euo pipefail
|
||||
|
||||
# The LLM handlers' artifact gates are the serve scripts' own library, so
|
||||
# the deployer and the unit refuse the same missing payload with the same
|
||||
# words — no deploy can pass a gate the unit would fail. GATE_PREFIX is set
|
||||
# per handler.
|
||||
# shellcheck disable=SC1091
|
||||
source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/llm-gates.sh"
|
||||
|
||||
TARGET="${1:-}"
|
||||
REVISION="${2:-}"
|
||||
shift 2 || true
|
||||
@@ -80,31 +87,10 @@ cd "$REMOTE_DIR"
|
||||
mkdir -p "$STATE_DIR"
|
||||
|
||||
service_is_allowed() {
|
||||
local service="$1"
|
||||
case "$TARGET:$service" in
|
||||
dankclaw:gluetun|dankclaw:searxng-valkey|dankclaw:searxng|dankclaw:pihole|\
|
||||
dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
|
||||
dankclaw:comfyui|dankclaw:hermes|\
|
||||
dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
|
||||
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
|
||||
dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
|
||||
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
|
||||
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
||||
dankserver:frank-archon|dankserver:node-exporter|\
|
||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||
frankbook-pro:readarr)
|
||||
return 0
|
||||
;;
|
||||
*)
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
# The inventory (shared/services.yml via scripts/service-map.py) is the
|
||||
# only allow-list: a service is deployable on a target iff the inventory
|
||||
# places it there. Retiring a service from a host is a data edit.
|
||||
python3 scripts/service-map.py allows --host "$TARGET" --service "$1"
|
||||
}
|
||||
|
||||
mark_status() {
|
||||
@@ -274,13 +260,11 @@ install_signal_relay() {
|
||||
}
|
||||
|
||||
copy_loki_config() {
|
||||
run_privileged mkdir -p /opt/loki/config
|
||||
run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
|
||||
copy_config /opt/loki/config shared/loki/loki-config.yaml
|
||||
}
|
||||
|
||||
copy_alloy_config() {
|
||||
run_privileged mkdir -p /opt/alloy/config
|
||||
run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
|
||||
copy_config /opt/alloy/config shared/alloy/config.alloy
|
||||
}
|
||||
|
||||
copy_grafana_provisioning() {
|
||||
@@ -289,14 +273,11 @@ copy_grafana_provisioning() {
|
||||
# datasource would be invisible to review, lost with the data volume, and
|
||||
# unknown to anyone reading the repository. The unit already mounts
|
||||
# /opt/grafana/provisioning read-only, so this only has to populate it.
|
||||
run_privileged mkdir -p /opt/grafana/provisioning/datasources \
|
||||
/opt/grafana/provisioning/dashboards
|
||||
run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
|
||||
/opt/grafana/provisioning/datasources/prometheus.yml
|
||||
run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
|
||||
/opt/grafana/provisioning/dashboards/provider.yml
|
||||
run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
|
||||
/opt/grafana/provisioning/dashboards/llm-fleet.json
|
||||
copy_config /opt/grafana/provisioning/datasources \
|
||||
shared/grafana/provisioning/datasources/prometheus.yml
|
||||
copy_config /opt/grafana/provisioning/dashboards \
|
||||
shared/grafana/provisioning/dashboards/provider.yml \
|
||||
shared/grafana/provisioning/dashboards/llm-fleet.json
|
||||
}
|
||||
|
||||
render_caddyfile() {
|
||||
@@ -441,31 +422,6 @@ deploy_caddy() {
|
||||
fi
|
||||
}
|
||||
|
||||
deploy_compose_service() {
|
||||
local service="$1"
|
||||
if [[ "$TARGET" == dankserver ]]; then
|
||||
# Dankserver's compose file is the source-of-truth inventory, but its
|
||||
# production containers are individually systemd-managed/bare. Do not
|
||||
# use compose up here: it can create a second container with a wrong
|
||||
# name and can interfere with a running workload.
|
||||
echo "WARN: $service is recorded in dankserver/compose.yaml; production container management is host-specific and no restart was attempted."
|
||||
return 0
|
||||
fi
|
||||
if [[ "$TARGET" == dankclaw ]]; then
|
||||
# Dankclaw has no compose file since 2026-09-16: every service is a
|
||||
# Quadlet unit with its own handler above. Reaching this means the
|
||||
# dispatcher is missing a case, and the old fallback (`podman-compose
|
||||
# up -d`) would now be a silent no-op or a duplicate container.
|
||||
echo "no compose services on dankclaw; $service has no quadlet handler" >&2
|
||||
return 2
|
||||
fi
|
||||
case "$service" in
|
||||
*)
|
||||
echo "unknown compose service: $service" >&2
|
||||
return 2
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# grafana is the one observability service that holds a credential: the
|
||||
# Authentik OAuth2 client secret. It is seeded into a per-service env file (the
|
||||
@@ -568,6 +524,35 @@ quadlet_unit_dir="$HOME/.config/containers/systemd"
|
||||
# dankserver's user manager had no systemd container generator dir before the
|
||||
# quadlet migration; every handler that installs a unit must find it present.
|
||||
mkdir -p "$quadlet_unit_dir"
|
||||
# copy_config DEST SRC... — the privileged config copy every service uses:
|
||||
# ensure DEST, copy each tracked SRC into it. The deployer owns /opt state;
|
||||
# the files in git are the source of truth, so a service config that only
|
||||
# exists on the host is a gap by policy.
|
||||
copy_config() {
|
||||
local dest="$1"; shift
|
||||
run_privileged mkdir -p "$dest"
|
||||
local src
|
||||
for src in "$@"; do
|
||||
run_privileged cp "$src" "$dest/"
|
||||
done
|
||||
}
|
||||
|
||||
# retire_unit_file PATH TRASH [UNIT] — the repo Deletion & Trash policy, in
|
||||
# code: stop and disable the named unit, move the unit FILE into the home
|
||||
# trash (never rm), report. The dragoon quadlet retirement and the
|
||||
# compose-era comfyui.service retirement were the same six lines twice.
|
||||
retire_unit_file() {
|
||||
local path="$1" trash="$2" unit="${3:-}"
|
||||
[[ -e "$path" ]] || return 0
|
||||
if [[ -n "$unit" ]]; then
|
||||
systemctl --user stop "$unit" 2>/dev/null || true
|
||||
systemctl --user disable "$unit" 2>/dev/null || true
|
||||
fi
|
||||
mkdir -p "$trash"
|
||||
mv -- "$path" "$trash/$(basename "$path")"
|
||||
echo "retired $(basename "$path") -> $trash"
|
||||
}
|
||||
|
||||
install_quadlet() {
|
||||
local service="$1"
|
||||
local source_dir="$2"
|
||||
@@ -655,15 +640,8 @@ deploy_comfyui() {
|
||||
# generator produces, and it is what kept a stale, device-less container
|
||||
# alive through `restart: unless-stopped` (9,146 failures on
|
||||
# "No HIP GPUs are available"). Never `rm`: it moves into the home trash.
|
||||
local legacy="$HOME/.config/systemd/user/comfyui.service"
|
||||
if [[ -e "$legacy" ]]; then
|
||||
local trash="$HOME/.AGENTTRASH/20260916-comfyui-compose-unit"
|
||||
mkdir -p "$trash"
|
||||
systemctl --user stop comfyui.service 2>/dev/null || true
|
||||
systemctl --user disable comfyui.service 2>/dev/null || true
|
||||
mv -- "$legacy" "$trash/comfyui.service"
|
||||
echo "retired the compose-era comfyui.service -> $trash"
|
||||
fi
|
||||
retire_unit_file "$HOME/.config/systemd/user/comfyui.service" \
|
||||
"$HOME/.AGENTTRASH/20260916-comfyui-compose-unit" comfyui.service
|
||||
# The stale container carries the old config (no /dev/kfd, no /dev/dri) and
|
||||
# would keep the quadlet's container from taking the name. It is replaced
|
||||
# by the unit in the next step, so nothing is lost.
|
||||
@@ -721,9 +699,15 @@ deploy_gitea_unit() {
|
||||
systemctl --user restart "$service.service"
|
||||
}
|
||||
|
||||
# The root units are symlinks from the systemd system directory into the
|
||||
# checkout. SYSTEMD_SYSTEM_DIR is production (/etc/systemd/system); the knob
|
||||
# exists so the deploy seam can run a whole handler — relink included — in a
|
||||
# sandbox and read the unit links as evidence, unprivileged.
|
||||
SYSTEMD_SYSTEM_DIR="${SYSTEMD_SYSTEM_DIR:-/etc/systemd/system}"
|
||||
|
||||
relink_unit() {
|
||||
local service="$1"
|
||||
run_privileged ln -sfn "$REMOTE_DIR/scripts/$service.service" "/etc/systemd/system/$service.service"
|
||||
run_privileged ln -sfn "$REMOTE_DIR/scripts/$service.service" "$SYSTEMD_SYSTEM_DIR/$service.service"
|
||||
}
|
||||
|
||||
# Authentik runs as three boot-persistent Quadlet units (postgres -> server ->
|
||||
@@ -942,17 +926,17 @@ deploy_plex() {
|
||||
systemctl --user restart plex.service
|
||||
# Readiness gate: /identity answers MediaContainer XML once PMS is up.
|
||||
#
|
||||
# HTTPS, NOT HTTP. PMS drops plain HTTP off-loopback: the probe's
|
||||
# connection arrives on the container's eth0, so an http:// request is
|
||||
# accepted and then answered with nothing (curl 52, Empty reply) against a
|
||||
# perfectly healthy server. Measured 2026-10-02 — the gate burned its full
|
||||
# 600s and failed a plex whose https://127.0.0.1:32400/identity answered
|
||||
# 200 in 44 ms the whole time. The same law is why the Caddyfile dials the
|
||||
# plex upstream over TLS (tests/caddyfile.bats pins it).
|
||||
# HTTPS, NOT HTTP. PMS serves plain HTTP on loopback only: a probe
|
||||
# arriving on the container's interface gets its connection accepted and
|
||||
# then an empty reply (curl 52) from a perfectly healthy server, so an
|
||||
# http:// probe would burn the whole timeout and fail a service that
|
||||
# answers https://127.0.0.1:32400/identity in milliseconds. The Caddyfile
|
||||
# dials the same upstream over TLS for the same reason
|
||||
# (tests/caddyfile.bats pins it).
|
||||
# -k: PMS presents its own self-signed cert on 32400.
|
||||
#
|
||||
# Reality check from the 2026-09 cutover: the FIRST container boot of a
|
||||
# years-old library ran ~10 min, so the gate still allows 600s.
|
||||
# The budget accounts for a first boot of a large pre-existing library,
|
||||
# which spends minutes indexing before it answers anything.
|
||||
# Fail the deploy (pending state preserved) rather than mark it deployed blind.
|
||||
for ((i = 1; i <= 120; i++)); do
|
||||
curl -fsSk --max-time 3 https://127.0.0.1:32400/identity >/dev/null && return 0
|
||||
@@ -1125,13 +1109,8 @@ LLM_DRAGOON_TOK_DIR="${LLM_DRAGOON_TOK_DIR:-/home/dank/models/halogen-flashnext-
|
||||
LLM_DRAGOON_TOK_FILES="${LLM_DRAGOON_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
|
||||
|
||||
retire_halogen_dragoon_quadlet() {
|
||||
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
|
||||
systemctl --user stop frank-dragoon.service 2>/dev/null || true
|
||||
if [[ -e "$quadlet_unit_dir/frank-dragoon.container" ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$quadlet_unit_dir/frank-dragoon.container" "$trash/frank-dragoon.container"
|
||||
echo "retired quadlet frank-dragoon.container -> $trash"
|
||||
fi
|
||||
retire_unit_file "$quadlet_unit_dir/frank-dragoon.container" \
|
||||
"$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet" frank-dragoon.service
|
||||
systemctl --user daemon-reload
|
||||
}
|
||||
|
||||
@@ -1153,73 +1132,71 @@ wait_port_free() {
|
||||
ss -tan "sport = :$port" >&2 || true
|
||||
return 1
|
||||
}
|
||||
# assert_unit_owns_port SERVICE PORT — the false-green gate every LLM
|
||||
# endpoint shares. A health answer proves the PORT replied, never WHO
|
||||
# replied: on 2026-09-26 a leftover probe held :7770, satisfied the curl,
|
||||
# and the deploy reported green over a stopped unit. Green requires the
|
||||
# unit's own MainPID to be alive behind the answer.
|
||||
assert_unit_owns_port() {
|
||||
local service="$1" port="$2" main_pid
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value "$service.service")"
|
||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||
echo "FAIL: :$port answered but $service.service has no MainPID — a stray process holds the port" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "$service: health answered by unit MainPID $main_pid"
|
||||
}
|
||||
|
||||
# start_root_unit — the tail every LLM root unit deploys: relink the unit
|
||||
# file from the checkout into /etc/systemd/system, (re)enable, restart,
|
||||
# wait for /health, and prove the answer came from the unit's own process.
|
||||
# The three endpoints differ in gates, ports, and budgets, never in this.
|
||||
start_root_unit() {
|
||||
local service="$1" port="$2" health_limit="$3"
|
||||
relink_unit "$service"
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable "$service.service"
|
||||
run_privileged systemctl restart "$service.service"
|
||||
wait_for_health "$service" "http://127.0.0.1:${port}/health" "$health_limit" || return 1
|
||||
assert_unit_owns_port "$service" "$port"
|
||||
}
|
||||
|
||||
deploy_llm_dragoon() {
|
||||
# Refuse rather than restart-loop; every gate below has a twin in
|
||||
# serve-frank-dragoon.sh and they must agree (2026-10-01 lesson).
|
||||
# Capture podman output before grepping: `podman images | grep -q`
|
||||
# races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
|
||||
if ! grep -qF "${LLM_DRAGOON_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
|
||||
echo "ERROR: halogen image absent: $LLM_DRAGOON_IMAGE" >&2
|
||||
echo "frank-dragoon: pull it: podman pull ${LLM_DRAGOON_IMAGE%@*}" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$LLM_DRAGOON_CKPT" ]]; then
|
||||
echo "ERROR: halogen checkpoint absent at $LLM_DRAGOON_CKPT" >&2
|
||||
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$(stat -c %s "$LLM_DRAGOON_CKPT")" != "$LLM_DRAGOON_CKPT_BYTES" ]]; then
|
||||
echo "ERROR: halogen checkpoint is not $LLM_DRAGOON_CKPT_BYTES bytes (partial download?)" >&2
|
||||
echo "frank-dragoon: re-check: bash scripts/download-frank-dragoon-flashnext.sh --verify-only" >&2
|
||||
return 1
|
||||
fi
|
||||
for gate in "$LLM_DRAGOON_NGRAM:$LLM_DRAGOON_NGRAM_BYTES:n-gram table" \
|
||||
"$LLM_DRAGOON_VISION:$LLM_DRAGOON_VISION_BYTES:vision tower"; do
|
||||
IFS=: read -r gpath gbytes gname <<<"$gate"
|
||||
if [[ ! -f "$gpath" ]]; then
|
||||
echo "ERROR: halogen $gname absent at $gpath" >&2
|
||||
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
|
||||
echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
|
||||
echo "frank-dragoon: re-check: bash scripts/download-frank-dragoon-flashnext.sh --verify-only" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
if [[ -L "$LLM_DRAGOON_TOK_DIR" || ! -d "$LLM_DRAGOON_TOK_DIR" ]]; then
|
||||
echo "ERROR: tokenizer dir missing or a symlink at $LLM_DRAGOON_TOK_DIR" >&2
|
||||
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
for tf in $LLM_DRAGOON_TOK_FILES; do
|
||||
if [[ ! -f "$LLM_DRAGOON_TOK_DIR/$tf" || -L "$LLM_DRAGOON_TOK_DIR/$tf" ]]; then
|
||||
echo "ERROR: tokenizer file missing or a symlink: $LLM_DRAGOON_TOK_DIR/$tf" >&2
|
||||
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
# Refuse rather than restart-loop; every gate here is the SAME library
|
||||
# call the serve script makes, so the two twins cannot drift (the
|
||||
# 2026-10-01 crash-loop lesson).
|
||||
# Capture podman output before grepping is moot: require_image is one
|
||||
# command, no pipe (the 2026-09-30 SIGPIPE lesson).
|
||||
export GATE_PREFIX=frank-dragoon
|
||||
require_image "$LLM_DRAGOON_IMAGE" \
|
||||
"pull it: podman pull docker://${LLM_DRAGOON_IMAGE%@*}" || return 1
|
||||
require_bytes "$LLM_DRAGOON_CKPT" checkpoint "$LLM_DRAGOON_CKPT_BYTES" \
|
||||
"fetch it: bash scripts/download-frank-dragoon-flashnext.sh; re-check: --verify-only" || return 1
|
||||
require_bytes "$LLM_DRAGOON_NGRAM" n-gram "$LLM_DRAGOON_NGRAM_BYTES" \
|
||||
"fetch it: bash scripts/download-frank-dragoon-flashnext.sh; re-check: --verify-only" || return 1
|
||||
require_bytes "$LLM_DRAGOON_VISION" vision "$LLM_DRAGOON_VISION_BYTES" \
|
||||
"fetch it: bash scripts/download-frank-dragoon-flashnext.sh; re-check: --verify-only" || return 1
|
||||
# shellcheck disable=SC2086
|
||||
require_dir_files "$LLM_DRAGOON_TOK_DIR" tokenizer \
|
||||
"fetch it: bash scripts/download-frank-dragoon-flashnext.sh" \
|
||||
$LLM_DRAGOON_TOK_FILES || return 1
|
||||
|
||||
retire_halogen_dragoon_quadlet
|
||||
# The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
|
||||
# the container) before waiting on the port. The stray sweep is keyed
|
||||
# on the FULL container name, which appears in the argv of a manual
|
||||
# `podman run --name frank-dragoon-halogen` / `podman wait` but NOT in
|
||||
# this deployer's own command line (`bash deploy-remote.sh
|
||||
# frank-dragoon`) — so it cannot shoot itself in the foot (the fleet
|
||||
# pkill law), and it can never match frank-zealot's llama-server.
|
||||
# on the FULL container name: it appears in the argv of a manual
|
||||
# `podman run --name frank-dragoon-halogen` / `podman wait`, while a
|
||||
# pattern naming only the runtime would ALSO match this deployer's own
|
||||
# command line — a pkill whose pattern hits its own argv takes the
|
||||
# running deploy down with the target. The full name can never match
|
||||
# frank-zealot's llama-server either.
|
||||
run_privileged systemctl stop frank-dragoon.service 2>/dev/null || true
|
||||
pkill -f 'frank-dragoon-halogen' 2>/dev/null || true
|
||||
podman rm -f frank-dragoon frank-dragoon-halogen >/dev/null 2>&1 || true
|
||||
# Nothing may start until the predecessor has released the port
|
||||
# (2026-09-16: "Deactivated successfully" was not "port free").
|
||||
wait_port_free 7770 || return 1
|
||||
relink_unit frank-dragoon
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-dragoon.service
|
||||
run_privileged systemctl restart frank-dragoon.service
|
||||
wait_for_health frank-dragoon http://127.0.0.1:7770/health 1200
|
||||
start_root_unit frank-dragoon 7770 1200
|
||||
}
|
||||
|
||||
# deploy_llm_zealot — frank-zealot is llama.cpp (master build, CUDA sm_86) on
|
||||
@@ -1243,48 +1220,26 @@ LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K
|
||||
deploy_llm_zealot() {
|
||||
# Refuse rather than restart-loop: a unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the log to say why.
|
||||
if [[ ! -x "$LLM_ZEALOT_RUNTIME" ]]; then
|
||||
echo "ERROR: llama.cpp runtime absent at $LLM_ZEALOT_RUNTIME" >&2
|
||||
echo "frank-zealot: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
|
||||
echo "frank-zealot: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
|
||||
echo "frank-zealot: && cmake --build build-cuda --target llama-server) first." >&2
|
||||
return 1
|
||||
fi
|
||||
for f in "$LLM_ZEALOT_MODEL_FILE" "$LLM_ZEALOT_MMPROJ"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "ERROR: frank-zealot payload file absent at $f" >&2
|
||||
echo "frank-zealot: run scripts/download-frank-zealot.sh first;" >&2
|
||||
echo "frank-zealot: refusing to start a unit that cannot load." >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
relink_unit frank-zealot
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-zealot.service
|
||||
run_privileged systemctl restart frank-zealot.service
|
||||
wait_for_health frank-zealot http://127.0.0.1:7771/health 900 || return 1
|
||||
# False-green hardening (frank-dragoon's lesson): the port answering
|
||||
# proves nothing about WHO answers — the unit must be active with a
|
||||
# live MainPID, or a stray SGLang holdover is masquerading as deployed.
|
||||
local main_pid
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-zealot.service)"
|
||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||
echo "FAIL: :7771 answered but frank-zealot.service has no MainPID — a stray process holds the port" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "frank-zealot: health answered by unit MainPID $main_pid"
|
||||
# endpoint dark with nothing in the log to say why. Same library calls
|
||||
# as serve-frank-zealot.sh, so the deployer's twin refuses identically.
|
||||
export GATE_PREFIX=frank-zealot
|
||||
require_exec "$LLM_ZEALOT_RUNTIME" runtime \
|
||||
"build it: cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86" \
|
||||
|| return 1
|
||||
require_file "$LLM_ZEALOT_MODEL_FILE" gguf \
|
||||
"fetch it: bash scripts/download-frank-zealot.sh" || return 1
|
||||
require_file "$LLM_ZEALOT_MMPROJ" mmproj \
|
||||
"fetch it: bash scripts/download-frank-zealot.sh" || return 1
|
||||
|
||||
start_root_unit frank-zealot 7771 900
|
||||
}
|
||||
|
||||
# deploy_llm_archon — frank-archon is the radiance engine (StillDeadcode,
|
||||
# digest-pinned stilldeadcode/radiance image) serving the PUBLISHER's
|
||||
# 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
|
||||
# (qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2) as a
|
||||
# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
|
||||
# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
|
||||
# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
|
||||
# ROOT unit whose serve script drives `podman run` directly (the engine is
|
||||
# the image entrypoint; the image lives in dank's rootless podman storage).
|
||||
# 2026-10-15 payload swap from the in-house w4dn8 (payload on disk =
|
||||
# rollback).
|
||||
# 2026-10-05 cutover from FreeToken (pre-cutover tag
|
||||
# frank-archon-freetoken-pr587-20261005); before that, the tcclaviger vLLM
|
||||
# fork (tag frank-archon-vllm-shapeG-20261003). Same port and alias; the
|
||||
@@ -1293,7 +1248,7 @@ deploy_llm_zealot() {
|
||||
# Gates: the image must exist under its digest ref in dank's rootless
|
||||
# storage (FRANK_ARCHON_IMAGE), the .rad container file must exist at the
|
||||
# env's size floor (FRANK_ARCHON_MODEL_FILE >= FRANK_ARCHON_RAD_MIN_MIB —
|
||||
# a truncated fetch is a guaranteed long-timeout boot), and the unit env
|
||||
# a truncated convert is a guaranteed long-timeout boot), and the unit env
|
||||
# must be complete. Per-file sha256 verification is the fetch script's job
|
||||
# at download time and rad-convert/rad-info's job at build time, never
|
||||
# re-hashed here.
|
||||
@@ -1303,11 +1258,16 @@ LLM_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-pr
|
||||
deploy_llm_archon() {
|
||||
# Refuse rather than restart-loop: a unit whose image or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the log to say why.
|
||||
local env_file="/home/dank/serverAppConfigs/scripts/frank-archon.env"
|
||||
local image model_dir model_file rad_min mib
|
||||
# endpoint dark with nothing in the log to say why. The gates are the
|
||||
# shared library's, called with the values the unit's own env file
|
||||
# declares — so serve-frank-archon.sh and this handler cannot disagree,
|
||||
# and the payload path lives in exactly one file.
|
||||
export GATE_PREFIX=frank-archon
|
||||
local env_file="${LLM_ARCHON_ENV_FILE:-$REMOTE_DIR/scripts/frank-archon.env}"
|
||||
local image model_dir model_file rad_min
|
||||
if [[ ! -f "$env_file" ]]; then
|
||||
echo "ERROR: frank-archon env file absent: $env_file" >&2
|
||||
echo "frank-archon: env: missing at $env_file" >&2
|
||||
echo " hint: deploy the repo checkout first ($REMOTE_DIR)" >&2
|
||||
return 1
|
||||
fi
|
||||
image="$(sed -n 's/^FRANK_ARCHON_IMAGE=//p' "$env_file" | tail -1)"
|
||||
@@ -1315,28 +1275,16 @@ deploy_llm_archon() {
|
||||
model_file="$(sed -n 's/^FRANK_ARCHON_MODEL_FILE=//p' "$env_file" | tail -1)"
|
||||
rad_min="$(sed -n 's/^FRANK_ARCHON_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
|
||||
if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
|
||||
echo "ERROR: frank-archon.env missing FRANK_ARCHON_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
|
||||
return 1
|
||||
fi
|
||||
# The image gate is the digest ref itself: a name@sha256 ref exists in
|
||||
# storage only if THAT manifest was pulled. One command — no
|
||||
# pipefail/grep race (the 2026-09-30 lesson).
|
||||
if ! podman image exists "$image"; then
|
||||
echo "ERROR: runtime image absent from storage: $image" >&2
|
||||
echo "frank-archon: podman pull docker://stilldeadcode/radiance:latest" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$model_dir/$model_file" ]]; then
|
||||
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
|
||||
echo "frank-archon: run scripts/download-frank-archon-iq4r.sh --fetch" >&2
|
||||
return 1
|
||||
fi
|
||||
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
|
||||
if [[ "$mib" -lt "$rad_min" ]]; then
|
||||
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
|
||||
echo "frank-archon: re-fetch it (scripts/download-frank-archon-iq4r.sh --fetch; delete the staged file first if its sha256 failed)" >&2
|
||||
echo "frank-archon: env: missing FRANK_ARCHON_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
|
||||
echo " hint: scripts/frank-archon.env" >&2
|
||||
return 1
|
||||
fi
|
||||
require_image "$image" \
|
||||
"pull it: podman pull docker://stilldeadcode/radiance:latest (as dank)" \
|
||||
|| return 1
|
||||
require_min_mib "$model_dir/$model_file" ".rad" "$rad_min" \
|
||||
"convert it: scripts/convert-swift15-w4dn8.sh (rad-info is the record)" \
|
||||
|| return 1
|
||||
|
||||
# A stale container with the same name makes `podman run --name` fail at
|
||||
# start; remove any leftover from a failed previous attempt.
|
||||
@@ -1355,41 +1303,15 @@ deploy_llm_archon() {
|
||||
# (2026-09-26) and the frank-dragoon cutover (2026-09-16) both failed
|
||||
# exactly here when the next engine bound 7770 a second too early.
|
||||
wait_port_free 7770 || return 1
|
||||
relink_unit frank-archon
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-archon.service
|
||||
run_privileged systemctl restart frank-archon.service
|
||||
# Cold start: .rad load + PLE pin + expert host-pool upload (the
|
||||
# publisher's compose healthcheck allows a 15-minute start_period).
|
||||
# Parity with every predecessor on this port. The HTTP listener binds
|
||||
# AFTER the full load (main.cpp: engine.start() then engine.serve()),
|
||||
# so during load the port REFUSES connections — a bare curl is honest
|
||||
# here, unlike FreeToken's 200-while-loading body. (The MainPID check
|
||||
# below still guards the stray-port case.)
|
||||
local i ok=0
|
||||
for ((i=1; i<=900; i++)); do
|
||||
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
|
||||
echo "frank-archon engine healthy after approximately $((i * 2))s"
|
||||
ok=1
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
if [[ "$ok" != 1 ]]; then
|
||||
echo "FAIL: frank-archon /health never answered 200 in 1800s" >&2
|
||||
return 1
|
||||
fi
|
||||
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
|
||||
# port cannot tell WHO answers — a stray probe holding :7770 satisfies
|
||||
# it while the unit is stopped. The deploy is only green if the unit
|
||||
# itself is active with a live MainPID behind the health answer.
|
||||
local main_pid
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-archon.service)"
|
||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||
echo "FAIL: :7770 answered but frank-archon.service has no MainPID — a stray process holds the port" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "frank-archon: health answered by unit MainPID $main_pid"
|
||||
# publisher's compose healthcheck allows a 15-minute start_period) —
|
||||
# the 900-iteration budget is 1800s, parity with every predecessor on
|
||||
# this port. The HTTP listener binds AFTER the full load (main.cpp:
|
||||
# engine.start() then engine.serve()), so during load the port REFUSES
|
||||
# connections — a bare curl is honest here, unlike FreeToken's
|
||||
# 200-while-loading body. (start_root_unit's MainPID check still
|
||||
# guards the stray-port case.)
|
||||
start_root_unit frank-archon 7770 900
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
@@ -1589,7 +1511,12 @@ deploy_service() {
|
||||
tautulli) deploy_tautulli ;;
|
||||
prometheus|grafana|loki|alloy|node-exporter|alertmanager) deploy_observability "$service" ;;
|
||||
comfyui) deploy_comfyui ;;
|
||||
*) deploy_compose_service "$service" ;;
|
||||
*)
|
||||
# Every deployable service on every target has a handler above
|
||||
# (inventory = allow-list, so nothing unreachable arrives).
|
||||
echo "no handler for service: $service" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
|
||||
+7
-25
@@ -131,31 +131,13 @@ else
|
||||
fi
|
||||
|
||||
valid_service() {
|
||||
local service="$1"
|
||||
case "$TARGET:$service" in
|
||||
dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
|
||||
dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
|
||||
dankclaw:gitea-server|dankclaw:homepage|\
|
||||
dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
|
||||
dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
|
||||
dankclaw:alertmanager|\
|
||||
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
|
||||
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
|
||||
dankserver:frank-archon|\
|
||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||
frankbook-pro:readarr)
|
||||
return 0
|
||||
;;
|
||||
*)
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
# The inventory is the only allow-list (see scripts/service-map.py);
|
||||
# service-map's "allows" exit code is the answer. The former literal
|
||||
# case list here drifted from reality (it still listed the *arr stack
|
||||
# under dankserver, which moved to frankbook-pro, and a
|
||||
# "searxng-network" name that is a generated systemd unit, not a
|
||||
# deployable service).
|
||||
python3 "$REPO_ROOT/scripts/service-map.py" allows --host "$TARGET" --service "$1" >/dev/null
|
||||
}
|
||||
|
||||
for service in "${EXPLICIT_SERVICES[@]}"; do
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"""derive-protected-experts.py — rewrite the recipe's protected-expert section
|
||||
from a calibration recording. RUNS ON DANKSERVER (venv: numpy).
|
||||
|
||||
The law (docs/MOE-W4.md "Protected experts", restated in the recipe):
|
||||
The selection rule (docs/MOE-W4.md "Protected experts", restated in the recipe):
|
||||
an expert whose share of its layer's down-input energy is at least 3% AND
|
||||
at least three times its share of the routing is kept bf16; the count must
|
||||
be EVEN on each rank's half of the layer (the four-bit GEMM takes the rest
|
||||
|
||||
@@ -1,83 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-archon-iq4r.sh — payload fetch/verify for the frank-archon
|
||||
# swap to the PUBLISHER's 4-bit-codebook container:
|
||||
# StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe — the official
|
||||
# Qwen/Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's own
|
||||
# author into ONE .rad: routed experts as 4-bit codes into a 16-level
|
||||
# non-uniform codebook (Walsh–Hadamard rotation, GPTQ + coordinate-descent,
|
||||
# 10M-token calibration), ten bf16-protected experts per layer, int8 trunk +
|
||||
# lm_head, fp8 HC mixtures, the PLE n-gram table at fp8 fixed scale, the MTP
|
||||
# head, and the bf16 vision tower. File:
|
||||
# qwen3.8-next-flash-fp8-iq4r-moe.rad — 121,969,901,568 B (113.6 GiB,
|
||||
# 116,322 MiB), card KL 0.0789 all-tokens / top-1 92.85 % vs bf16.
|
||||
# Repo is UNGATED — no token needed (never put one on the command line).
|
||||
#
|
||||
# bash scripts/download-frank-archon-iq4r.sh --fetch resume/complete
|
||||
# bash scripts/download-frank-archon-iq4r.sh verify only
|
||||
#
|
||||
# Fetch uses the vendored scripts/hfget.sh (curl+jq only): per-file size
|
||||
# checks against the Hub listing, `-C -` resume, disk-fit gate. The side
|
||||
# files (LICENSE, README, the w4nl recipe) stay in the staging dir as
|
||||
# provenance; the serve gate mounts ONLY /home/dank/models/rad, so the
|
||||
# verified .rad alone moves into it.
|
||||
#
|
||||
# Verify (default) is the AGENTS.md law in two parts: the file lands at the
|
||||
# final path with the Hub's EXACT byte count, and its sha256 equals the
|
||||
# Hub's LFS oid. A truncated transfer is a guaranteed long-timeout boot
|
||||
# without this gate.
|
||||
set -euo pipefail
|
||||
|
||||
REPO="StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe"
|
||||
RAD_NAME="qwen3.8-next-flash-fp8-iq4r-moe.rad"
|
||||
STAGE="/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe"
|
||||
DEST="/home/dank/models/rad/$RAD_NAME"
|
||||
SIZE=121969901568
|
||||
SHA256="0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
mode="verify"
|
||||
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
||||
|
||||
if [[ "$mode" == "fetch" ]]; then
|
||||
# HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
|
||||
bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$STAGE" main
|
||||
|
||||
staged="$STAGE/$RAD_NAME"
|
||||
if [[ ! -f "$staged" ]]; then
|
||||
echo "FETCH FAILED: $staged absent after hfget" >&2
|
||||
exit 1
|
||||
fi
|
||||
got=$(stat -c %s "$staged")
|
||||
if [[ "$got" != "$SIZE" ]]; then
|
||||
echo "FETCH FAILED: staged .rad is $got B, want $SIZE B" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "staged .rad byte-exact ($SIZE B); hashing (one full read, minutes)"
|
||||
got_sha=$(sha256sum "$staged" | cut -d' ' -f1)
|
||||
if [[ "$got_sha" != "$SHA256" ]]; then
|
||||
echo "FETCH FAILED: sha256 $got_sha, want $SHA256 — DELETE the staged" \
|
||||
"file before re-fetching (curl -C - cannot repair a corrupt body)" >&2
|
||||
exit 1
|
||||
fi
|
||||
mkdir -p "$(dirname "$DEST")"
|
||||
mv "$staged" "$DEST"
|
||||
echo "payload placed: $DEST"
|
||||
fi
|
||||
|
||||
echo "verifying deployed payload: $DEST"
|
||||
if [[ ! -f "$DEST" ]]; then
|
||||
echo " BAD payload absent: $DEST" >&2
|
||||
exit 1
|
||||
fi
|
||||
got=$(stat -c %s "$DEST")
|
||||
if [[ "$got" != "$SIZE" ]]; then
|
||||
echo " BAD payload is $got B, want $SIZE B" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "byte-exact ($SIZE B); hashing (one full read, minutes)"
|
||||
got_sha=$(sha256sum "$DEST" | cut -d' ' -f1)
|
||||
if [[ "$got_sha" != "$SHA256" ]]; then
|
||||
echo " BAD sha256 $got_sha, want $SHA256" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "payload verified: $RAD_NAME, size and sha256 match the Hub LFS oid"
|
||||
@@ -14,10 +14,11 @@
|
||||
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
||||
# listing/resume downloader, curl+jq only): per-file size checks against
|
||||
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
|
||||
# Verify (default) is the AGENTS.md law in three parts: every listed file
|
||||
# present with the listed size, the shard count matches the weight-map
|
||||
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
|
||||
# only the public API.
|
||||
# Verify (default) is three independent agreements: every listed file
|
||||
# present with the listed size (catches truncation), the shard count
|
||||
# matching the weight-map index (catches omissions), and every shard's
|
||||
# sha256 equal to the Hub's LFS oid (catches substituted content).
|
||||
# Verify needs only the public API.
|
||||
set -euo pipefail
|
||||
|
||||
REPO="ukisai/Swift1.5-Qwen3.8-Flash-Next"
|
||||
|
||||
@@ -12,10 +12,11 @@
|
||||
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
||||
# listing/resume downloader, curl+jq only): per-file size checks against
|
||||
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
|
||||
# Verify (default) is the AGENTS.md law in three parts: every listed file
|
||||
# present with the listed size, the shard count matches the weight-map
|
||||
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
|
||||
# only the public API.
|
||||
# Verify (default) is three independent agreements: every listed file
|
||||
# present with the listed size (catches truncation), the shard count
|
||||
# matching the weight-map index (catches omissions), and every shard's
|
||||
# sha256 equal to the Hub's LFS oid (catches substituted content).
|
||||
# Verify needs only the public API.
|
||||
set -euo pipefail
|
||||
|
||||
REPO="Qwen/Qwen3.8-Flash-Next-FP8"
|
||||
|
||||
@@ -4,12 +4,15 @@
|
||||
# .hgn, the REQUIRED lookup table and the optional vision tower from
|
||||
# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
|
||||
# card pins), and the flat tokenizer the two repos share. All public, no
|
||||
# token — and per the fleet secret law, none would ever touch argv.
|
||||
# token needed — and a secret would never be passed via argv anyway: the
|
||||
# command line is world-readable through /proc, so any token placed there
|
||||
# leaks to every process-listing tool on the host.
|
||||
#
|
||||
# The fetch law, same as every sibling script: curl -f (an access error
|
||||
# must not become a "successful" 153-byte artifact), exact byte count from
|
||||
# the Hub tree API, LFS sha256 verification, --verify-only to re-check an
|
||||
# existing payload without downloading.
|
||||
# The fetch rules, uniform across every sibling script: curl -f so an
|
||||
# access error fails loudly instead of writing the error body as a
|
||||
# "successful" tiny artifact; the exact byte count from the Hub tree API;
|
||||
# LFS sha256 verification; --verify-only to re-check an existing payload
|
||||
# without downloading.
|
||||
#
|
||||
# File manifest (verified against the Hub tree API on 2026-10-06):
|
||||
# qwen38-flash-next-v2-swift15-abliterated.hgn 66,687,678,464 B sha256
|
||||
@@ -26,7 +29,9 @@
|
||||
# tokenizer/* @e053f488 — byte-identical to the 27B-era tokenizer
|
||||
# already on disk (tokenizer.json LFS
|
||||
# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3);
|
||||
# fetched fresh rather than symlinked, per the no-symlink law.
|
||||
# fetched fresh, never symlinked: a snapshot link points into the
|
||||
# host-side cache directory, which does not exist inside the
|
||||
# container mount, so the link dangles exactly where it is consumed.
|
||||
set -euo pipefail
|
||||
|
||||
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-dragoon-halogen.sh — fetch frank-dragoon's halogen payload:
|
||||
# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
|
||||
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
|
||||
# secret law, none would ever touch argv).
|
||||
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and a token
|
||||
# would never travel in argv: command lines are world-readable through
|
||||
# /proc, so a secret placed there leaks to any process-listing tool).
|
||||
#
|
||||
# The fetch law, same as every sibling script: curl -f (an access error
|
||||
# must not become a "successful" 153-byte artifact), exact byte count from
|
||||
# the model card, LFS sha256 verification, --verify-only to re-check an
|
||||
# existing payload without downloading.
|
||||
# The fetch rules, uniform across every sibling script: curl -f so an
|
||||
# access error fails loudly instead of writing the error body as a
|
||||
# "successful" tiny artifact; the exact byte count from the model card;
|
||||
# LFS sha256 verification; --verify-only to re-check an existing payload
|
||||
# without downloading.
|
||||
set -euo pipefail
|
||||
|
||||
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
|
||||
|
||||
@@ -11,13 +11,13 @@ Tensor extents are DERIVED from consecutive GGUF header offsets — no quant
|
||||
type-size table (those are easy to get subtly wrong; a wrong per-element size
|
||||
silently under-covers each tensor and dumps tail samples into a junk bucket).
|
||||
|
||||
Fleet law this instrument established (2026-09-26, 75 s window, 4 mixed
|
||||
streams, 2.7M expert-tensor samples): per-BLOCK read volume is FLAT
|
||||
(0.93x-1.08x of mean across all 40 CPU-resident blocks) because MoE top-k
|
||||
routing fires every block every token — there are no "most active layers" at
|
||||
`-ot` granularity. The real skew is per-EXPERT (top 10% of (block,expert)
|
||||
cells carry ~40% of samples), which tensor-level overrides cannot reach.
|
||||
Any 8 GPU-resident blocks are therefore equivalent; COUNT is the only lever.
|
||||
What this instrument settles: per-BLOCK expert read volume is FLAT across
|
||||
all CPU-resident blocks, because MoE top-k routing fires every block on
|
||||
every token — there are no "most active layers" at `-ot` granularity. The
|
||||
real skew is per-EXPERT: a small fraction of (block,expert) cells carries
|
||||
most of the samples, and tensor-level overrides cannot reach that level.
|
||||
Any fixed number of GPU-resident blocks is therefore equivalent; COUNT is
|
||||
the only lever.
|
||||
|
||||
Usage (needs passwordless sudo for perf only; model files must be readable):
|
||||
fp-expert-activity.py <pid> [duration_s] # dump + sample + analyze
|
||||
|
||||
+87
-163
@@ -1,31 +1,12 @@
|
||||
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving the
|
||||
# PUBLISHER's 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
|
||||
# (MODEL_FILE below, fetched by scripts/download-frank-archon-iq4r.sh) on
|
||||
# BOTH R9700s (TP=2), dankserver :7770.
|
||||
# 2026-10-15 PAYLOAD SWAP: the model moved from the in-house w4dn8 conversion
|
||||
# of the Swift1.5 finetune to the engine author's published container
|
||||
# qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, sha256-verified at fetch —
|
||||
# the w4dn8 measured 116,416 MiB, so this shape carries over with 94 MiB to
|
||||
# spare). NO argument changed: every number below is the 2026-10-13 shape,
|
||||
# kept verbatim. The swap commit is the rollback (the w4dn8 payload stays on
|
||||
# disk).
|
||||
|
||||
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving
|
||||
# ukisai/Swift1.5-Qwen3.8-Flash-Next converted to GPTQ w4 (MODEL_FILE below,
|
||||
# built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2),
|
||||
# dankserver :7770.
|
||||
#
|
||||
# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
|
||||
# with the per-card budget spent on resident experts: 25,600 MiB of slab +
|
||||
# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own
|
||||
# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized
|
||||
# to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120
|
||||
# predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab
|
||||
# is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT
|
||||
# +25 % at the full window) while KV beyond the contract buys nothing.
|
||||
# Consequence, deliberate: a FOURTH full-height conversation does not fit this
|
||||
# pair, and the engine refuses it — it never truncates. See the KV block for the
|
||||
# arithmetic and the measured dtype law. The w4dn8 payload is the swap
|
||||
# rollback and keeps its trio under the w4dn8 runbook; the fp8-native payload
|
||||
# that came before it was DELETED from disk 2026-10-15 (Frank's call, to free
|
||||
# the volume for this download — rebuildable via the fetch + build scripts
|
||||
# named in the fp8-native runbook).
|
||||
# SHAPE (2026-10-11): 2 decode slots x 200,000 tokens with an UNQUANTIZED
|
||||
# (bf16) KV cache — the pool is sized for the pair, and the weights slab pays
|
||||
# for it (see the KV block). The fp8-native payload this file was written for
|
||||
# is the rollback target and keeps its own trio under the fp8-native runbook.
|
||||
#
|
||||
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
||||
# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
|
||||
@@ -40,30 +21,20 @@
|
||||
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
||||
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
||||
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
||||
# THE 4-bit-codebook publisher shape is the DEPLOYED payload since
|
||||
# 2026-10-15. The card's own run line (--host-pool-mib 12288,
|
||||
# --expert-vs-cache-ratio 0.82, seqs 8, spec 3) is the publisher's
|
||||
# suggestion on 32 GB cards; this file's shape is the fleet's measured one.
|
||||
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
|
||||
# --host-pool-mib 12288) remains the documented fallback if the native
|
||||
# container proves unbootable.
|
||||
#
|
||||
# PAYLOAD: /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad — the
|
||||
# official Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's
|
||||
# own author (recipe qwen4exp-w4nl64-i8-hc8m: experts as 4-bit codes into a
|
||||
# 16-level non-uniform codebook after a 128-point Walsh–Hadamard rotation,
|
||||
# GPTQ + 3 coordinate-descent sweeps on a 10M-token calibration, ten
|
||||
# bf16-protected experts per layer, int8 W8A8 trunk + lm_head, fp8 HC
|
||||
# mixtures, PLE n-gram fp8 fixed-scale, MTP head, bf16 vision tower; card KL
|
||||
# 0.0789 all-tokens / top-1 92.85 % vs bf16). Fetched byte-exact + sha256 by
|
||||
# scripts/download-frank-archon-iq4r.sh; the recipe/LICENSE stay in the
|
||||
# staging dir beside it as provenance. ROLLBACK: revert the swap commit — the
|
||||
# in-house qwen4exp-swift15-w4dn8-i8-hc8m.rad (scripts/convert-swift15-w4dn8.sh,
|
||||
# KLD-gated) stays on disk. Behind it, the fp8-native
|
||||
# qwen4exp-fp8-native.rad was DELETED from disk 2026-10-15 (Frank's call,
|
||||
# one-time rm, to free the volume for this payload); rebuild =
|
||||
# scripts/download-frank-archon.sh + scripts/build-frank-archon-rad.sh — a
|
||||
# NO-RECIPE rad-convert copy of the official 131-shard checkpoint under the
|
||||
# descriptor law (no shipped plugin claims (qwen4exp, "fp8"), so the staging
|
||||
# config.json drops quantization_config; per-weight encodings are data, not
|
||||
# the descriptor).
|
||||
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
|
||||
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
|
||||
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
|
||||
# staging dir whose config.json drops `quantization_config`: the descriptor
|
||||
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
|
||||
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
|
||||
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
|
||||
# containers carry. Per-weight encodings are data, not the descriptor —
|
||||
# removing the key changes nothing about what the weights hold. Built by
|
||||
# scripts/build-frank-archon-rad.sh (staging + convert + rad-info).
|
||||
#
|
||||
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
||||
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
||||
@@ -79,12 +50,12 @@
|
||||
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
||||
# models.yml's extraBody set is fully inside it. Unknown params are
|
||||
# refused (400), which is why nothing unverified rides the wire.
|
||||
# - MTP: the .rad carries the model's mtp.* head (the publisher's recipe
|
||||
# names the mtp.fc_hidden/fc_embedding/draft_head rules; card default
|
||||
# depth 3). Served at the depth named by FRANK_ARCHON_SPEC_TOKENS below —
|
||||
# the depth this shape's decode ladder was measured at.
|
||||
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
|
||||
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
|
||||
# publisher's throughput peak for this model. Stated explicitly with
|
||||
# --num-speculative-tokens 3.
|
||||
# - The fused TP all-reduce is declared over max-num-seqs x
|
||||
# (1 + --num-speculative-tokens) rows; 3 x (1+2) = 9 rows, under the 25-row
|
||||
# (1 + --num-speculative-tokens) rows; 2 x 3 = 6 rows, under the 25-row
|
||||
# exactness bound of the wht6 wire.
|
||||
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
|
||||
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
|
||||
@@ -108,7 +79,7 @@ FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd7
|
||||
# .rad files; the raw HF download and its staging dir live beside it and
|
||||
# are never mounted.
|
||||
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
|
||||
FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad
|
||||
FRANK_ARCHON_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||
FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state
|
||||
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
|
||||
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
|
||||
@@ -142,29 +113,19 @@ FRANK_ARCHON_TP=2
|
||||
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
|
||||
# engine default.
|
||||
FRANK_ARCHON_TP_WIRE=wht6
|
||||
# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
|
||||
# Decode slots: TWO, so the served window is a CAPACITY CONTRACT — two
|
||||
# concurrent conversations of --max-model-len fit the KV pool (see the KV
|
||||
# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
|
||||
# tcclaviger vLLM fork's torch.compile shape specialization on
|
||||
# query_start_loc (ConstraintViolationError, measured 2026-10-02, retired
|
||||
# with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
|
||||
# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
|
||||
# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency,
|
||||
# not speed, so a slot costs KV, never decode.
|
||||
# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
|
||||
# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
|
||||
# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
|
||||
# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
|
||||
# steps are lossy).
|
||||
# The pool is sized to this contract and nothing more (see the KV block):
|
||||
# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
|
||||
# the resident expert slab. The trade is explicit and it is Frank's call:
|
||||
# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
|
||||
# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
|
||||
# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
|
||||
# traffic that wants four full-height sessions is a --vram-kv-mib change, not
|
||||
# a code change.
|
||||
FRANK_ARCHON_MAX_SEQS=3
|
||||
# block below). Slot-count limits observed on other runtimes do NOT
|
||||
# transfer: a "this slot count cannot boot" refusal is usually that
|
||||
# runtime's compiler specializing shapes (e.g. a ConstraintViolationError
|
||||
# from shape specialization on query_start_loc), not a property of the
|
||||
# engine now running. This engine boots seqs 2, and seqs 2
|
||||
# vs 3 decode measurements land inside each other's noise floor. Side
|
||||
# benefit: the fused TP all-reduce is declared over seqs x (1 + spec) = 6
|
||||
# rows, under the 25-row bound at which the wht6 wire stays EXACT
|
||||
# (single-sequence decode and any step of <= 25 rows is bit-exact; only
|
||||
# busier steps are lossy).
|
||||
FRANK_ARCHON_MAX_SEQS=2
|
||||
# Served window. Native is 262,144; 200,000 is the publisher's production
|
||||
# value (compose) and what models.yml's contextWindow must quote — a longer
|
||||
# prompt is refused by the engine, never truncated. The engine treats
|
||||
@@ -185,52 +146,25 @@ FRANK_ARCHON_HOST_POOL_MIB=44928
|
||||
# VRAM left unclaimed per card (publisher default 96 MiB).
|
||||
FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
||||
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
|
||||
# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13,
|
||||
# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is
|
||||
# neither budgeted nor accounted — the driver's context, the activation arena
|
||||
# and --gpu-headroom-mib come out of it; anything past those is throughput on
|
||||
# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the
|
||||
# 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`.
|
||||
# So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the
|
||||
# ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120
|
||||
# ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not
|
||||
# the engine's; the pair below spends the difference and lands 458 MiB above
|
||||
# the engine's floor. `weights + kv <= 30,969` is the gate.
|
||||
# THE CEILING, measured on this host 2026-10-11: card 31,856 MiB, a boot
|
||||
# consumes 1,710 MiB OUTSIDE both budgets (driver context + activation arena
|
||||
# 990 + hardware queues 100 + headroom 96 + step staging), and the fp8 pair
|
||||
# settled at 1,379 MiB free — so `vram_weights + vram_kv <= 29,120 MiB` keeps
|
||||
# >= 1 GiB free. The pair below is exactly that sum.
|
||||
# THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the
|
||||
# expert cache's spillover (`mover: 2275 slots can be lent out of the KV
|
||||
# cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`),
|
||||
# so a bigger KV pool steals weight residency from the slab AND from the
|
||||
# lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
|
||||
# a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
|
||||
# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
|
||||
# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
|
||||
# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
|
||||
# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
|
||||
# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120)
|
||||
# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's
|
||||
# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction
|
||||
# the A/B data points: prefill is slab-limited, KV beyond the contract is not.
|
||||
# The margin is thin by design (608,027 predicted tokens against the 600,000
|
||||
# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
|
||||
# once` line is the referee for THIS pair and the concurrent probe is its
|
||||
# regression test.
|
||||
# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
|
||||
# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
|
||||
# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
|
||||
# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
|
||||
# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
|
||||
# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
|
||||
# pair landed inside noise of EACH OTHER, which is what made it read as a
|
||||
# plateau; two points are not a curve.
|
||||
# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
|
||||
# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
|
||||
# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
|
||||
# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
|
||||
# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
|
||||
FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
|
||||
FRANK_ARCHON_VRAM_KV_MIB=5120
|
||||
# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
|
||||
# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
|
||||
# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. Expect -3..-8%
|
||||
# vs the fp8 control (136 t/s median at 44.5k depth, acceptance 0.7246,
|
||||
# ~190 t/s on a live 512-token probe); re-tune from a boot ledger + probe,
|
||||
# never from that estimate.
|
||||
FRANK_ARCHON_VRAM_WEIGHTS_MIB=22208
|
||||
FRANK_ARCHON_VRAM_KV_MIB=6912
|
||||
# KV cache UNQUANTIZED (bf16). This engine's flag takes ONLY bf16|fp8 — there
|
||||
# is ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
|
||||
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
|
||||
# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
|
||||
# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
|
||||
@@ -244,63 +178,53 @@ FRANK_ARCHON_VRAM_KV_MIB=5120
|
||||
# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
|
||||
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
|
||||
# checkpoint region 8 x 55.99 = 448 MiB. So:
|
||||
# pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
|
||||
# 3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
|
||||
# and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
|
||||
# three-slot contract with 1.3 % slack, every MiB above 5,063 having been
|
||||
# handed to the expert slab instead. The ledger line is the referee ("backs N
|
||||
# sequences all at 200000 tokens at once") — a pair this tight is read off the
|
||||
# boot, never assumed.
|
||||
# At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
|
||||
# the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
|
||||
# 11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
|
||||
# slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
|
||||
# tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
|
||||
# sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
|
||||
# Secondary saving: the prefix-cache entry shrinks with the attention plane
|
||||
# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
|
||||
# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
|
||||
# many conversations as it did at bf16 — no re-size needed to park three slots.
|
||||
FRANK_ARCHON_KV_DTYPE=fp8
|
||||
# Spec decode: the checkpoint's own MTP head, served at depth 2 — the
|
||||
# publisher's card ships depth 3 as its peak; this fleet's ladder was
|
||||
# measured at 2 and the swap keeps every argument, so 2 stays (rows
|
||||
# 3 x (1+2) = 9, inside the wht6 25-row exactness bound).
|
||||
# pool_mib >= tokens x 13.8125 KiB + 110 x seqs + 448
|
||||
# 6912 -> (6912 - 220 - 448) x 1024 / 13.8125 = 462,946 tokens; the engine
|
||||
# reported 462,752 (0.04% page rounding) — 2 x 200,000 with 62k of slack,
|
||||
# and the ledger says it outright: "backs 2 sequences all at 200000 tokens
|
||||
# at once, which is the worst case".
|
||||
# (the fp8 pool of 3543 gave 386,904 — it never held 2 x 200k, it held 1.93
|
||||
# sessions, and a third long conversation preempted to the host/disk tiers).
|
||||
# Secondary cost: the prefix-cache entry grows with the attention plane
|
||||
# (one entry 58.50 KiB at fp8 -> 110.50 KiB at bf16, measured on the two boot
|
||||
# ledgers' `kv tier transfer` lines), so the host tier below doubles to keep
|
||||
# the same number of parked conversations, out of RAM the host has
|
||||
# (104 GiB available with the service loaded).
|
||||
FRANK_ARCHON_KV_DTYPE=bf16
|
||||
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
||||
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
||||
# flag states it so the ledger records the intent).
|
||||
FRANK_ARCHON_SPEC_TOKENS=2
|
||||
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
||||
# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
|
||||
# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per
|
||||
# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions
|
||||
# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full
|
||||
# parked set fits with the tier as it stands — at RAM the host has (104 GiB
|
||||
# available with the service loaded).
|
||||
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
|
||||
# The host tier is sized in ENTRIES, and one entry doubles with the cache
|
||||
# dtype (58.50 KiB at fp8 -> 117 KiB at bf16, per the ledger's `kv tier
|
||||
# transfer` line): 4096 MiB at bf16 parks ~1.4 sessions of 200k where it
|
||||
# parked ~2.9 at fp8. Doubled to keep the capacity, at RAM the host has
|
||||
# (104 GiB available with the service loaded).
|
||||
FRANK_ARCHON_PREFIX_HOST_MIB=8192
|
||||
FRANK_ARCHON_PREFIX_DISK_MIB=65536
|
||||
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
|
||||
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
|
||||
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
|
||||
# The PLE n-gram table (47.7 GiB) serves from pinned host RAM the cards
|
||||
# read. `ram` REFUSES the start if the memlock limit is too small — a
|
||||
# named refusal beats a mid-load allocator failure — and the unit's
|
||||
# LimitMEMLOCK=infinity is the other half of the arrangement.
|
||||
FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
||||
# TTM preflight: the pinned ledger (2 x 44 GiB pools + 47.7 GiB PLE + 8 GiB
|
||||
# prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
|
||||
# pages_limit (the current kernel exposes NO pages_max; the cap function
|
||||
# takes the min over the knobs that exist). Below the plan the load dies
|
||||
# mid-pin with an allocator error instead of this named refusal.
|
||||
# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every
|
||||
# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise
|
||||
# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER
|
||||
# FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the
|
||||
# start 6,919 times in ~20 h while Restart=always re-ran it. The gate in
|
||||
# scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the
|
||||
# root unit is privileged; the knob is a 0644 root module parameter), so the
|
||||
# recovery is a boot, not a visit to the machine.
|
||||
# mid-pin with an allocator error instead of this named refusal — raise the
|
||||
# knob first (runtime sysfs write; kernel cmdline fallback + one reboot only
|
||||
# if the file is read-only), see the runbook. Cap raised 2026-10-05 to
|
||||
# 52,428,800 pages = 200 GiB, which still clears this floor.
|
||||
FRANK_ARCHON_TTM_MIN_MIB=150000
|
||||
FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
|
||||
# multi-flag values as one string; the serve script expands it unquoted on
|
||||
# purpose.
|
||||
FRANK_ARCHON_EXTRA_ARGS=
|
||||
# The .rad's minimum size in MiB (deployer gate; a truncated fetch is a
|
||||
# guaranteed long-timeout boot instead of a fast failure). The deployed
|
||||
# publisher container measures 116,322 MiB (121,969,901,568 B); the
|
||||
# in-house w4dn8 rollback measured 116,416 MiB. Floor sits just under both.
|
||||
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
|
||||
# guaranteed long-timeout boot instead of a fast failure). The converted
|
||||
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
|
||||
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
|
||||
# tail) - floor sits just under the payload.
|
||||
FRANK_ARCHON_RAD_MIN_MIB=110000
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# frank-archon.service — radiance (StillDeadcode, Apache-2.0) endpoint
|
||||
# (the publisher's 4-bit-codebook container of the official
|
||||
# Qwen/Qwen3.8-Flash-Next, qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2, wht6 wire),
|
||||
# dankserver :7770. 2026-10-15 payload swap from the in-house w4dn8 (the
|
||||
# swap commit is the revert; its .rad stays on disk).
|
||||
# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
|
||||
# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
|
||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
|
||||
# frank-archon-freetoken-pr587-20261005 is the pre-cutover state; the
|
||||
# retired unit/env/serve trio and Dockerfile are in that tag).
|
||||
#
|
||||
# Same port, same served alias, same container identity as the unit name;
|
||||
# the harness models.yml id is unchanged. The serve script drives
|
||||
@@ -19,7 +19,7 @@
|
||||
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
|
||||
# engine still runs out of dank's rootless podman storage.
|
||||
[Unit]
|
||||
Description=frank-archon — radiance Qwen3.8-Flash-Next (publisher iq4r) on 2x R9700 TP=2
|
||||
Description=frank-archon — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
||||
@@ -35,13 +35,6 @@ Group=dank
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_ARCHON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-archon.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-archon.sh
|
||||
# The `+` prefix runs this as root even though the service is User=dank — the
|
||||
# ttm pages cap is a root-owned knob and the pinned-RAM ledger (~147 GiB) is
|
||||
# unusable below it. After a reboot the knob is back at its default (half of
|
||||
# RAM), so the raise must be part of the start, not a note in a runbook: the
|
||||
# 2026-10-12 power failure reset it and the endpoint refused to boot 6,919
|
||||
# times. The serve script re-verifies the cap after this hook.
|
||||
ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise
|
||||
# systemd stop: wait alone cannot stop the container child; stop it
|
||||
# explicitly. No graceful drain contract is claimed (the OpenAI front end
|
||||
# holds no queue to drain — in-flight decodes are lost, which a stop
|
||||
|
||||
@@ -42,10 +42,11 @@
|
||||
# form of the engine's default, so the seam test can pin the product.
|
||||
# KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
|
||||
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
||||
# drafter is NOT slot-locked on this engine (batching and speculation
|
||||
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
||||
# quickstart ships 2 slots WITH the head), so the second slot is
|
||||
# concurrent capacity, not a speculation tax.
|
||||
# drafter is NOT slot-locked on this engine. A "slots > 1 disables the
|
||||
# drafter" restriction is a property of the engine generation that
|
||||
# imposed it, not of speculation itself — batching and speculation
|
||||
# coexist here, and the card's own quickstart ships 2 slots WITH the
|
||||
# head — so the second slot is concurrent capacity, not a speculation tax.
|
||||
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
||||
# mmap'd page cache beside the checkpoint, not a pinned charge — the
|
||||
# card's own quickstart arithmetic does not bill it):
|
||||
@@ -88,9 +89,10 @@
|
||||
# before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
|
||||
# amdgpu.noretry=0 (check at deploy).
|
||||
|
||||
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
||||
# is what was qualified). 0.16.2 = the version the model card names as its
|
||||
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
|
||||
# Image: digest-pinned, not tag-pinned — a tag can be re-pointed to a
|
||||
# build that was never qualified here, while the digest pins exactly the
|
||||
# build that was tested. 0.16.2 = the version the model card names as its
|
||||
# quickstart pin; the RepoDigest was captured at a pull of that tag.
|
||||
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||
# Container identity: kept from the 27B era (unit ExecStop names it).
|
||||
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
|
||||
|
||||
@@ -16,11 +16,11 @@
|
||||
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
|
||||
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
|
||||
#
|
||||
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||||
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
|
||||
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
|
||||
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
|
||||
# where the 26B MoE fit only one.
|
||||
# The KV rate follows from the architecture: 8 full layers × 4 KV heads ×
|
||||
# 256 dim = 8,192 elements per token per cache. f16 K + f16 V = 32,768
|
||||
# B/token = 32 KiB/token, so the 401,408-token pool is 12.25 GiB. The 24
|
||||
# DeltaNet layers cost a fixed recurrent state per SLOT (not per token) —
|
||||
# that is why two 200k slots fit where the 26B MoE fit only one.
|
||||
#
|
||||
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
|
||||
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
|
||||
|
||||
@@ -12,7 +12,12 @@ Group=dank
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-zealot.sh
|
||||
Restart=on-failure
|
||||
# Restart=always is the right policy for a serving daemon: a process that
|
||||
# exits on its own — exit 0 — is a CLEAN stop to systemd, so on-failure
|
||||
# never fires and the endpoint stays dark (with the unit reporting
|
||||
# success) until the next boot. frank-zealot is the fleet's compaction
|
||||
# hub — every endpoint's compaction dies when it does.
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
|
||||
Executable
+133
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env bash
|
||||
# llm-gates.sh — the artifact gates every LLM endpoint runs before it starts.
|
||||
#
|
||||
# Why this file exists: every LLM endpoint gates its payload twice — the
|
||||
# serve script at start, the deployer before dispatch. Two hand-written
|
||||
# copies of the same checks drift apart in wording and strictness, and the
|
||||
# drift surfaces as deploy-time disagreement: one side passes, the other
|
||||
# refuses, and the failure lands in production instead of in the gate.
|
||||
# Both sides source this library, so the refusal is one text and one rule.
|
||||
#
|
||||
# Contract (public interface):
|
||||
# require_exec PATH LABEL HINT
|
||||
# require_file PATH LABEL HINT
|
||||
# require_bytes PATH LABEL WANT_BYTES HINT
|
||||
# require_min_mib PATH LABEL MIN_MIB HINT
|
||||
# require_dir_files DIR LABEL HINT FILE...
|
||||
# require_image REF HINT
|
||||
#
|
||||
# Each returns 0 and prints nothing on success. On failure it prints two
|
||||
# lines to stderr — the first naming the endpoint ($GATE_PREFIX) and the
|
||||
# artifact, the second the remedy — and returns 1. It NEVER exits: the serve
|
||||
# scripts run under `set -e` (so a refusal is terminal), while the deployer
|
||||
# wants `|| return 1` so a failed gate reports a red deploy for that one
|
||||
# service and the dispatcher's state marker stays pending.
|
||||
# Lines are labelled by ARTIFACT CLASS, and the class is the whole label
|
||||
# (the path rides in the same line): the class + path is what an operator
|
||||
# acts on, so no wording order and no payload rename can make the two twins
|
||||
# disagree. The deployer's old "halogen vision tower" and the serve script's
|
||||
# "vision-tower" were one gate in two vocabularies that no grep could join.
|
||||
|
||||
# Size checks use equality, not a floor, for `require_bytes`: a fetch that
|
||||
# truncates looks like a success (an HTTP error exit is often swallowed, and
|
||||
# the error body is written in the artifact's place), so "exists and is big
|
||||
# enough" is not proof of the right artifact. The declared byte count is a
|
||||
# contract in both directions — a bigger file is a DIFFERENT artifact.
|
||||
|
||||
# GATE_PREFIX names the endpoint in every refusal (e.g. frank-dragoon). The
|
||||
# caller exports it; shellcheck cannot follow it across the source boundary,
|
||||
# so the read site carries the directive.
|
||||
# shellcheck disable=SC2034
|
||||
: "${GATE_PREFIX:=gate}"
|
||||
|
||||
# Two bash behaviors make the `return 1` at every call site load-bearing:
|
||||
# * a function whose body ends inside an `if` returns the CONDITION's
|
||||
# status (true = 0), so a gate that ends on its own refusal prints the
|
||||
# refusal and reports SUCCESS;
|
||||
# * `set -e` is suspended for any command in a `… || …` chain, so a gate
|
||||
# that runs inside one must not be trusted to abort the caller.
|
||||
# Every gate's refusal therefore carries an explicit `|| return 1`, and the
|
||||
# caller consumes the status (`|| exit 1`, `|| return 1`).
|
||||
_gate_fail() {
|
||||
printf '%s: %s\n' "$GATE_PREFIX" "$1" >&2
|
||||
printf ' hint: %s\n' "$2" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# require_file — PATH must be a regular file. A directory (an unpacked
|
||||
# staging dir wearing the payload's name) and a symlink (a HuggingFace cache
|
||||
# snapshot's entry, which dangles inside the container mount) both refuse.
|
||||
require_file() {
|
||||
local path="$1" label="$2" hint="$3"
|
||||
if [[ -L "$path" || ! -f "$path" ]]; then
|
||||
_gate_fail "$label: missing at $path" "$hint" || return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# require_exec — PATH must be an executable regular file. The native runtimes
|
||||
# (llama.cpp's llama-server) are binaries the operator builds; a tree that
|
||||
# compiled but lost its +x, or a directory wearing the binary's name, must
|
||||
# refuse the same way a missing one does.
|
||||
require_exec() {
|
||||
local path="$1" label="$2" hint="$3"
|
||||
if [[ -L "$path" || ! -x "$path" ]]; then
|
||||
_gate_fail "$label: missing at $path" "$hint" || return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# require_bytes — PATH must exist AND be exactly WANT_BYTES bytes.
|
||||
require_bytes() {
|
||||
local path="$1" label="$2" want="$3" hint="$4" got
|
||||
require_file "$path" "$label" "$hint" || return 1
|
||||
got="$(stat -c %s "$path")"
|
||||
if [[ "$got" != "$want" ]]; then
|
||||
_gate_fail "$label: is $got bytes, expected $want" "$hint" || return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# require_min_mib — PATH must be at least MIN_MIB mebibytes. The floor form
|
||||
# is for build products (a .rad container is a range of legal sizes; a
|
||||
# truncated convert is not).
|
||||
require_min_mib() {
|
||||
local path="$1" label="$2" min="$3" hint="$4" mib
|
||||
require_file "$path" "$label" "$hint" || return 1
|
||||
mib=$(( $(stat -c %s "$path") / 1048576 ))
|
||||
if (( mib < min )); then
|
||||
_gate_fail "$label: is $mib MiB, expected >= $min MiB" "$hint" || return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# require_dir_files — DIR must be a directory and every named member a
|
||||
# regular file, never a symlink: snapshot layouts link files into place, and
|
||||
# a link dangles inside the container mount even when the directory itself
|
||||
# is real. A symlinked DIRECTORY is refused too — the same layout can link
|
||||
# the directory itself, and the link dangles identically.
|
||||
require_dir_files() {
|
||||
local dir="$1" label="$2" hint="$3"
|
||||
shift 3
|
||||
local f
|
||||
if [[ -L "$dir" || ! -d "$dir" ]]; then
|
||||
_gate_fail "$label: missing at $dir" "$hint" || return 1
|
||||
fi
|
||||
for f in "$@"; do
|
||||
if [[ -L "$dir/$f" || ! -f "$dir/$f" ]]; then
|
||||
_gate_fail "$label: $dir/$f is missing or a symlink" "$hint" || return 1
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
# require_image — the image ref must be in podman storage. A name@sha256 ref
|
||||
# exists only if THAT manifest was pulled, so the gate is the ref itself, in
|
||||
# one command: no pipefail/grep race (the 2026-09-30 lesson).
|
||||
require_image() {
|
||||
local ref="$1" hint="$2"
|
||||
if ! podman image exists "$ref"; then
|
||||
_gate_fail "image: $ref is not in podman storage" "$hint" || return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
@@ -29,10 +29,11 @@
|
||||
# its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
|
||||
# getattr_static()s each one off the instance, breaking on the first miss.
|
||||
# So BOTH wrappers need all three attributes.
|
||||
# * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
|
||||
# "Cannot create a consistent method resolution order (MRO) for bases ..."
|
||||
# (GLM's wrappers already carry four/five Protocol-derived bases). The
|
||||
# attribute route avoids the MRO entirely. Also proven by simulation:
|
||||
# * Why not inheritance: adding SupportsEagle3 as a BASE class dies at
|
||||
# class creation with "Cannot create a consistent method resolution
|
||||
# order (MRO) for bases ..." (GLM's wrappers already carry four/five
|
||||
# Protocol-derived bases; the base ordering cannot take another). The
|
||||
# attribute route avoids the MRO entirely; simulation confirms that
|
||||
# the four-base ForCausalLM ordering cannot take SupportsEagle3 in ANY
|
||||
# position.
|
||||
# * Method placement: without inheriting, the Protocol's default
|
||||
|
||||
@@ -1,35 +1,29 @@
|
||||
#!/usr/bin/env python3
|
||||
"""probe-frank-archon-kv.py — verify the seqs x max-model-len KV contract on a
|
||||
"""probe-frank-archon-kv.py — verify the 2 x 200,000 KV contract on a LIVE
|
||||
frank-archon (radiance) service. Run it ON dankserver: it talks to
|
||||
127.0.0.1:7770 and builds its prompts from the repo checkout there.
|
||||
No scratch container, no mask: the service under test is the service.
|
||||
|
||||
ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'
|
||||
|
||||
Four probes, in increasing cost:
|
||||
Three probes, in increasing cost:
|
||||
A decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
|
||||
Reports the engine's own `timings` (prefill, decode, draft acceptance).
|
||||
The bench law applies: >=128 generated tokens, greedy, real text.
|
||||
B the capacity contract — as many CONCURRENT chats of ~197k prompt tokens as
|
||||
the deployed `FRANK_ARCHON_MAX_SEQS` (read from the repo env file on this
|
||||
host), each carrying a unique marker at its head. All must answer their own
|
||||
marker. This is what the endpoint owes: N full-height conversations
|
||||
resident at once, exact retrieval in every one of them.
|
||||
Method: >=128 generated tokens (below that, first-token latency
|
||||
dominates the decode rate), greedy (sampling noise is unattributable),
|
||||
real text (synthetic text inflates draft acceptance).
|
||||
B the capacity contract — TWO concurrent chats of ~197k prompt tokens
|
||||
(~394k tokens resident) each carrying a unique marker at its head. Both
|
||||
must answer their own marker. This is what a 2 x 200,000 endpoint owes:
|
||||
two full-height conversations resident at once, exact retrieval in both.
|
||||
C prefix-cache restore — probe A's cold prompt again, so the disk tier has
|
||||
to give the blocks back (cache_n > 0, TTFT collapses).
|
||||
D concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of
|
||||
the window. This is the cost a slot count imposes on DECODE, which no
|
||||
prefill number shows: the fused-step schedule is unfair, so every stream is
|
||||
printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8:
|
||||
single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly
|
||||
window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate.
|
||||
Concurrency, not the slab, is what decode pays for slots. The 2026-10-13
|
||||
shape hunt missed this for a whole day because leg B generates 24 tokens.
|
||||
PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
|
||||
|
||||
PROBE PITFALL: radiance routes the thinking block to
|
||||
`reasoning_content`, so a chat request whose budget is spent inside
|
||||
`<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
|
||||
reads `content` alone scores 0/2 on a healthy server. Every request here sets
|
||||
chat_template_kwargs.enable_thinking=false and reads both fields.
|
||||
reads `content` alone scores 0/2 on a healthy server. Every request here
|
||||
sets chat_template_kwargs.enable_thinking=false and reads both fields.
|
||||
|
||||
A/B method for a shape change (no scratch container): write a variant env file
|
||||
on the host, point a transient systemd drop-in at it
|
||||
@@ -122,51 +116,23 @@ def show(tag, j, wall, expect=None):
|
||||
print(line)
|
||||
|
||||
|
||||
def deployed_shape():
|
||||
"""The contract is the deployed env, not a constant: slot count and window
|
||||
come from the checkout on this host, so the probe cannot drift from the
|
||||
service it is testing (the 2026-10-13 shape is 4 x 200,000; before that
|
||||
2 x 200,000, and the fp8 era 3 x 200,000)."""
|
||||
shape = {}
|
||||
with open(f"{REPO}/scripts/frank-archon.env") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if line.startswith("FRANK_ARCHON_") and "=" in line:
|
||||
key, _, val = line.partition("=")
|
||||
shape[key] = val.strip().strip('"')
|
||||
return (int(shape["FRANK_ARCHON_MAX_SEQS"]),
|
||||
int(shape["FRANK_ARCHON_MAX_MODEL_LEN"]),
|
||||
shape["FRANK_ARCHON_KV_DTYPE"])
|
||||
|
||||
|
||||
def main():
|
||||
seqs, maxlen, dtype = deployed_shape()
|
||||
print(f"shape: {seqs} x {maxlen:,} tokens, kv {dtype}")
|
||||
print("health:", json.dumps(get("/health")))
|
||||
print("models:", [m.get("id") for m in get("/v1/models").get("data", [])])
|
||||
deep = corpus(160_000)
|
||||
wide = corpus(1_100_000)
|
||||
|
||||
# 3.28 chars/token is the measured ratio of this corpus through the
|
||||
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
|
||||
# deploy); 98.5 % of the window leaves the response room under the ceiling.
|
||||
chars = int(0.985 * maxlen * 3.28)
|
||||
# Each marker's FIRST hyphen-separated word is unique, because show()
|
||||
# scores the hit on that word — a shared prefix would score every slot HIT
|
||||
# on any answer that says the word.
|
||||
words = ["cobalt", "quartz", "basalt", "ember", "willow", "cobweb", "flint"]
|
||||
facts = {f"B{i+1}": f"{words[i % len(words)]}-flag-{(i + 1) * 137}"
|
||||
for i in range(seqs)}
|
||||
bodies = {s: chat(wide, chars, s, f, 24) for s, f in facts.items()}
|
||||
facts = {"B1": "cobalt-harbor-13", "B2": "quartz-meadow-42"}
|
||||
bodies = {s: chat(wide, 646_000, s, f, 24) for s, f in facts.items()}
|
||||
print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per "
|
||||
f"request, x{seqs} concurrent")
|
||||
f"request, x2 concurrent")
|
||||
|
||||
# A + C first (cheap), B last (the expensive set).
|
||||
# A + C first (cheap), B last (the expensive pair).
|
||||
for tag, salt in (("A-cold", "A1-9137"), ):
|
||||
j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512))
|
||||
show(tag, j, wall, "zephyr-lantern-7")
|
||||
t0 = time.time()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as ex:
|
||||
futs = {s: ex.submit(post, b) for s, b in bodies.items()}
|
||||
for s, fut in futs.items():
|
||||
try:
|
||||
@@ -175,24 +141,6 @@ def main():
|
||||
except Exception as e: # a preemption/timeout lands here
|
||||
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
|
||||
print(f"[B] total {time.time() - t0:.1f}s")
|
||||
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
|
||||
dchars = int(0.25 * maxlen * 3.28)
|
||||
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
|
||||
for i in range(seqs)]
|
||||
print(f"[D] {seqs} concurrent streams x 512 tokens, "
|
||||
f"{dchars:,} chars each")
|
||||
t0 = time.time()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
|
||||
futs = [ex.submit(post, b) for b in dbodies]
|
||||
for i, fut in enumerate(futs, 1):
|
||||
try:
|
||||
j, wall, _ = fut.result()
|
||||
show(f"D-s{i}", j, wall)
|
||||
except Exception as e: # a preemption/timeout lands here
|
||||
print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}")
|
||||
tot = time.time() - t0
|
||||
print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = "
|
||||
f"{seqs * 512 / tot:.1f} t/s aggregate decode")
|
||||
j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
|
||||
show("C-prefix-hit", j, wall, "zephyr-lantern-7")
|
||||
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
# qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
|
||||
# (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
|
||||
# base) as a radiance container for frank-archon: EVERY routed expert -- gate_up AND down --
|
||||
# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
|
||||
# ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
|
||||
# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
|
||||
# gate_up w4): the served kernel is w4nl64a8h for the whole expert set, bf16 experts beside it.
|
||||
# through GPTQ into the w4nl64a8h table. The converter declares ONE
|
||||
# encoding per expert table (qwen4exp_fp8 rad_arch_declare): the served
|
||||
# kernel covers the whole expert set, and the only legal mixed layout is a
|
||||
# WHOLE expert kept plain bf16 beside it. That rules out the original
|
||||
# split tier (down block-fp8 + gate_up w4), which asks one table for two
|
||||
# encodings.
|
||||
# The rest is qwen4exp-w5dn64-i8-hc8m.recipe verbatim: int8 trunk linears, block-fp8 hc mixers
|
||||
# and MTP fc, the fixed-scale n-gram table, the 2-bit draft head.
|
||||
#
|
||||
@@ -25,11 +27,14 @@
|
||||
|
||||
# The protected experts, cast to bf16 ahead of both expert rules. THE NAMES BELOW ARE THE BASE
|
||||
# CHECKPOINT'S (from its 1M-token recording, qwen4exp-w5dn64-i8-hc8m.recipe) -- a warm start only.
|
||||
# The Swift 1.5 finetune's routing distribution is not assumed equal: the derivation step rewrites
|
||||
# this section from Swift's OWN calibration recording before this recipe is ever run, by the base's
|
||||
# law -- an expert whose share of its layer's down-input energy is at least 3% and at least three
|
||||
# times its share of the routing, an even count on each rank's half of the layer (the four-bit GEMM
|
||||
# takes the rest as two equal tables by parity) by adding that half's next most energetic expert.
|
||||
# The Swift 1.5 finetune's routing distribution is not assumed equal: the
|
||||
# derivation step rewrites this section from Swift's OWN calibration
|
||||
# recording before this recipe is ever run. The selection rule is the base
|
||||
# checkpoint's: an expert whose share of its layer's down-input energy is
|
||||
# at least 3% AND at least three times its share of the routing; an even
|
||||
# count on each rank's half of the layer (the four-bit GEMM takes the rest
|
||||
# as two equal tables by parity), padded with that half's next most
|
||||
# energetic expert.
|
||||
# PROTECTED-BEGIN (rewritten from Swift's OWN grams by derive-protected-experts.py)
|
||||
blk.1.ffn_*_exps.271.weight cast dtype=bf16
|
||||
blk.1.ffn_*_exps.355.weight cast dtype=bf16
|
||||
|
||||
@@ -1,13 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# serve-frank-archon.sh — frank-archon endpoint: radiance (StillDeadcode,
|
||||
# Apache-2.0) serving the PUBLISHER's 4-bit-codebook container of the
|
||||
# official Qwen/Qwen3.8-Flash-Next (the engine author's own rad-convert of
|
||||
# the bf16 checkpoint: qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2, wht6 wire),
|
||||
# dankserver :7770. 2026-10-15 payload swap from the in-house w4dn8 of the
|
||||
# Swift1.5 finetune (payload on disk = rollback; the swap commit is the
|
||||
# revert); 2026-10-05 radiance cutover from FreeToken (git tag
|
||||
# frank-archon-freetoken-pr587-20261005 is the pre-radiance state).
|
||||
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
|
||||
# NATIVE block-FP8 precision from a rad-convert container
|
||||
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
|
||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
|
||||
# one card; git tag frank-archon-freetoken-pr587-20261005 is the
|
||||
# pre-cutover state).
|
||||
#
|
||||
# The run line below IS the engine argv: the image's entrypoint is the
|
||||
# engine and its tools sit beside it on PATH. Unlike every predecessor
|
||||
@@ -15,11 +13,9 @@
|
||||
# log states every budget before anything is allocated (weights vs KV
|
||||
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
||||
# the P2P answer per pair) — read it after every resize.
|
||||
# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
|
||||
# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
|
||||
# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
|
||||
# own budget table makes claimable, spent on the slab because prefill is
|
||||
# slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
|
||||
# Deployed shape (2026-10-11): 2 decode slots x 200,000 tokens, bf16
|
||||
# (unquantized) KV in a 6,912 MiB pool, the weights slab cut to 22,208 MiB to
|
||||
# pay for it, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
|
||||
# 8 GiB + disk 64 GiB, PLE pinned in RAM.
|
||||
#
|
||||
# The root unit (scripts/frank-archon.service, User=dank,
|
||||
@@ -30,6 +26,14 @@
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# The artifact gates are the shared library: the serve script and the
|
||||
# deployer gate the same payload, and two hand-written copies of the same
|
||||
# checks drift apart until one side passes what the other refuses. One
|
||||
# library, one refusal text, one rule.
|
||||
export GATE_PREFIX=frank-archon
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/lib/llm-gates.sh"
|
||||
ENV_FILE="${FRANK_ARCHON_ENV_FILE:-$SCRIPT_DIR/frank-archon.env}"
|
||||
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
@@ -65,43 +69,26 @@ set +a
|
||||
: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
|
||||
: "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
|
||||
: "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
|
||||
: "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
|
||||
: "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"
|
||||
|
||||
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
|
||||
# in storage only if THAT manifest was pulled — a retagged look-alike or a
|
||||
# newer latest does not satisfy it. `podman image exists` takes a
|
||||
# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
|
||||
# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
|
||||
# storage (root unit, User=dank — the fleet's frank-archon shape).
|
||||
if ! podman image exists "$FRANK_ARCHON_IMAGE"; then
|
||||
echo "ERROR: radiance image absent: $FRANK_ARCHON_IMAGE" >&2
|
||||
echo "frank-archon: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
|
||||
exit 1
|
||||
fi
|
||||
# newer latest does not satisfy it. One command, no pipefail/grep race (the
|
||||
# 2026-09-30 lesson). Runs as dank: the unit's process uses dank's ROOTLESS
|
||||
# podman storage (root unit, User=dank — the fleet's frank-archon shape).
|
||||
require_image "$FRANK_ARCHON_IMAGE" \
|
||||
"podman pull docker://stilldeadcode/radiance:latest (as dank)" || exit 1
|
||||
|
||||
# The payload gate: the .rad container file, sized. A truncated convert
|
||||
# would otherwise buy a guaranteed long-timeout boot instead of this fast
|
||||
# failure. The staging dir / raw checkpoint are NOT this script's concern
|
||||
# (scripts/build-frank-archon-rad.sh built the file; its rad-info output
|
||||
# is the record).
|
||||
RAD="$FRANK_ARCHON_MODEL_DIR/$FRANK_ARCHON_MODEL_FILE"
|
||||
if [ ! -f "$RAD" ]; then
|
||||
echo "ERROR: .rad container missing: $RAD" >&2
|
||||
echo "frank-archon: run scripts/build-frank-archon-rad.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
|
||||
if [ "$RAD_MIB" -lt "$FRANK_ARCHON_RAD_MIN_MIB" ]; then
|
||||
echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_ARCHON_RAD_MIN_MIB" >&2
|
||||
echo "frank-archon: the convert was truncated — rebuild it" >&2
|
||||
exit 1
|
||||
fi
|
||||
# (the convert script built the file; its rad-info output is the record).
|
||||
require_min_mib "$FRANK_ARCHON_MODEL_DIR/$FRANK_ARCHON_MODEL_FILE" \
|
||||
container "$FRANK_ARCHON_RAD_MIN_MIB" \
|
||||
"build it: scripts/convert-swift15-w4dn8.sh (rad-info is the record)" || exit 1
|
||||
|
||||
if [ ! -f "$FRANK_ARCHON_TEMPLATE" ]; then
|
||||
echo "ERROR: chat template missing: $FRANK_ARCHON_TEMPLATE" >&2
|
||||
exit 1
|
||||
fi
|
||||
require_file "$FRANK_ARCHON_TEMPLATE" template \
|
||||
"the chat template is vendored: scripts/qwen38-froggeric-v22.5.jinja" || exit 1
|
||||
|
||||
mkdir -p "$FRANK_ARCHON_STATE_DIR/kvcache/frank-archon"
|
||||
# The engine's create is NOT recursive: with only the state dir mounted, its
|
||||
@@ -117,24 +104,44 @@ mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
|
||||
# that wrote it. The dir is pre-created for the same reason as kvcache
|
||||
# (the engine's create is not recursive).
|
||||
|
||||
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
||||
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
|
||||
# GPU access is mandatory; the node paths are knobs so the seam can drive
|
||||
# this gate on a box without AMD GPUs (production defaults).
|
||||
DEV_KFD="${FRANK_ARCHON_DEV_KFD:-/dev/kfd}"
|
||||
DEV_DRI="${FRANK_ARCHON_DEV_DRI:-/dev/dri}"
|
||||
if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
|
||||
echo "ERROR: GPU device nodes absent ($DEV_KFD, $DEV_DRI)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# TTM preflight: the pinned ledger (2 x FRANK_ARCHON_HOST_POOL_MIB + the
|
||||
# 47.7 GiB PLE table + the prefix host tier, ~147 GiB) rides TTM's shared
|
||||
# pages, capped by ttm's pages_limit (the current kernel exposes NO
|
||||
# pages_max; the cap function takes the min over the knobs that exist).
|
||||
# Below FRANK_ARCHON_TTM_MIN_MIB the load dies mid-pin with an allocator
|
||||
# error, so the gate refuses first — and RAISES the knob before refusing:
|
||||
# the cap is a runtime knob whose default is half of RAM, so a power failure
|
||||
# resets it (measured 2026-10-12 23:17 on this host: 200 GiB -> 125,988 MiB,
|
||||
# and Restart=always re-ran the refusal 6,919 times over ~20 h).
|
||||
# FRANK_ARCHON_TTM_TARGET_MIB is the recorded host ceiling, not the floor.
|
||||
# shellcheck source=/dev/null
|
||||
source "$SCRIPT_DIR/ttm-preflight.sh"
|
||||
ttm_ensure_cap || exit 1
|
||||
# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
||||
# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
|
||||
# pages_limit (the current kernel exposes NO pages_max; the cap function
|
||||
# takes the min over the knobs that exist). Below FRANK_ARCHON_TTM_MIN_MIB
|
||||
# the load dies mid-pin with an allocator error instead of this named
|
||||
# refusal — raise the knob first (runtime sysfs write; kernel cmdline
|
||||
# fallback + one reboot only if the file is read-only), see the runbook.
|
||||
# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
|
||||
ttm_cap_mib() {
|
||||
# min over the knobs THIS kernel exposes — the current kernel has
|
||||
# pages_limit only (no pages_max); a knob that is absent is skipped,
|
||||
# not a zero (a zero would refuse a healthy cap).
|
||||
local v cur="" dir
|
||||
dir="${FRANK_ARCHON_TTM_SYSFS:-/sys/module/ttm/parameters}"
|
||||
for p in "$dir/pages_limit" "$dir/pages_max"; do
|
||||
[ -r "$p" ] || continue
|
||||
v="$(cat "$p" 2>/dev/null)" || continue
|
||||
[ -n "$v" ] || continue
|
||||
v=$(( v * 4 / 1024 ))
|
||||
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
|
||||
done
|
||||
echo "${cur:-0}"
|
||||
}
|
||||
TTM_CAP="$(ttm_cap_mib)"
|
||||
if [ "$TTM_CAP" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
||||
echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
||||
echo "frank-archon: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_ARCHON_TTM_MIN_MIB in 4 KiB pages" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# A crashed/killed unit can leave the container name registered (systemd
|
||||
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
||||
@@ -145,7 +152,7 @@ for CTR in $FRANK_ARCHON_LEGACY_CONTAINERS "$FRANK_ARCHON_CONTAINER"; do
|
||||
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
||||
done
|
||||
|
||||
LOG=/home/dank/logs/frank_archon.log
|
||||
LOG="${FRANK_ARCHON_LOG:-/home/dank/logs/frank_archon.log}"
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
|
||||
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
|
||||
|
||||
@@ -29,7 +29,14 @@
|
||||
# exactly like frank-archon's.
|
||||
set -euo pipefail
|
||||
|
||||
# The artifact gates are the shared library — the serve script and the
|
||||
# deployer refuse the same missing/truncated payload with the same words
|
||||
# (the 2026-10-01 crash-loop was the two hand-written twins disagreeing).
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
export GATE_PREFIX=frank-dragoon
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/lib/llm-gates.sh"
|
||||
|
||||
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "frank-dragoon: env file not found: $ENV_FILE" >&2
|
||||
@@ -57,55 +64,38 @@ set +a
|
||||
: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set}"
|
||||
: "${HALOGEN_SPEC_ADAPT:?HALOGEN_SPEC_ADAPT must be set (0 = head always on)}"
|
||||
|
||||
# Capture podman's output BEFORE grepping: under `set -o pipefail` the
|
||||
# `podman images | grep -q` pattern races (grep exits on first match,
|
||||
# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
|
||||
# 2026-09-30 deploy refusal).
|
||||
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
|
||||
if ! grep -qF "${FRANK_DRAGOON_IMAGE#*@}" <<<"$DIGESTS"; then
|
||||
echo "frank-dragoon: image $FRANK_DRAGOON_IMAGE not in podman storage;" >&2
|
||||
echo " pull it: podman pull ${FRANK_DRAGOON_IMAGE%@*}" >&2
|
||||
exit 1
|
||||
fi
|
||||
# The image gate is the digest-pinned ref itself: `podman image exists` takes
|
||||
# a name@sha256 ref in one command, so a retagged look-alike or a newer
|
||||
# latest cannot satisfy it and there is no pipefail/SIGPIPE race with grep
|
||||
# (the 2026-09-30 deploy refusal). Runs as dank: the unit's process uses
|
||||
# dank's rootless storage.
|
||||
require_image "$FRANK_DRAGOON_IMAGE" \
|
||||
"pull it: podman pull docker://$FRANK_DRAGOON_IMAGE" || exit 1
|
||||
|
||||
# Size-exact gates on all three payload files: a truncated curl exits 0 on
|
||||
# some failures and a partial .hgn loads far enough to look alive, so the
|
||||
# byte counts from the Hub tree API are the contract (fleet fetch law).
|
||||
ck() {
|
||||
# ck <env-name> <path> <expected-bytes>
|
||||
if [ ! -f "$2" ]; then
|
||||
echo "frank-dragoon: $1 $2 missing;" >&2
|
||||
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
local sz
|
||||
sz="$(stat -c %s "$2")"
|
||||
if [ "$sz" != "$3" ]; then
|
||||
echo "frank-dragoon: $1 is $sz bytes, expected $3 (partial download?)" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
ck checkpoint "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_CHECKPOINT" "$FRANK_DRAGOON_CHECKPOINT_BYTES"
|
||||
ck n-gram-table "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_NGRAM_TABLE" "$FRANK_DRAGOON_NGRAM_BYTES"
|
||||
ck vision-tower "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_VISION_TOWER" "$FRANK_DRAGOON_VISION_BYTES"
|
||||
# Size-exact gates on all three payload files: a truncated download can
|
||||
# exit 0, and a partial .hgn loads far enough to look alive, so existence
|
||||
# proves nothing. The byte count published by the Hub tree API is the
|
||||
# contract: any other size is a different (broken) artifact.
|
||||
require_bytes "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_CHECKPOINT" \
|
||||
checkpoint "$FRANK_DRAGOON_CHECKPOINT_BYTES" \
|
||||
"fetch it: bash scripts/download-frank-dragoon-flashnext.sh" || exit 1
|
||||
require_bytes "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_NGRAM_TABLE" \
|
||||
n-gram "$FRANK_DRAGOON_NGRAM_BYTES" \
|
||||
"fetch it: bash scripts/download-frank-dragoon-flashnext.sh" || exit 1
|
||||
require_bytes "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_VISION_TOWER" \
|
||||
vision "$FRANK_DRAGOON_VISION_BYTES" \
|
||||
"fetch it: bash scripts/download-frank-dragoon-flashnext.sh" || exit 1
|
||||
|
||||
# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
|
||||
# snapshot symlinks each file into a sibling blobs/, and those symlinks
|
||||
# dangle inside the container mount — so the directory must be real and
|
||||
# every named file must be a regular file, not a link.
|
||||
TOK_DIR="$FRANK_DRAGOON_MODEL_DIR/tokenizer"
|
||||
if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
|
||||
echo "frank-dragoon: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
|
||||
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
for TF in $FRANK_DRAGOON_TOKENIZER_FILES; do
|
||||
if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
|
||||
echo "frank-dragoon: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
|
||||
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
# The tokenizer file list is word-separated ON PURPOSE (one argument per
|
||||
# file), so the unquoted expansion below is the interface, not a slip.
|
||||
# shellcheck disable=SC2086
|
||||
require_dir_files "$FRANK_DRAGOON_MODEL_DIR/tokenizer" tokenizer \
|
||||
"fetch it: bash scripts/download-frank-dragoon-flashnext.sh" \
|
||||
$FRANK_DRAGOON_TOKENIZER_FILES || exit 1
|
||||
|
||||
# GPU access is mandatory but the node paths are variables so the seam can
|
||||
# drive the gate on a box without an AMD GPU.
|
||||
@@ -115,6 +105,7 @@ if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
|
||||
echo "frank-dragoon: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
|
||||
# The engine refuses to pin when MemAvailable is below the checkpoint plus
|
||||
# a 16 GiB floor: pinned pages cannot be swapped, and under the floor the
|
||||
|
||||
@@ -28,6 +28,14 @@
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# The artifact gates are the shared library (scripts/lib/llm-gates.sh): the
|
||||
# serve script and the deployer gate the same payload, and two hand-written
|
||||
# copies of the same checks drift apart until one side passes what the
|
||||
# other refuses.
|
||||
export GATE_PREFIX=frank-zealot
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/lib/llm-gates.sh"
|
||||
ENV_FILE="${FRANK_ZEALOT_ENV_FILE:-$SCRIPT_DIR/frank-zealot.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
@@ -62,9 +70,9 @@ fi
|
||||
# FP8 class; no true fp8 KV dtype exists in this build).
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
|
||||
# there is no in-file head (the law: MTP is a property of the file, not of the
|
||||
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
|
||||
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors,
|
||||
# so there is no in-file head — a speculative head is a property of the
|
||||
# model file, not of the flags; no flag creates tensors that are absent.
|
||||
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
|
||||
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
|
||||
: "${SPEC_TYPE:=none}"
|
||||
@@ -81,13 +89,18 @@ fi
|
||||
|
||||
# Fail loudly rather than half-loading: a missing file is a download that
|
||||
# has not finished, and Restart=always would turn it into a crash loop that
|
||||
# reads like a runtime fault.
|
||||
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
||||
[[ -e "$path" ]] || { echo "frank-zealot: missing $path" >&2; exit 1; }
|
||||
done
|
||||
# reads like a runtime fault. The refusal text is the shared gate library's,
|
||||
# so the deployer's twin refuses identically.
|
||||
require_file "$LLAMA_BIN" runtime \
|
||||
"build it: scripts/build-llama-master-cuda.sh" || exit 1
|
||||
require_file "$MODEL_FILE" gguf \
|
||||
"fetch it: bash scripts/download-frank-zealot.sh" || exit 1
|
||||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||||
require_file "$MMPROJ_FILE" mmproj \
|
||||
"fetch it: bash scripts/download-frank-zealot.sh" || exit 1
|
||||
fi
|
||||
EXTRA_ARGS=()
|
||||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||||
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-zealot: missing $MMPROJ_FILE" >&2; exit 1; }
|
||||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||||
fi
|
||||
|
||||
|
||||
Executable
+750
@@ -0,0 +1,750 @@
|
||||
#!/usr/bin/env python3
|
||||
"""service-map.py — the only reader of the service inventory.
|
||||
|
||||
Four lists used to agree by discipline (the planner's path rules, the
|
||||
deployer's allow-list, the changed-test mapper, and the scrape/probe/DNS
|
||||
facts written by hand in three config files). They now all read
|
||||
`shared/services.yml` through this tool:
|
||||
|
||||
hosts the deploy targets
|
||||
services [--host H] deployables a target accepts, canonical order
|
||||
allows --host H --service S exit 0 when H accepts S (silent)
|
||||
plan --host H PATH... services selected by changed paths, order-safe
|
||||
seams [--host H] PATH... the Bats seams a changed path must re-run
|
||||
owner --host H PATH the service that owns a file (empty: none)
|
||||
units --host H --service S "<unit> <unit_dir>" lines the handler installs
|
||||
stack --host H --service S the deployable + members one handler deploys
|
||||
get --service S [--host H] F one field as JSON
|
||||
render pihole-hosts the Pi-hole addn-hosts file
|
||||
check [NAME...] integrity + drift gates (exit 1 on drift)
|
||||
|
||||
Two rules the engine enforces structurally:
|
||||
|
||||
* A path selects only services that RUN on the planned target. Path
|
||||
matching that is host-blind (match first, filter later) lets a file on
|
||||
one host plan a service on another; selection is scoped to the host the
|
||||
path belongs to.
|
||||
* The netns family is DERIVED, never declared: a unit file that says
|
||||
`Network=gluetun.container` rides that tunnel, so a gluetun change selects
|
||||
every sharer. The unit files are the source of that coupling.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from functools import lru_cache
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
DEFAULT_INVENTORY = REPO_ROOT / "shared" / "services.yml"
|
||||
|
||||
REQUIRED_KEYS = ("name", "hosts", "kind", "tests")
|
||||
KINDS = {"quadlet", "root-unit", "vm-quadlet", "host"}
|
||||
UNIT_SUFFIXES = (".container", ".network")
|
||||
|
||||
# Repo material that is never a deployed runtime's state.
|
||||
NON_RUNTIME_ROOTS = {"docs", "report", "old", ".githooks", ".obsidian", ".github",
|
||||
"tests", "serverAppConfigs-docs"}
|
||||
NON_RUNTIME_NAMES = {".env.example"}
|
||||
NON_RUNTIME_PATHS = {"README.md", "AGENTS.md", ".gitignore"}
|
||||
|
||||
|
||||
def is_trash(path: str) -> bool:
|
||||
return ".AGENTTRASH" in PurePosixPath(path).parts
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# inventory
|
||||
# --------------------------------------------------------------------------
|
||||
class Inventory:
|
||||
def __init__(self, doc: dict[str, Any], origin: str) -> None:
|
||||
self.origin = origin
|
||||
self.hosts: dict[str, dict[str, Any]] = doc["hosts"]
|
||||
self.services: list[dict[str, Any]] = doc["services"]
|
||||
self.unowned: dict[str, str] = doc.get("unowned", {})
|
||||
for index, svc in enumerate(self.services):
|
||||
svc["_index"] = index
|
||||
|
||||
@classmethod
|
||||
def load(cls, path: str | Path = DEFAULT_INVENTORY) -> "Inventory":
|
||||
doc = yaml.safe_load(Path(path).read_text())
|
||||
if not isinstance(doc, dict) or "services" not in doc:
|
||||
raise SystemExit(f"{path}: not a service inventory document")
|
||||
return cls(doc, str(path))
|
||||
|
||||
def on(self, host: str) -> list[dict[str, Any]]:
|
||||
return [s for s in self.services if host in s["hosts"]]
|
||||
|
||||
def named(self, host: str, name: str) -> list[dict[str, Any]]:
|
||||
return [s for s in self.services if s["name"] == name and host in s["hosts"]]
|
||||
|
||||
def any_named(self, name: str) -> list[dict[str, Any]]:
|
||||
return [s for s in self.services if s["name"] == name]
|
||||
|
||||
@property
|
||||
def order(self) -> list[str]:
|
||||
seen: list[str] = []
|
||||
for svc in self.services:
|
||||
if svc["name"] not in seen:
|
||||
seen.append(svc["name"])
|
||||
return seen
|
||||
|
||||
def edge_ip(self) -> str:
|
||||
"""The single address every DNS name resolves at (Caddy's edge)."""
|
||||
for host in self.hosts.values():
|
||||
if "dns_edge" in host:
|
||||
return str(host["dns_edge"])
|
||||
raise SystemExit(f"{self.origin}: no host declares dns_edge")
|
||||
|
||||
def service(self, name: str) -> dict | None:
|
||||
for svc in self.services:
|
||||
if svc["name"] == name:
|
||||
return svc
|
||||
return None
|
||||
|
||||
def unit_dirs(self, svc: dict[str, Any], host: str | None = None) -> list[str]:
|
||||
"""The repo dirs holding this service's unit files, for one host."""
|
||||
dirs = svc.get("unit_dir")
|
||||
if not dirs:
|
||||
return []
|
||||
if isinstance(dirs, dict):
|
||||
if host is not None:
|
||||
return [dirs[host]] if host in dirs else []
|
||||
return [dirs[h] for h in svc["hosts"] if h in dirs]
|
||||
return [dirs]
|
||||
|
||||
def owns_unit(self, svc: dict[str, Any], host: str, unit: str, directory: str) -> bool:
|
||||
return unit in svc.get("units", []) and directory in self.unit_dirs(svc, host)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# repo facts
|
||||
# --------------------------------------------------------------------------
|
||||
@lru_cache(maxsize=1)
|
||||
def tracked_files() -> tuple[str, ...]:
|
||||
# Tracked + untracked (a check that only saw HEAD would ignore the very
|
||||
# files a work-in-progress commit is adding).
|
||||
out = subprocess.run(
|
||||
["git", "-C", str(REPO_ROOT), "ls-files", "--cached", "--others",
|
||||
"--exclude-standard"],
|
||||
text=True, stdout=subprocess.PIPE, check=True)
|
||||
return tuple(line for line in out.stdout.splitlines() if line)
|
||||
|
||||
|
||||
def _network_values(container: Path) -> list[str]:
|
||||
"""Every `Network=` value a quadlet unit declares (comma lists included)."""
|
||||
values: list[str] = []
|
||||
try:
|
||||
text = container.read_text()
|
||||
except OSError:
|
||||
return values
|
||||
for line in text.splitlines():
|
||||
if line.startswith("Network="):
|
||||
values.extend(v.strip() for v in line[len("Network="):].split(",") if v.strip())
|
||||
return values
|
||||
|
||||
|
||||
class Graph:
|
||||
"""The unit-file coupling, read out of the tree: netns + netns-net users.
|
||||
|
||||
Quadlet installs every user unit flat, so `Network=gluetun.container`
|
||||
names a unit by name, not by path; the same host is the scope, which is
|
||||
also the deployment scope.
|
||||
"""
|
||||
|
||||
def __init__(self, hosts: list[str]) -> None:
|
||||
self.units: dict[tuple[str, str], str] = {} # (host, unit) -> path
|
||||
self.networks: dict[tuple[str, str], str] = {} # (host, net) -> path
|
||||
self.netns_children: dict[tuple[str, str], set[tuple[str, str]]] = {}
|
||||
self.network_users: dict[tuple[str, str], set[tuple[str, str]]] = {}
|
||||
for path in tracked_files():
|
||||
parts = PurePosixPath(path).parts
|
||||
if not parts or parts[0] not in hosts or is_trash(path):
|
||||
continue
|
||||
host, directory, name = parts[0], str(PurePosixPath(*parts[1:-1])), parts[-1]
|
||||
if name.endswith(".container"):
|
||||
self.units[(host, name[: -len(".container")])] = path
|
||||
elif name.endswith(".network"):
|
||||
self.networks[(host, name[: -len(".network")])] = path
|
||||
for (host, unit), path in self.units.items():
|
||||
for value in _network_values(REPO_ROOT / path):
|
||||
if value.endswith(".container"):
|
||||
parent = (host, value[: -len(".container")])
|
||||
self.netns_children.setdefault(parent, set()).add((host, unit))
|
||||
elif value.endswith(".network"):
|
||||
net = (host, value[: -len(".network")])
|
||||
self.network_users.setdefault(net, set()).add((host, unit))
|
||||
|
||||
def rides(self, start: list[tuple[str, str]]) -> set[tuple[str, str]]:
|
||||
"""The transitive netns closure: a unit, everything riding it, and
|
||||
everything riding those (a netns inside a netns)."""
|
||||
found: set[tuple[str, str]] = set()
|
||||
frontier = list(start)
|
||||
while frontier:
|
||||
node = frontier.pop()
|
||||
if node in found:
|
||||
continue
|
||||
found.add(node)
|
||||
frontier.extend(self.netns_children.get(node, ()))
|
||||
return found
|
||||
|
||||
|
||||
@lru_cache(maxsize=8)
|
||||
def graph_for(hosts: tuple[str, ...]) -> Graph:
|
||||
return Graph(list(hosts))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# selection
|
||||
# --------------------------------------------------------------------------
|
||||
def _owner_index(inv: Inventory) -> dict[tuple[str, str, int]]:
|
||||
"""(host, unit) -> service index, for unit-file ownership.
|
||||
|
||||
A quadlet unit name is unique per host (``check units`` enforces one
|
||||
claim), so the coupling closure can resolve a riding unit that lives in
|
||||
a DIFFERENT directory than the changed file — gluetun's netns children
|
||||
are the point.
|
||||
"""
|
||||
index: dict[tuple[str, str], int] = {}
|
||||
for svc in inv.services:
|
||||
for host in svc["hosts"]:
|
||||
for unit in svc.get("units", []):
|
||||
key = (host, unit)
|
||||
if key in index and index[key] != svc["_index"]:
|
||||
raise SystemExit(
|
||||
f"{inv.origin}: unit {unit!r} on {host} claimed by "
|
||||
f"{inv.services[index[key]]['name']} and {svc['name']}"
|
||||
)
|
||||
index[key] = svc["_index"]
|
||||
return index
|
||||
|
||||
|
||||
def select(inv: Inventory, host: str, paths: list[str]) -> list[str]:
|
||||
"""Canonical-order service names the changed paths select for one host."""
|
||||
if host not in inv.hosts:
|
||||
raise SystemExit(f"{inv.origin}: no host {host!r}")
|
||||
graph = graph_for(tuple(inv.hosts))
|
||||
owners = _owner_index(inv)
|
||||
hits: set[int] = set()
|
||||
|
||||
def add(index: int) -> None:
|
||||
hits.add(index)
|
||||
|
||||
for raw in paths:
|
||||
path = raw.replace("\\", "/")
|
||||
parts = PurePosixPath(path).parts
|
||||
if not parts or is_trash(path) or path in NON_RUNTIME_PATHS:
|
||||
continue
|
||||
if PurePosixPath(path).name in NON_RUNTIME_NAMES:
|
||||
continue
|
||||
head = parts[0]
|
||||
if head in NON_RUNTIME_ROOTS:
|
||||
continue
|
||||
# A host's directory is the scope: a frankbook-pro file can never
|
||||
# select a dankserver service.
|
||||
scoped_host = head if head in inv.hosts else None
|
||||
if scoped_host is not None and scoped_host != host:
|
||||
continue
|
||||
|
||||
name = parts[-1]
|
||||
directory = str(PurePosixPath(*parts[1:-1])) if len(parts) > 1 else ""
|
||||
|
||||
if scoped_host and name.endswith(UNIT_SUFFIXES):
|
||||
stem = name.rsplit(".", 1)[0]
|
||||
key = (scoped_host, stem)
|
||||
if name.endswith(".network") and key in graph.networks:
|
||||
users = graph.network_users.get(key, set())
|
||||
for unit in graph.rides(sorted(users)):
|
||||
index = owners.get(unit)
|
||||
if index is not None:
|
||||
add(index)
|
||||
elif key in graph.units:
|
||||
for unit in graph.rides([key]):
|
||||
index = owners.get(unit)
|
||||
if index is not None:
|
||||
add(index)
|
||||
|
||||
for index in list(hits):
|
||||
for dependent in inv.services[index].get("dependents", []):
|
||||
for svc in inv.named(host, dependent):
|
||||
add(svc["_index"])
|
||||
|
||||
for svc in inv.on(host):
|
||||
if path in svc.get("files", []):
|
||||
add(svc["_index"])
|
||||
if path in svc.get("retires", []):
|
||||
add(svc["_index"])
|
||||
for directory_rule in svc.get("dirs", []):
|
||||
if path == directory_rule or path.startswith(directory_rule + "/"):
|
||||
add(svc["_index"])
|
||||
|
||||
return [svc["name"] for svc in sorted((inv.services[i] for i in hits),
|
||||
key=lambda s: s["_index"])]
|
||||
|
||||
|
||||
def seams_for(inv: Inventory, host: str, paths: list[str]) -> list[str]:
|
||||
"""The Bats seams the selected services require, deduplicated, in order."""
|
||||
seams: list[str] = []
|
||||
for name in select(inv, host, paths):
|
||||
for svc in inv.named(host, name):
|
||||
for seam in svc.get("tests", []):
|
||||
if seam not in seams:
|
||||
seams.append(seam)
|
||||
if seams and "tests/deployment_scoping.bats" not in seams:
|
||||
seams.append("tests/deployment_scoping.bats")
|
||||
return seams
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# renderers + facts
|
||||
# --------------------------------------------------------------------------
|
||||
def render_pihole_hosts(inv: Inventory) -> str:
|
||||
edge = inv.edge_ip()
|
||||
lines = [
|
||||
"# A records for the flat franksdank.cloud scheme (dnsmasq addn-hosts",
|
||||
"# format: <IP> <name>). RENDERED — do not edit by hand:",
|
||||
"#",
|
||||
"# scripts/service-map.py render pihole-hosts > dankclaw/pihole/addn-hosts",
|
||||
"#",
|
||||
"# The source of truth is the `dns:` field of shared/services.yml; the",
|
||||
"# drift gate in tests/service_inventory.bats fails when this file and",
|
||||
"# the inventory disagree. Caddy is consolidated on Dankclaw, so every",
|
||||
"# name — including the services running on dankserver and",
|
||||
"# frankbook-pro — resolves at the Dankclaw edge.",
|
||||
"#",
|
||||
"# dnsmasq strips ONLY whole-line comments in addn-hosts: a trailing",
|
||||
"# comment on a data line becomes a HOSTNAME token. No inline notes",
|
||||
"# here; the service attribution lives in the inventory.",
|
||||
"",
|
||||
]
|
||||
for svc in inv.services:
|
||||
for name in svc.get("dns", []):
|
||||
lines.append(f"{edge} {name}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def scrape_facts(inv: Inventory) -> list[dict[str, str]]:
|
||||
facts = []
|
||||
for svc in inv.services:
|
||||
for job in svc.get("scrape", []):
|
||||
facts.append({"job": job["job"], "target": job["target"],
|
||||
"metrics_path": job.get("metrics_path", "/")})
|
||||
return facts
|
||||
|
||||
|
||||
def prometheus_facts(path: Path) -> list[dict[str, str]]:
|
||||
doc = yaml.safe_load(path.read_text())
|
||||
facts = []
|
||||
for job in doc.get("scrape_configs", []):
|
||||
for sc in job.get("static_configs", []):
|
||||
for target in sc.get("targets", []):
|
||||
facts.append({"job": job["job_name"], "target": target,
|
||||
"metrics_path": job.get("metrics_path", "/")})
|
||||
return facts
|
||||
|
||||
|
||||
def probe_facts(inv: Inventory) -> list[str]:
|
||||
facts: list[str] = []
|
||||
for svc in inv.services:
|
||||
probe = svc.get("probe")
|
||||
if isinstance(probe, dict):
|
||||
facts.extend(str(v) for v in probe.values())
|
||||
elif probe:
|
||||
facts.append(str(probe))
|
||||
return facts
|
||||
|
||||
|
||||
def homepage_probes(path: Path) -> list[str]:
|
||||
return sorted({s["probe"] for s in json.loads(path.read_text()) if s.get("probe")})
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# checks
|
||||
# --------------------------------------------------------------------------
|
||||
def check_ownership(inv: Inventory, graph: Graph) -> list[str]:
|
||||
"""Every repo file belongs to a service, a rule, or the unowned list."""
|
||||
problems: list[str] = []
|
||||
exact: set[str] = set()
|
||||
directories: set[str] = set()
|
||||
for svc in inv.services:
|
||||
for host in svc["hosts"]:
|
||||
for directory in inv.unit_dirs(svc, host):
|
||||
for unit in svc.get("units", []):
|
||||
exact.add(f"{directory}/{unit}.container")
|
||||
for field in ("files", "retires"):
|
||||
exact.update(svc.get(field, []))
|
||||
directories.update(svc.get("dirs", []))
|
||||
# A shared network file is claimed by the units that ride it; the unit
|
||||
# graph is the referrer, and check_units proves every reference resolves.
|
||||
exact.update(graph.networks.values())
|
||||
|
||||
def is_claimed(path: str) -> bool:
|
||||
if path in exact:
|
||||
return True
|
||||
first = PurePosixPath(path).parts[0]
|
||||
if first in NON_RUNTIME_ROOTS or path in NON_RUNTIME_PATHS or is_trash(path):
|
||||
return True
|
||||
if PurePosixPath(path).name in NON_RUNTIME_NAMES:
|
||||
return True
|
||||
for owned in directories | set(inv.unowned):
|
||||
if path == owned or path.startswith(owned + "/"):
|
||||
return True
|
||||
return False
|
||||
|
||||
for path in tracked_files():
|
||||
if not is_claimed(path):
|
||||
problems.append(f"file belongs to no service and is not in unowned: {path}")
|
||||
|
||||
# The reverse direction: an unowned entry that names nothing is stale.
|
||||
for directory in inv.unowned:
|
||||
if not any(p == directory or p.startswith(directory + "/") for p in tracked_files()):
|
||||
problems.append(f"unowned entry names no tracked file: {directory}")
|
||||
|
||||
# Every inventory file/dir claim names a tracked file.
|
||||
for svc in inv.services:
|
||||
# `retires` names a file whose DELETION selects the service, so by
|
||||
# definition it is not in the tree.
|
||||
for field in ("files", "dirs"):
|
||||
for entry in svc.get(field, []):
|
||||
if entry not in tracked_files() and not any(
|
||||
p.startswith(entry + "/") for p in tracked_files()):
|
||||
problems.append(f"{svc['name']}: {field} entry names no tracked file: {entry}")
|
||||
del graph
|
||||
return problems
|
||||
|
||||
|
||||
def check_units(inv: Inventory, graph: Graph) -> list[str]:
|
||||
"""Every unit file has exactly one owner; every unit reference resolves."""
|
||||
problems: list[str] = []
|
||||
owners: dict[str, list[str]] = {}
|
||||
for svc in inv.services:
|
||||
for host in svc["hosts"]:
|
||||
for directory in inv.unit_dirs(svc, host):
|
||||
for unit in svc.get("units", []):
|
||||
path = f"{directory}/{unit}.container"
|
||||
owners.setdefault(path, []).append(f"{svc['name']}@{host}")
|
||||
if path not in tracked_files():
|
||||
problems.append(f"{svc['name']}: unit file is not tracked: {path}")
|
||||
for path in tracked_files():
|
||||
if not path.endswith(".container") or is_trash(path):
|
||||
continue
|
||||
found = owners.get(path, [])
|
||||
if not found:
|
||||
problems.append(f"unit file claimed by no service: {path}")
|
||||
elif len(found) > 1:
|
||||
problems.append(f"unit file claimed by {len(found)} services: {path}: {found}")
|
||||
|
||||
# Root units (scripts/*.service) are claimed by exactly one `files:` list.
|
||||
for path in tracked_files():
|
||||
if not (path.startswith("scripts/") and path.endswith(".service")) or is_trash(path):
|
||||
continue
|
||||
found = [s["name"] for s in inv.services if path in s.get("files", [])]
|
||||
if not found:
|
||||
problems.append(f"root unit claimed by no service: {path}")
|
||||
elif len(found) > 1:
|
||||
problems.append(f"root unit claimed by {found}: {path}")
|
||||
|
||||
# Every Network= reference in a unit file names a file on the same host.
|
||||
for (host, unit), path in graph.units.items():
|
||||
for value in _network_values(REPO_ROOT / path):
|
||||
if value.endswith(".container"):
|
||||
target = (host, value[: -len(".container")])
|
||||
if target not in graph.units:
|
||||
problems.append(f"{path}: Network={value} names no unit on {host}")
|
||||
elif value.endswith(".network"):
|
||||
target = (host, value[: -len(".network")])
|
||||
if target not in graph.networks:
|
||||
problems.append(f"{path}: Network={value} names no network on {host}")
|
||||
return problems
|
||||
|
||||
|
||||
def check_config(inv: Inventory) -> list[str]:
|
||||
problems: list[str] = []
|
||||
for svc in inv.services:
|
||||
label = f"{'+'.join(svc['hosts'])}/{svc['name']}"
|
||||
for key in REQUIRED_KEYS:
|
||||
if key not in svc:
|
||||
problems.append(f"{label}: missing required key {key}")
|
||||
if svc.get("kind") not in KINDS:
|
||||
problems.append(f"{label}: kind {svc.get('kind')!r} not one of {sorted(KINDS)}")
|
||||
unknown = set(svc.get("hosts", [])) - set(inv.hosts)
|
||||
if unknown:
|
||||
problems.append(f"{label}: undeclared hosts: {sorted(unknown)}")
|
||||
if svc.get("units") and not svc.get("unit_dir"):
|
||||
problems.append(f"{label}: units declared without unit_dir")
|
||||
if svc.get("deployed_by") and not inv.any_named(svc["deployed_by"]):
|
||||
problems.append(f"{label}: deployed_by names no service: {svc['deployed_by']}")
|
||||
if (svc.get("health") and svc.get("wait") is None) or \
|
||||
(svc.get("wait") is not None and not svc.get("health")):
|
||||
problems.append(f"{label}: health and wait are a pair")
|
||||
for seam in svc.get("tests", []):
|
||||
if seam not in tracked_files():
|
||||
problems.append(f"{label}: seam does not exist: {seam}")
|
||||
for name in svc.get("dns", []):
|
||||
if not str(name).replace(".", "").replace("-", "").isalnum():
|
||||
problems.append(f"{label}: dns name is not a plain hostname: {name}")
|
||||
if svc.get("kind") == "root-unit" and svc.get("unit_dir"):
|
||||
problems.append(f"{label}: a root unit lives in scripts/, not unit_dir")
|
||||
seen: set[str] = set()
|
||||
for svc in inv.services:
|
||||
key = (svc["name"], tuple(svc["hosts"]))
|
||||
if key in seen:
|
||||
problems.append(f"duplicate inventory entry: {svc['name']} on {svc['hosts']}")
|
||||
seen.add(key)
|
||||
return problems
|
||||
|
||||
|
||||
# --- node-exporter quadlet template (rank 6) --------------------------------
|
||||
# Parallel per-host copies kept in step by a comment drift apart; the
|
||||
# template is the single source, and `check` drift-gates every tracked
|
||||
# rendering against it.
|
||||
NODE_EXPORTER_TEMPLATE = """\
|
||||
# Quadlet unit: node-exporter (rootless, boot-persistent) — RENDERED ARTIFACT.
|
||||
# Regenerate: python3 scripts/service-map.py render node-exporter-unit --host {host}
|
||||
# Source: shared/services.yml (entry node-exporter) + this template.
|
||||
#
|
||||
# Install path on {host}: ~/.config/containers/systemd/node-exporter.container
|
||||
# Unit name: node-exporter.service (container name: node-exporter)
|
||||
#
|
||||
# Host metrics source for Prometheus. Replaces the compose service of the same
|
||||
# name (the "Observability" block): Dankclaw moved to this quadlet on
|
||||
# 2026-09-11, Dankserver on 2026-09-16 — its compose-only copy died at a reboot
|
||||
# and stayed dead (rootless Podman does not restart bare containers after the
|
||||
# explicit stop shutdown performs), dead for a day and a half before anyone
|
||||
# noticed. One owner per container name; no compose file remains anywhere.
|
||||
#
|
||||
# HOST NAMESPACE. It must run in the host network namespace to report the
|
||||
# host's own interfaces/connections, and --pid=host for the processes and
|
||||
# cgroups collectors. "localhost" inside the container is therefore the host's
|
||||
# loopback, which is what the homepage status timer probes.
|
||||
#
|
||||
# LISTEN ADDRESSES. Exactly two: loopback for the homepage status probe, and
|
||||
# the LAN address, because Prometheus — which runs in a container on
|
||||
# dankclaw_default — scrapes http://{lan}:{port}/metrics
|
||||
# (shared/prometheus/prometheus.yml, job "node-{host}"). node_exporter has no
|
||||
# authentication, so the wildcard bind stays off the table; this is the
|
||||
# pairing that removes the unauthenticated observability surface on every
|
||||
# host, and both consumers are accounted for.
|
||||
#
|
||||
# MOUNTS. The container reads the host's /proc, /sys and / as /host/*
|
||||
# read-only and is pointed at them with --path.*; it writes nothing anywhere.
|
||||
|
||||
[Unit]
|
||||
Description=Prometheus node exporter ({title} host metrics)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=node-exporter
|
||||
Image=docker.io/prom/node-exporter:latest
|
||||
Network=host
|
||||
PodmanArgs=--pid=host
|
||||
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:{port} --web.listen-address={lan}:{port}
|
||||
Volume=/proc:/host/proc:ro
|
||||
Volume=/sys:/host/sys:ro
|
||||
Volume=/:/rootfs:ro
|
||||
HealthCmd=wget --spider -q http://localhost:{port}/metrics
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=30s
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
"""
|
||||
|
||||
|
||||
def render_node_exporter_unit(inv: Inventory, host: str) -> str:
|
||||
"""The node-exporter quadlet for one host: shared template, per-host
|
||||
title, LAN address from the hosts map, port from the scrape job."""
|
||||
svc = inv.service("node-exporter")
|
||||
if svc is None:
|
||||
raise SystemExit("inventory has no node-exporter entry")
|
||||
job = f"node-{host}"
|
||||
port = next((str(e["target"]).rsplit(":", 1)[1] for e in svc.get("scrape", [])
|
||||
if e.get("job") == job), None)
|
||||
if port is None:
|
||||
raise SystemExit(f"node-exporter scrape declares no job {job}")
|
||||
lan = inv.hosts[host]["address"]
|
||||
return NODE_EXPORTER_TEMPLATE.format(host=host, title=host.capitalize(),
|
||||
lan=lan, port=port)
|
||||
|
||||
|
||||
def check_drift(inv: Inventory) -> list[str]:
|
||||
problems: list[str] = []
|
||||
rendered = render_pihole_hosts(inv)
|
||||
hosts_file = REPO_ROOT / "dankclaw" / "pihole" / "addn-hosts"
|
||||
if hosts_file.read_text() != rendered:
|
||||
problems.append("dankclaw/pihole/addn-hosts is not the render of the inventory "
|
||||
"(scripts/service-map.py render pihole-hosts > dankclaw/pihole/addn-hosts)")
|
||||
|
||||
want = sorted(json.dumps(f, sort_keys=True) for f in scrape_facts(inv))
|
||||
have = sorted(json.dumps(f, sort_keys=True) for f in
|
||||
prometheus_facts(REPO_ROOT / "shared" / "prometheus" / "prometheus.yml"))
|
||||
for extra in [f for f in have if f not in want]:
|
||||
problems.append(f"prometheus scrapes a target the inventory does not declare: {extra}")
|
||||
for missing in [f for f in want if f not in have]:
|
||||
problems.append(f"inventory declares a scrape prometheus.yml lacks: {missing}")
|
||||
|
||||
svc = inv.service("node-exporter")
|
||||
for host in (svc or {}).get("hosts", []):
|
||||
for directory in inv.unit_dirs(svc, host):
|
||||
path = REPO_ROOT / directory / "node-exporter.container"
|
||||
if path.exists() and path.read_text() != render_node_exporter_unit(inv, host):
|
||||
problems.append(
|
||||
f"{directory}/node-exporter.container is not the render of the "
|
||||
f"template (scripts/service-map.py render node-exporter-unit --host {host})")
|
||||
|
||||
probes_want = sorted(set(probe_facts(inv)))
|
||||
probes_have = homepage_probes(REPO_ROOT / "shared" / "homepage" / "services.json")
|
||||
for extra in [p for p in probes_have if p not in probes_want]:
|
||||
problems.append(f"homepage probes a target the inventory does not declare: {extra}")
|
||||
for missing in [p for p in probes_want if p not in probes_have]:
|
||||
problems.append(f"inventory declares a probe homepage does not list: {missing}")
|
||||
return problems
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"config": lambda inv, graph: check_config(inv),
|
||||
"units": check_units,
|
||||
"ownership": check_ownership,
|
||||
"drift": lambda inv, graph: check_drift(inv),
|
||||
}
|
||||
|
||||
|
||||
def check(inv: Inventory, names: list[str]) -> list[str]:
|
||||
graph = graph_for(tuple(inv.hosts))
|
||||
problems: list[str] = []
|
||||
for name in names or list(CHECKS):
|
||||
problems.extend(CHECKS[name](inv, graph))
|
||||
return problems
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# CLI
|
||||
# --------------------------------------------------------------------------
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--inventory", type=Path, default=DEFAULT_INVENTORY)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
sub.add_parser("hosts")
|
||||
p = sub.add_parser("services")
|
||||
p.add_argument("--host")
|
||||
p = sub.add_parser("allows")
|
||||
p.add_argument("--host", required=True)
|
||||
p.add_argument("--service", required=True)
|
||||
p = sub.add_parser("plan")
|
||||
p.add_argument("--host", required=True)
|
||||
p.add_argument("paths", nargs="+")
|
||||
p = sub.add_parser("seams")
|
||||
p.add_argument("--host")
|
||||
p.add_argument("paths", nargs="+")
|
||||
p = sub.add_parser("owner")
|
||||
p.add_argument("--host", required=True)
|
||||
p.add_argument("path")
|
||||
p = sub.add_parser("units")
|
||||
p.add_argument("--host", required=True)
|
||||
p.add_argument("--service", required=True)
|
||||
p = sub.add_parser("stack")
|
||||
p.add_argument("--host", required=True)
|
||||
p.add_argument("--service", required=True)
|
||||
p = sub.add_parser("get")
|
||||
p.add_argument("--host")
|
||||
p.add_argument("--service", required=True)
|
||||
p.add_argument("field")
|
||||
p = sub.add_parser("render")
|
||||
p.add_argument("artifact", choices=("pihole-hosts", "node-exporter-unit"))
|
||||
p.add_argument("--host", choices=list(Hosts) if False else None)
|
||||
p = sub.add_parser("check")
|
||||
p.add_argument("names", nargs="*")
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
inv = Inventory.load(args.inventory)
|
||||
|
||||
if args.command == "hosts":
|
||||
print("\n".join(inv.hosts))
|
||||
elif args.command == "services":
|
||||
services = inv.on(args.host) if args.host else inv.services
|
||||
seen: list[str] = []
|
||||
for svc in services:
|
||||
if svc["name"] not in seen:
|
||||
seen.append(svc["name"])
|
||||
print("\n".join(seen))
|
||||
elif args.command == "allows":
|
||||
return 0 if inv.named(args.host, args.service) else 1
|
||||
elif args.command == "plan":
|
||||
print("\n".join(select(inv, args.host, args.paths)))
|
||||
elif args.command == "seams":
|
||||
if args.host is not None:
|
||||
seams = seams_for(inv, args.host, args.paths)
|
||||
else:
|
||||
# No host: the union across every host (the dev machine's
|
||||
# pre-commit mapper has no deployment target).
|
||||
seams = []
|
||||
for host in inv.hosts:
|
||||
for seam in seams_for(inv, host, args.paths):
|
||||
if seam not in seams:
|
||||
seams.append(seam)
|
||||
print("\n".join(seams))
|
||||
elif args.command == "owner":
|
||||
found = select(inv, args.host, [args.path])
|
||||
print(found[-1] if found else "")
|
||||
elif args.command == "units":
|
||||
matches = inv.named(args.host, args.service)
|
||||
if not matches:
|
||||
print(f"no service {args.service} on {args.host}", file=sys.stderr)
|
||||
return 2
|
||||
for svc in matches:
|
||||
for directory in inv.unit_dirs(svc, args.host):
|
||||
for unit in svc.get("units", []):
|
||||
print(f"{unit} {directory}")
|
||||
elif args.command == "stack":
|
||||
matches = inv.named(args.host, args.service)
|
||||
if not matches:
|
||||
print(f"no service {args.service} on {args.host}", file=sys.stderr)
|
||||
return 2
|
||||
owner = matches[0].get("deployed_by") or args.service
|
||||
print(owner)
|
||||
for svc in inv.on(args.host):
|
||||
if svc.get("deployed_by") == owner and svc["name"] != owner:
|
||||
print(svc["name"])
|
||||
elif args.command == "get":
|
||||
matches = inv.on(args.host) if args.host else inv.services
|
||||
matches = [s for s in matches if s["name"] == args.service]
|
||||
if not matches:
|
||||
print(f"no service {args.service}" + (f" on {args.host}" if args.host else ""),
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
values = [s.get(args.field) for s in matches if args.field in s]
|
||||
print(json.dumps(values[0] if len(values) == 1 else values))
|
||||
elif args.command == "render":
|
||||
if args.artifact == "pihole-hosts":
|
||||
sys.stdout.write(render_pihole_hosts(inv))
|
||||
else:
|
||||
if not args.host:
|
||||
print("render node-exporter-unit requires --host", file=sys.stderr)
|
||||
return 2
|
||||
sys.stdout.write(render_node_exporter_unit(inv, args.host))
|
||||
elif args.command == "check":
|
||||
problems = check(inv, args.names)
|
||||
if problems:
|
||||
for problem in problems:
|
||||
print(f"FAIL: {problem}", file=sys.stderr)
|
||||
return 1
|
||||
print("OK: the service inventory is sound (schema, units, ownership, drift)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -293,12 +293,12 @@ def main():
|
||||
ddt, dsh = meta[f"{pre0}.mlp.experts.0.down_proj.weight"][3:5]
|
||||
gsd, gssh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight_scale_inv"][3:5]
|
||||
dsd, dssh = meta[f"{pre0}.mlp.experts.0.down_proj.weight_scale_inv"][3:5]
|
||||
scale_law = (gdt == F8 and ddt == F8 and gsd == BF16 and dsd == BF16
|
||||
scale_contract = (gdt == F8 and ddt == F8 and gsd == BF16 and dsd == BF16
|
||||
and list(gssh) == [(gsh[0] + 127) // 128, (gsh[1] + 127) // 128]
|
||||
and list(dssh) == [(dsh[0] + 127) // 128, (dsh[1] + 127) // 128])
|
||||
if not scale_law:
|
||||
if not scale_contract:
|
||||
log(f"ERROR: unexpected expert shapes: gate {gdt}{gsh} scale {gsd}{gssh} "
|
||||
f"down {ddt}{dsh} scale {dsd}{dssh} (scale law 128x128)")
|
||||
f"down {ddt}{dsh} scale {dsd}{dssh} (fp8 block scales cover 128x128 tiles)")
|
||||
return 2
|
||||
shapes = {"ffn2_h": (gsh[0] * 2, gsh[1]), "ffn": (dsh[1],),
|
||||
"gs": tuple(gssh), "ds": tuple(dssh)}
|
||||
|
||||
+57
-190
@@ -62,234 +62,101 @@ declare -a TESTS=()
|
||||
add_test() {
|
||||
local test="$1"
|
||||
local existing
|
||||
MAPPED=1
|
||||
for existing in "${TESTS[@]:-}"; do
|
||||
[ "$existing" != "$test" ] || return 0
|
||||
done
|
||||
TESTS+=("$test")
|
||||
}
|
||||
|
||||
# The path->test mapping has two layers:
|
||||
# 1. cross-cutting seams that no single service owns (compose validity,
|
||||
# env consistency, the deployer itself, docs, syntax);
|
||||
# 2. the service inventory (shared/services.yml via service-map.py
|
||||
# "seams"): every service the path selects on ANY host contributes its
|
||||
# declared tests. Service seams are the services' own — the former
|
||||
# shotgun fallbacks (every dankclaw quadlet unit ran the open-webui,
|
||||
# pihole, and searxng seams) are retired; the inventory names each
|
||||
# service's real seams, and adding a service is a data edit.
|
||||
for path in "${CHANGED[@]}"; do
|
||||
MAPPED=0
|
||||
case "$path" in
|
||||
tests/*.bats)
|
||||
add_test "$path"
|
||||
continue
|
||||
;;
|
||||
tests/helpers.bash|tests/run.sh)
|
||||
die "$path is shared test infrastructure; run ./tests/run.sh explicitly"
|
||||
;;
|
||||
.githooks/*)
|
||||
add_test tests/changed_tests.bats
|
||||
;;
|
||||
dankserver/compose.yaml|dankclaw/compose.yaml)
|
||||
add_test tests/compose_validity.bats
|
||||
add_test tests/env_consistency.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
dankserver/Caddyfile|dankclaw/Caddyfile)
|
||||
add_test tests/caddyfile.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
tests/e2e/*)
|
||||
add_test tests/e2e_scaffold.bats
|
||||
continue
|
||||
;;
|
||||
.githooks/*)
|
||||
add_test tests/changed_tests.bats
|
||||
continue
|
||||
;;
|
||||
# Archive material and host-level facts: a path under any
|
||||
# .AGENTTRASH/ directory is retired-by-policy (the planner plans
|
||||
# nothing from it), udev rules are applied by manual sudo install,
|
||||
# reports are frozen artifacts, and ignore-lists change nothing at
|
||||
# runtime. They select no focused test; without this case they would
|
||||
# trip the mapping gate below.
|
||||
.AGENTTRASH/*|*/.AGENTTRASH/*|dankclaw/udev/*|report/*|.gitignore|.env.example)
|
||||
continue
|
||||
;;
|
||||
esac
|
||||
|
||||
# Cross-cutting seams.
|
||||
case "$path" in
|
||||
dankserver/.env.example|dankclaw/.env.example)
|
||||
add_test tests/env_consistency.bats
|
||||
;;
|
||||
shared/homepage/*)
|
||||
add_test tests/homepage_caddy.bats
|
||||
add_test tests/homepage_config.bats
|
||||
add_test tests/homepage_deploy.bats
|
||||
add_test tests/homepage_pages.bats
|
||||
add_test tests/homepage_checker.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
shared/searxng/*)
|
||||
add_test tests/searxng_limiter.bats
|
||||
add_test tests/searxng_quadlet.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# The five observability containers left compose for Quadlet units on
|
||||
# 2026-09-11. Their seam pins boot persistence and the loopback-only
|
||||
# publish posture that keeps them off every host interface. This case
|
||||
# precedes the dankclaw/*/quadlet/*.container fallback below, which
|
||||
# would otherwise swallow it into unrelated seams.
|
||||
dankclaw/observability/*)
|
||||
add_test tests/observability_quadlet.bats
|
||||
dankserver/Caddyfile|dankclaw/Caddyfile)
|
||||
add_test tests/caddyfile.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
# Caddy left compose.yaml on 2026-08-26 for a quadlet; caddyfile.bats
|
||||
# now asserts against the quadlet itself. Route every change under
|
||||
# dankclaw/caddy/ to both seams (this block must precede the
|
||||
# dankclaw/*.container glob, which would otherwise swallow the unit).
|
||||
dankclaw/caddy/*)
|
||||
add_test tests/caddy_quadlet.bats
|
||||
add_test tests/caddyfile.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
dankclaw/*.container|dankclaw/*.network)
|
||||
add_test tests/open_webui_quadlet.bats
|
||||
add_test tests/pihole.bats
|
||||
add_test tests/searxng_quadlet.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# Anubis is a quadlet + Caddyfile pair: the chain (public listener ->
|
||||
# anubis:8923 -> internal :8016 listener) spans both, so any change
|
||||
# under the service dir must re-run the caddyfile seam too. This case
|
||||
# precedes the dankclaw/*/quadlet/*.container fallback below, which
|
||||
# would otherwise swallow it into unrelated seams.
|
||||
dankclaw/anubis/*)
|
||||
add_test tests/anubis.bats
|
||||
add_test tests/caddyfile.bats
|
||||
add_test tests/homepage_caddy.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# frank-dragoon is a rootless quadlet again since 2026-09-16 (the
|
||||
# halogen-flash-server restore). Its unit carries the fleet's GPU and
|
||||
# podman posture, so it gets its own seam rather than the generic
|
||||
# fallback below.
|
||||
dankclaw/frank-dragoon/*)
|
||||
add_test tests/frank_dragoon_unit.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
# Quadlets live under dankclaw/<svc>/quadlet/ (reorganized out of the
|
||||
# top-level dankclaw/*.container globs above). Map the whole subtree so
|
||||
# changes to the pihole, open-webui, searxng, and gitea quadlets hit
|
||||
# their seam tests. The pihole-specific block must come first: shell
|
||||
# case matches the first pattern, and dankclaw/pihole/quadlet/*.container
|
||||
# also satisfies the dankclaw/*/quadlet/*.container fallback.
|
||||
dankclaw/pihole/*)
|
||||
add_test tests/pihole.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
dankclaw/*/quadlet/*.container|dankclaw/*/quadlet/*.network)
|
||||
add_test tests/open_webui_quadlet.bats
|
||||
add_test tests/pihole.bats
|
||||
add_test tests/searxng_quadlet.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
scripts/deploy.sh|scripts/deploy-remote.sh|scripts/deploy-plan.py|scripts/deploy-*-quadlet.sh)
|
||||
add_test tests/deployment_flow.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
scripts/service-map.py|shared/services.yml)
|
||||
add_test tests/service_inventory.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
scripts/lib/*.sh)
|
||||
add_test tests/llm_gates.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
serverAppConfigs-docs/*|README.md|AGENTS.md)
|
||||
add_test tests/docs_completeness.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
scripts/mount-entropy.sh)
|
||||
add_test tests/mount_entropy.bats
|
||||
;;
|
||||
# Archive material: a path under any .AGENTTRASH/ directory is
|
||||
# retired-by-policy (deploy-plan.py plans nothing from it), so it
|
||||
# selects no focused test. It still needs its own case: without one
|
||||
# the wrapper refuses retirement commits, whose trash moves show up
|
||||
# as additions, with "no focused-test mapping".
|
||||
.AGENTTRASH/*|*/.AGENTTRASH/*)
|
||||
;;
|
||||
# frank-archon is llama.cpp (the unsloth b11030-mix prebuilt) on the
|
||||
# two R9700s as a ROOT unit — the 2026-09-26 swap that replaced the
|
||||
# Lucebox DS4.1 quadlet on :7770. Its env, unit and serve script all
|
||||
# live in scripts/, which is why this case has to precede the generic
|
||||
# scripts/*.service and scripts/* fallbacks below.
|
||||
scripts/frank-archon.*|scripts/serve-frank-archon.sh)
|
||||
add_test tests/frank_archon_unit.bats
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
# frank-dragoon's chat template lives in scripts/ (moved there on
|
||||
# 2026-09-26 when its old home was retired): a template edit changes
|
||||
# what the live endpoint renders, so it selects the large unit seam.
|
||||
scripts/qwen-sharp-v22.4.0.jinja)
|
||||
add_test tests/frank_dragoon_unit.bats
|
||||
;;
|
||||
# frank-zealot is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||
# binary, so no generated quadlet is involved. Its env, unit and serve
|
||||
# script all live in scripts/, which is why this case has to precede
|
||||
# the generic scripts/*.service and scripts/* fallbacks below.
|
||||
scripts/frank-zealot.*|scripts/serve-frank-zealot.sh)
|
||||
add_test tests/frank_zealot_unit.bats
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
old/scripts/*)
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
scripts/*.service)
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
scripts/*)
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
serverAppConfigs-docs/*)
|
||||
add_test tests/docs_completeness.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
dankclaw/hermes/*)
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
README.md|AGENTS.md)
|
||||
# Project guidance changes can affect the documented deployment
|
||||
# contract; validate the docs vault and scoped deployment seams.
|
||||
add_test tests/docs_completeness.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
dankclaw/comfyui/*)
|
||||
add_test tests/docs_completeness.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# loki and alloy images are built from this repo's Containerfiles; the
|
||||
# loki one carries the healthcheck command (see its comment), so both
|
||||
# seams read it.
|
||||
dankclaw/loki/*|dankclaw/alloy/*)
|
||||
add_test tests/observability_quadlet.bats
|
||||
add_test tests/caddyfile.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
dankclaw/udev/*)
|
||||
# Host-level udev rules are applied by manual sudo install + udevadm
|
||||
# trigger, not dispatched as a service; nothing here is bats-testable.
|
||||
;;
|
||||
.AGENTTRASH/*)
|
||||
# Trash moves are not testable; skip.
|
||||
;;
|
||||
.gitignore|.env.example)
|
||||
# Ignore/ignore-list changes change nothing at runtime; not testable.
|
||||
;;
|
||||
old/scripts/*|old/scripts/**)
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
# Cutover-era dankserver quadlet units (romm trio, jellyfin, calibre-web,
|
||||
# copyparty, plex, tautulli): the seam checks boot-persistence +
|
||||
# v1-generator contract; deploy-flow seams guard the deployer routing.
|
||||
dankserver/romm/*|dankserver/jellyfin/*|dankserver/calibre-web/*|dankserver/copyparty/*|dankserver/plex/*|dankserver/tautulli/*)
|
||||
add_test tests/dankserver_quadlet.bats
|
||||
add_test tests/deployment_flow.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# FrankBook-Pro units live inside the podman machine VM; the seam
|
||||
# pins boot persistence, the gluetun netns invariant, and the
|
||||
# plan-locked volume layout. Deployer changes route here too.
|
||||
frankbook-pro/*)
|
||||
add_test tests/frankbook_quadlet.bats
|
||||
add_test tests/deployment_flow.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# Seerr unit on dankclaw: boot persistence, loopback-only port
|
||||
# posture, and the deployer routing that installs it.
|
||||
dankclaw/seerr/*)
|
||||
add_test tests/seerr_quadlet.bats
|
||||
add_test tests/deployment_flow.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
report/*)
|
||||
# Frozen audit reports are point-in-time artifacts; not testable.
|
||||
;;
|
||||
*)
|
||||
die "no focused-test mapping for '$path'; add one before committing"
|
||||
;;
|
||||
esac
|
||||
|
||||
# The inventory's declared seams for every service this path selects.
|
||||
while IFS= read -r seam; do
|
||||
[ -n "$seam" ] && add_test "$seam"
|
||||
done < <(python3 "$REPO_ROOT/scripts/service-map.py" seams -- "$path" 2>/dev/null || true)
|
||||
|
||||
# Mapping gate: a path that selected no test at all — no service seam,
|
||||
# no cross-cutting rule — is an unmapped area of the repo; refuse to
|
||||
# commit it silently (the old catch-all `die`).
|
||||
if [ "$MAPPED" -eq 0 ]; then
|
||||
die "no focused-test mapping for '$path'; map it in the inventory (shared/services.yml) before committing"
|
||||
fi
|
||||
done
|
||||
|
||||
# Mapped-but-untestable paths (udev rules, trash moves) legitimately produce an
|
||||
|
||||
@@ -1,104 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ttm-preflight.sh — the TTM pages-cap gate for engines that PIN host RAM.
|
||||
#
|
||||
# Sourced, never executed. An engine that pins tens of GiB of host RAM (the
|
||||
# radiance expert pools + the PLE n-gram table) has its pinned pages charged
|
||||
# against the kernel's ttm page cap: /sys/module/ttm/parameters/pages_limit
|
||||
# (and pages_max on kernels that expose one). The cap function below takes the
|
||||
# MIN over the knobs the running kernel actually exposes — an absent knob is
|
||||
# skipped, never read as zero, because a zero would refuse a healthy cap.
|
||||
#
|
||||
# THE LAW THIS FILE EXISTS FOR: the cap is a RUNTIME knob. The kernel default is
|
||||
# half of RAM, so every raise is lost on reboot. On 2026-10-12 23:17 a power
|
||||
# failure on dankserver reset it 200 GiB -> 125,988 MiB, and frank-archon's
|
||||
# preflight refused the start — 6,919 crash-restarts over ~20 hours, with
|
||||
# `Restart=always` cheerfully re-running a refusal. A manual sysfs write is
|
||||
# therefore not an acceptable repair procedure: the gate raises the knob itself
|
||||
# (the units that need it run as root, and the knob is a 0644 root-owned module
|
||||
# parameter), re-reads, and only then refuses.
|
||||
#
|
||||
# Contract (set by the sourcing script's env file):
|
||||
# FRANK_ARCHON_TTM_MIN_MIB floor the pinned ledger needs; refuse below it
|
||||
# FRANK_ARCHON_TTM_TARGET_MIB value to raise the cap to when it is below the
|
||||
# floor (the recorded host ceiling — writing the
|
||||
# floor itself would leave the ledger no slack)
|
||||
# FRANK_ARCHON_TTM_SYS_DIR knob directory; defaults to the real sysfs
|
||||
# path, a test points it at a temp dir.
|
||||
# Pages are 4 KiB: pages = MiB x 256.
|
||||
|
||||
: "${FRANK_ARCHON_TTM_SYS_DIR:=/sys/module/ttm/parameters}"
|
||||
|
||||
# Prints the effective cap in MiB, 0 when no knob is readable.
|
||||
ttm_cap_mib() {
|
||||
local p v cur=""
|
||||
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
|
||||
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
|
||||
[ -r "$p" ] || continue
|
||||
v="$(cat "$p" 2>/dev/null)" || continue
|
||||
[ -n "$v" ] || continue
|
||||
v=$(( v * 4 / 1024 ))
|
||||
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
|
||||
done
|
||||
echo "${cur:-0}"
|
||||
}
|
||||
|
||||
# Writes $1 MiB (as 4 KiB pages) into every knob present. Failures are
|
||||
# swallowed: the caller re-reads the cap, and a write the kernel refused shows
|
||||
# up there (ttm refuses a limit below the pages it has already pinned).
|
||||
#
|
||||
# PRIVILEGE, measured the hard way: the knob is root-owned 0644 and the service
|
||||
# process runs as User=dank, so `-w` is FALSE there and a plain echo fails. The
|
||||
# canonical privileged path is the unit's ExecStartPre with the `+` prefix
|
||||
# (systemd runs it as root even under User=); `sudo -n` is the fallback for a
|
||||
# human running the gate from a shell (passwordless sudo on both hosts).
|
||||
ttm_raise_cap() {
|
||||
local pages=$(( $1 * 256 )) p
|
||||
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
|
||||
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
|
||||
[ -e "$p" ] || continue
|
||||
if [ -w "$p" ]; then
|
||||
echo "$pages" > "$p" 2>/dev/null || true
|
||||
elif command -v sudo >/dev/null 2>&1; then
|
||||
sudo -n sh -c "echo $pages > '$p'" >/dev/null 2>&1 || true
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Returns 0 when the cap clears FRANK_ARCHON_TTM_MIN_MIB, raising it first if
|
||||
# needed; returns 1 with a named refusal when it cannot be made to.
|
||||
ttm_ensure_cap() {
|
||||
local cap
|
||||
cap="$(ttm_cap_mib)"
|
||||
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
||||
echo "frank-archon: ttm pages cap is ${cap} MiB, below the pinned-ledger" \
|
||||
"floor $FRANK_ARCHON_TTM_MIN_MIB — raising the knob to" \
|
||||
"$FRANK_ARCHON_TTM_TARGET_MIB MiB ($(( FRANK_ARCHON_TTM_TARGET_MIB * 256 )) pages)" >&2
|
||||
ttm_raise_cap "$FRANK_ARCHON_TTM_TARGET_MIB"
|
||||
cap="$(ttm_cap_mib)"
|
||||
fi
|
||||
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
||||
echo "ERROR: ttm pages cap is ${cap} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
||||
echo "frank-archon: raise $FRANK_ARCHON_TTM_SYS_DIR/pages_limit (and pages_max if the kernel exposes one) to >= $(( FRANK_ARCHON_TTM_MIN_MIB * 256 )) in 4 KiB pages; a read-only knob needs the module option (options ttm pages_limit=N) and a reboot" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "frank-archon: ttm pages cap ${cap} MiB clears the pinned-ledger floor $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
||||
}
|
||||
|
||||
# Executed (not sourced) it is the privileged pre-start hook: systemd runs this
|
||||
# as root from the unit's `ExecStartPre=+…`, where the knob write is legal and
|
||||
# the `Restart=always` loop can never outrun it.
|
||||
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
|
||||
set -euo pipefail
|
||||
case "${1:-}" in
|
||||
--raise)
|
||||
TTM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${FRANK_ARCHON_ENV_FILE:-$TTM_DIR/frank-archon.env}"
|
||||
ttm_ensure_cap
|
||||
;;
|
||||
*)
|
||||
echo "usage: ${0##*/} --raise (raise the ttm pages cap, then verify)" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
type: reference
|
||||
title: Service Inventory
|
||||
description: shared/services.yml is the one inventory of deployable services; scripts/service-map.py is its only reader. The planner, both deploy allow-lists, the changed-test seam mapper, the Pi-hole hosts render, the Prometheus scrape facts, and the homepage probes all derive from it.
|
||||
tags: [deployment, inventory, planner, quadlet, prometheus, pihole, homepage]
|
||||
timestamp: 2026-10-13T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Service Inventory
|
||||
|
||||
Before 2026-10-13 the same facts lived in four hand-maintained lists — the
|
||||
planner's path rules (`deploy-plan.py`), the `--service` allow-list in
|
||||
`deploy.sh`, the near-copy in `deploy-remote.sh`, and the changed-test mapper
|
||||
(`test-changed.sh`) — plus per-fact copies in `prometheus.yml`,
|
||||
`addn-hosts`, and `services.json`. They drifted on every swap (the planner
|
||||
still mapped the retired `qwen-sharp` template to frank-dragoon; the allow-list
|
||||
still listed the *arr stack under Dankserver after it moved to FrankBook-Pro).
|
||||
Now there is **one inventory and one reader**:
|
||||
|
||||
- `shared/services.yml` — the data: every deployable service, its hosts, unit
|
||||
dir(s), unit files, path claims, tests, scrape/DNS/probe facts, aliases.
|
||||
- `scripts/service-map.py` — the only reader. CLI subcommands:
|
||||
`hosts`, `services`, `allows`, `plan`, `seams`, `owner`, `units`, `stack`,
|
||||
`get`, `render`, `check`.
|
||||
|
||||
## 1. Consumers
|
||||
|
||||
|Consumer|Call|
|
||||
|---|---|
|
||||
|`scripts/deploy-plan.py`|thin wrapper → `plan --host TARGET [paths...]` (keeps the `--repo/--base/--head` and `--paths` CLIs)|
|
||||
|`scripts/deploy.sh` (`--service` gate)|`allows --host T --service S`|
|
||||
|`scripts/deploy-remote.sh` (`service_is_allowed`)|same|
|
||||
|`scripts/test-changed.sh`|`seams [paths...]` — a service's declared `tests` are its seams|
|
||||
|`dankclaw/pihole/addn-hosts`|`render pihole-hosts` — the tracked file is a rendered artifact|
|
||||
|`*/observability/quadlet/node-exporter.container`|`render node-exporter-unit --host H` — the MIRROR pair became one template; per-host title, LAN address (hosts map), and port (scrape job) fill it|
|
||||
|Prometheus scrape jobs, LAN DNS, homepage probes|`check` compares the tracked configs against the inventory's `scrape`/`dns`/`probe` fields (drift gate)|
|
||||
|
||||
Selection rules the engine encodes (see `tests/fixtures/service-plan-golden.tsv`,
|
||||
the pinned characterization of every tracked path × every target):
|
||||
|
||||
- **Host scoping** — a host's top-level directory can only select that host's
|
||||
services. `frankbook-pro/**` plans nothing on Dankserver (the old planner
|
||||
had a stale Dankserver *arr list).
|
||||
- **Quadlet coupling graph** — `Network=x.container` rides x's netns;
|
||||
`Network=y.network` joins a shared network. Changing the parent (or the
|
||||
network file) selects every rider transitively, in their own directories
|
||||
(`dankclaw/searxng/quadlet/gluetun.container` → gluetun + searxng + marreta;
|
||||
`frankbook-pro/gluetun/quadlet/gluetun.container` → the whole arr stack).
|
||||
- **Stacks** — a service's `stack` members deploy and restart as one handler
|
||||
(romm trio, hermes trio).
|
||||
- **Retirement** — removing a service from `services.yml` removes it from the
|
||||
planner, both allow-lists, and the seam mapper in one data edit; a pending
|
||||
deploy-state entry for it is auto-triaged (see [[Deploy State Triage]]).
|
||||
|
||||
## 2. Integrity gates (`check`)
|
||||
|
||||
`python3 scripts/service-map.py check` runs, and
|
||||
`tests/service_inventory.bats` pins:
|
||||
|
||||
- **schema** — required fields per entry, known keys only;
|
||||
- **units** — every unit file named by an entry exists under its `unit_dir`
|
||||
on the right host; each unit name is claimed by exactly one service per host;
|
||||
- **ownership** — every tracked quadlet `.container`/`.network` file is claimed
|
||||
by exactly one service (trash excluded);
|
||||
- **paths** — every claimed `files`/`dirs`/`retires` glob matches ≥1 tracked file;
|
||||
- **compose** — the container names in each host's `compose.yaml` have exactly
|
||||
one quadlet owner (the [[Quadlet Lifecycle]] two-owners trap);
|
||||
- **drift** — `addn-hosts` equals its render, each tracked node-exporter unit
|
||||
equals its per-host render; the Prometheus job set, the DNS names, and the
|
||||
homepage probe ports equal the inventory's fields.
|
||||
|
||||
## 3. Adding, moving, retiring a service
|
||||
|
||||
- **Add**: append an entry (name, hosts, kind, unit_dir, units, paths, tests,
|
||||
…), add the quadlet file, run `check` — `plan`/`allows`/`seams` pick it up
|
||||
with zero script edits.
|
||||
- **Move between hosts**: edit `hosts` (and per-host `unit_dir`/`probe` maps).
|
||||
- **Retire**: delete the entry, move files to `.AGENTTRASH/` (see
|
||||
[[Testing Policy]] for the seam the mapper applies to trash paths).
|
||||
|
||||
Field notes: `alias` carries deployable names that differ from the service
|
||||
name (`calibre-web` deploys the unit `calibre-server`); `probe` for multi-host
|
||||
services is a `{host: port}` map; `dependents` names services whose config
|
||||
embeds a reference to this one (config-only coupling — a `.env` edit selects
|
||||
both, a unit edit selects the graph closure).
|
||||
|
||||
## 4. Known cutover deltas (2026-10-13)
|
||||
|
||||
The 17 deltas against the old planner are all in these classes: anubis paths
|
||||
now select `anubis` (the old planner had no rule at all); cross-host paths
|
||||
select nothing; `scripts/qwen-sharp-v22.4.0.jinja` (retired template) plans
|
||||
nothing while `scripts/qwen38-froggeric-v22.5.jinja` (the live frank-archon
|
||||
template, previously unmapped) plans frank-archon;
|
||||
`dankserver/romm/romm.container` (top-level quadlet dir the old unit-file rule
|
||||
missed) now plans the romm trio.
|
||||
|
||||
---
|
||||
|
||||
Related: [[Deployment Flow]], [[Testing Policy]], [[Quadlet Lifecycle]],
|
||||
[[Container Network Namespaces]], [[Deploy State Triage]]
|
||||
@@ -15,10 +15,9 @@ This repository tests its configuration at defined seams. The tests run with `ba
|
||||
|
||||
| Seam | File | What it asserts |
|
||||
|------|------|-----------------|
|
||||
| Compose validity | `tests/compose_validity.bats` | YAML parses, services have images, no named volumes |
|
||||
| Env consistency | `tests/env_consistency.bats` | every `${VAR}` is defined in `.env.example`; no real secrets; the live gitea password never appears |
|
||||
| Caddyfile validity | `tests/caddyfile.bats` | per-host Caddyfile, `.local` convention, balanced braces, basic auth present |
|
||||
| Docs completeness | `tests/docs_completeness.bats` | every compose service has a docs page; index links every page |
|
||||
| Docs completeness | `tests/docs_completeness.bats` | every service has a docs page; index links every page; no unresolved vault links |
|
||||
| Script syntax | `tests/scripts_syntax.bats` | `bash -n` clean, executable bits, shellcheck when available |
|
||||
| Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
|
||||
| Quadlet units | `tests/*quadlet*.bats` | every unit is boot-persistent, self-healing, repo-provenanced, secret-free |
|
||||
|
||||
@@ -357,11 +357,15 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
|
||||
routing matches on hostname, verified 2026-09-02).
|
||||
`internal_host = http://caddy:8017`, all three flows set, and an
|
||||
Application attached — the two provisioning rules above.
|
||||
2. **Route flip in ONE commit/deploy**: the vhost becomes
|
||||
`reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
|
||||
inside the `http://:8017` listener add `@<svc> host <vhost>` +
|
||||
2. **Route flip in ONE commit/deploy**: the vhost body becomes
|
||||
`import guarded` — the named snippet at the top of the Caddyfile is the
|
||||
whole pre-auth chain (`reverse_proxy anubis:8923` + `header_up X-Real-Ip
|
||||
{remote_host}`; rank 6 retired the nine verbatim copies), and inside the
|
||||
`http://:8017` listener add `@<svc> host <vhost>` +
|
||||
`handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
|
||||
edge, `:8017` routing table, and provider list change together.
|
||||
`tests/caddyfile.bats` pins that the chain exists exactly once as the
|
||||
`(guarded)` snippet and every guarded vhost imports it.
|
||||
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
|
||||
|
||||
**Verification (all three):**
|
||||
@@ -452,8 +456,7 @@ access is decided entirely by GROUP MEMBERSHIP consumed downstream —
|
||||
- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
|
||||
every human account).
|
||||
- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
|
||||
debo, freya, mitch, eric — mitch (pk 22) and eric (pk 23) created
|
||||
2026-10-15, kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
|
||||
debo, freya — kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
|
||||
and alex created 2026-10-08, artem (pk 16) created 2026-10-07): `username` +
|
||||
`<user>@franksdank.cloud` + display name +
|
||||
`is_active=True` + the password set via `set_password`, membership in
|
||||
|
||||
@@ -1,144 +0,0 @@
|
||||
# frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, 2× R9700 TP=2
|
||||
|
||||
**Status: LIVE — deployed 2026-10-15.** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
|
||||
|
||||
The in-house w4dn8 conversion of the Swift1.5 finetune
|
||||
([[frank-archon-radiance-swift15-w4dn8]]) is replaced by the **engine
|
||||
author's published container**:
|
||||
[`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`](https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe)
|
||||
— the official `Qwen/Qwen3.8-Flash-Next` bf16 checkpoint rad-converted by
|
||||
StillDeadcode. Same port, alias, container name, systemd unit, digest-pinned
|
||||
engine pin **1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…`), and
|
||||
**every serving argument carried verbatim** from the 2026-10-13 shape.
|
||||
Rollback: revert the swap commit — the w4dn8 `.rad` stays on disk.
|
||||
|
||||
## The payload
|
||||
|
||||
`qwen3.8-next-flash-fp8-iq4r-moe.rad` — **121,969,901,568 B = 116,322 MiB**,
|
||||
sha256 `0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20`
|
||||
(Hub LFS oid), fetched byte-exact by `scripts/download-frank-archon-iq4r.sh`
|
||||
(hfget resume + size + sha256; staging dir
|
||||
`/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe/` keeps the
|
||||
card, LICENSE, and recipe `qwen4exp-w4nl64-i8-hc8m` as provenance).
|
||||
|
||||
| Plane | Encoding (publisher recipe `qwen4exp-w4nl64-i8-hc8m`) |
|
||||
| --- | --- |
|
||||
| Routed experts | 4-bit codes into a 16-level **non-uniform** codebook after a 128-point Walsh–Hadamard rotation, one E4M3 scale per 64 weights; codes by GPTQ (10M-token calibration, per-expert down-proj Hessians) + 3 coordinate-descent sweeps |
|
||||
| Protected experts | **ten** per layer, whole-expert bf16 (down-projection energy) |
|
||||
| Trunk + lm_head | int8 W8A8, scale per 128 columns searched; HC mixtures E4M3 |
|
||||
| PLE n-gram | fp8 fixed-scale, in-container (served `--ngram-placement ram`, same 47.7 GiB pin law) |
|
||||
| Speculator | the model's own MTP head (card default depth 3; served at 2) |
|
||||
| Vision | bf16 vision tower, in-container |
|
||||
|
||||
**Base model: the OFFICIAL (censored) Qwen3.8-Flash-Next** — the fleet's
|
||||
only frontier model at base alignment; the abliterated family lives on
|
||||
frank-dragoon. Card quality: full-vocab teacher-forced KL vs bf16 over
|
||||
~75K positions — mean 0.0789 all-tokens / 0.0239 assistant turns, top-1
|
||||
agreement 92.85 %.
|
||||
|
||||
## Why this swap
|
||||
|
||||
- **Publisher-graded precision**: the w4nl codebook + coordinate-descent
|
||||
refinement is the engine author's own quality ceiling for this
|
||||
checkpoint; the card's KL ladder (p99 1.65, p99.9 5.33) is published and
|
||||
reproducible, where our w4dn8 gate measured against our own reference.
|
||||
- **Formal parity**: the served file is the one the engine author supports
|
||||
— decode/prefill behavior on 1.0.13 is their tested path (the card's run
|
||||
line speaks the 1.0.13-era flags; the `--vram-weights-mib`/`--vram-kv-mib`
|
||||
pair we ship is accepted by pin 1.0.13 per the 2026-10-06 A/B, so the
|
||||
shape needed no translation).
|
||||
- **Cost**: the Swift1.5 finetune's refusal-bending behavior leaves this
|
||||
endpoint (it returns to the official base alignment); the 4-slot/200k
|
||||
economics and all shape laws carry over unchanged.
|
||||
|
||||
## Shape (unchanged from 2026-10-13)
|
||||
|
||||
All of it, verbatim — the payload moved 116,416 → 116,322 MiB, inside the
|
||||
same budget:
|
||||
|
||||
- `--tp 2 --tp-wire wht6`, `--max-num-seqs 3`, `--max-model-len 200000`
|
||||
(capacity contract 3 × 200,000), `--kv-cache-dtype fp8`
|
||||
- `--placement expert_tiered --host-pool-mib 44928` + explicit
|
||||
`--vram-weights-mib 25600 --vram-kv-mib 5120` (30,720 of the engine's
|
||||
30,969 MiB ceiling; pool = contract + 1.3 %)
|
||||
- `--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`
|
||||
(MNBT 2048 law), prefix cache 8 GiB host + 64 GiB disk,
|
||||
`--ngram-placement ram`, froggeric v22.5 `--override-chat-template`
|
||||
- TTM preflight floor 150,000 / target 204,800 MiB; pinned ledger
|
||||
2 × 44 GiB pool + 47.7 GiB PLE + 8 GiB prefix ≈ 147 GiB (see
|
||||
[[frank-archon-radiance-swift15-w4dn8]] for the ledger arithmetic and
|
||||
the KV dtype law — both are engine properties, payload-independent)
|
||||
|
||||
The publisher's card run line (`--host-pool-mib 12288`,
|
||||
`--expert-vs-cache-ratio 0.82`, seqs 8, spec 3, ~32 GB host RAM) is their
|
||||
shape on stock 32 GB cards; the fleet ships its own measured shape against
|
||||
the same file.
|
||||
|
||||
## Cutover record — 2026-10-15
|
||||
|
||||
1. `rm` (Frank's one-time authorization): the raw
|
||||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` BF16 checkpoint (336 GiB, spent
|
||||
conversion source) and `qwen4exp-fp8-native.rad` (173 GiB, two eras
|
||||
back) left `/home/dank/models` — 150 GB free → 658 GB free. The fp8-native
|
||||
payload is **no longer on disk**: rebuild =
|
||||
`scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`.
|
||||
2. Fetch: `scripts/download-frank-archon-iq4r.sh --fetch` as transient unit
|
||||
`iq4r-fetch` (detached-job protocol; log
|
||||
`/home/dank/logs/iq4r-fetch.log`, rc sentinel `…rc`).
|
||||
3. Deploy: `scripts/deploy.sh dankserver --service frank-archon` — launched
|
||||
detached (the swap restarts the very endpoint the operator session runs
|
||||
on; the harness timeout catches the session on return).
|
||||
4. **Boot ledger (2026-10-15, first boot, pin 1.0.13 — the w4nl codebook
|
||||
loaded, no pin bump)**: `kv_attn` fp8 page **26.00 KiB** at **607,740
|
||||
tokens** (151,935 blocks = 3.77 GiB) — byte-identical to the w4dn8 boot,
|
||||
the sizing law carries; `kv_qsa_bkey` 3.25 KiB at the same token count;
|
||||
`kv_gdn_state` 54 MiB × 6 slots, `kv_gdn_conv` 1.76 MiB × 3,
|
||||
`kv_qsa_tail` 19.5 KiB × 6; pool **5.00 GiB**, prefix checkpoints 8 ×
|
||||
56 MiB; the sizing line reads `kv cache 5.00 GiB … backs 3 sequences all
|
||||
at 200000 tokens at once, which is the worst case`; samplers print **9
|
||||
rows** per rank (3 × (1+2), inside the wht6 25-row bound);
|
||||
`speculation: 'mtp' drafts 2 token(s) a step in 2 serial pass(es)` +
|
||||
prompt-lookup fallback; slab **17,610 (rank0) / 17,658 (rank1) slots**
|
||||
(64 shadow) — above the w4dn8 record of 17,548 (ten bf16-protected
|
||||
experts per layer replace twenty-four, so more of the 4-bit table
|
||||
resides), pinned pool 7,532/7,484 slots; KV lending **4.22 GiB**
|
||||
(3,571/3,568 mover slots); arenas 896.02/988.17 MiB charged before the
|
||||
pair split; prefix cache RECOVERED 900,176 blocks + 126 checkpoint slots
|
||||
from the 64 GiB disk tier (same arch + tokenizer as w4dn8 — the tier
|
||||
survived the payload swap); froggeric template 28,234 B; vision encoder
|
||||
live (≤ 4,194,304 px, 16,384 patches/pass). Health gate passed in ~100 s
|
||||
(radiance binds the port after the full load).
|
||||
5. **Vision probe (2026-10-15)**: 300 px solid-red PNG via `image_url` —
|
||||
reasoning stream: "a solid fill of a deep, saturated red hue". Content
|
||||
came back EMPTY at `max_tokens 64`: the thinking budget consumed all 64
|
||||
(`reasoning_tokens: 64`, finish `length`) — fleet law, probe thinking
|
||||
models with ≥ a few hundred tokens.
|
||||
6. **e2e gate**: deploy exited DEPLOY INCOMPLETE — 30/33 Playwright green,
|
||||
3 red (`12-crawl`, `13-sso` search SSO, `16-marreta`) on DANKCLAW
|
||||
surfaces (vhosts intermittently returned `ERR_CONNECTION_RESET` from the
|
||||
workstation); frank-archon's own health gate passed. The swap touched no
|
||||
dankclaw runtime file (only homepage services.json text). Census of the
|
||||
Caddy/auth stack pending; see `log.md`.
|
||||
7. **Post-boot measurements (pending)**: decode ladder (99.0 t/s single
|
||||
@ 43.5k, 93.3 @ 197k, 3-way 113.9/43.3/10.9 = 18.4 t/s aggregate —
|
||||
these are the **w4dn8** numbers, the record to beat, not this file's),
|
||||
slab-limited prefill (22208/3,397 → 25600/3,584 t/s A-cold 43.5k), draft
|
||||
acceptance, and `scripts/probe-frank-archon-kv.py` leg D.
|
||||
|
||||
## Rollback
|
||||
|
||||
Revert the swap commit (`scripts/frank-archon.env` MODEL_FILE line) +
|
||||
redeploy: `/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad`
|
||||
(116,416 MiB) stays on disk, KLD-gated, same shape. The fp8-native era is
|
||||
rebuild-only (see the cutover record). The w4dn8 runbook keeps every shape
|
||||
law, the KV dtype table, the slab ladder, and the tuning levers:
|
||||
[[frank-archon-radiance-swift15-w4dn8]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — the replaced payload and the
|
||||
full shape-law library (slab slope, KV dtype arithmetic, TTM law)
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — the fp8-native era
|
||||
(rebuild-only now; owns the descriptor-selection law)
|
||||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|
||||
- [[deployment]] — the service-scoped deploy this cutover rides
|
||||
@@ -145,18 +145,11 @@ dtype`) so a future resize cannot silently under-provision the pool.
|
||||
|
||||
### Why the weights slab pays — and why that is not pure loss
|
||||
|
||||
The two VRAM budgets are one budget, and the ceiling is the engine's own — not a
|
||||
hand-picked reserve. Card 32,624 MiB, and the table that prints at the pair
|
||||
24,000/5,120 says `3.42 GiB is neither budgeted nor accounted — the driver's
|
||||
context, the activation arena and --gpu-headroom-mib come out of it; anything
|
||||
past those is throughput on the floor`, with `card 0 settled at 2.01 GiB free
|
||||
against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the
|
||||
static region)`. Claimable reserve above the ENGINE's floor:
|
||||
2,058 − 209 = 1,849 MiB ⇒ **`weights + kv ≤ 30,969 MiB`**. (The 29,120 bound
|
||||
written on 2026-10-11 was my own 1 GiB "keep free" reserve, measured off a card
|
||||
figure of 31,856 MiB; the engine never asked for that reserve and the seam now
|
||||
derives the number from its table.) Paying for a 6,912 MiB pool therefore cuts
|
||||
the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
|
||||
The two VRAM budgets are one budget: card 31,856 MiB, minus the 1,710 MiB a
|
||||
boot consumes outside both (driver context, the 990 MiB activation arena,
|
||||
hardware queues, headroom, step staging), minus 1,024 MiB kept free ⇒
|
||||
**`weights + kv ≤ 29,120 MiB`**. Paying for a 6,912 MiB pool therefore cuts the
|
||||
resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
|
||||
into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
|
||||
slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
|
||||
floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).
|
||||
@@ -260,240 +253,6 @@ from the scored configuration — bf16 can only reduce the attention-state
|
||||
error, and a re-score is a drained-window job if the gate agreement is ever
|
||||
needed in the record.
|
||||
|
||||
## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV
|
||||
|
||||
### The outage: a refusal, not a crash
|
||||
|
||||
Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
|
||||
never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
|
||||
after starting, with one journal line on repeat:
|
||||
|
||||
```text
|
||||
ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
|
||||
frank-archon: raise /sys/module/ttm/parameters/pages_limit …
|
||||
```
|
||||
|
||||
**6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
|
||||
the name with a zero start time), so there was no engine, no port, and no engine
|
||||
log — the signature of a preflight refusal, which every deployer/monitor view
|
||||
outside the journal renders as an unexplained crash loop.
|
||||
|
||||
The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
|
||||
expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
|
||||
shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
|
||||
runtime state, and its default is half of RAM** — so the manual raise of
|
||||
2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
|
||||
fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
|
||||
script enforces. The fp8-native era had planned the persistent half of this
|
||||
("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
|
||||
reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
|
||||
a power failure is not scheduled.
|
||||
|
||||
### The repair: the gate raises the knob itself
|
||||
|
||||
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
|
||||
the MIN over the knobs the running kernel exposes, and when it is below
|
||||
`FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
|
||||
52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
|
||||
The unit is root and the knob is a 0644 root-owned module parameter, so the write
|
||||
is legal; a kernel that refuses the move produces the same named refusal as
|
||||
before, and a reboot is now a boot rather than a visit to the machine.
|
||||
|
||||
| Knob | Power-failure default | After the gate |
|
||||
|---|---|---|
|
||||
| `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |
|
||||
|
||||
Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
|
||||
refuses when the write fails* and *the ttm cap counts only the knobs the kernel
|
||||
exposes* drive the gate against a temp knob directory — a below-floor writable
|
||||
knob ends at the target and the start passes; a read-only knob leaves the cap
|
||||
untouched and the start refuses by name; an absent `pages_max` is skipped, never
|
||||
read as zero (a zero reading would refuse a healthy host).
|
||||
|
||||
### The shape: 4 slots × 200,000, fp8 KV (same day, superseded by the re-cuts below)
|
||||
|
||||
Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
|
||||
concurrent 200,000-token conversations with quantized KV.
|
||||
|
||||
fp8 is not a preference at four slots, it is the only dtype that affords them.
|
||||
The pool must back `seqs × max-model-len` (the engine's hard limit), so the
|
||||
fourth slot is bought in KV, out of the one per-card budget
|
||||
(`weights + kv ≤ 29,120 MiB`, the bound as it was understood then):
|
||||
|
||||
| dtype | pool for 4 × 200,000 | weights slab left |
|
||||
|---|---|---|
|
||||
| bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
|
||||
| **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |
|
||||
|
||||
So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
|
||||
2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
|
||||
44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
|
||||
move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
|
||||
**843,566 predicted tokens** = four full-height conversations with 5.4 % slack
|
||||
(the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
|
||||
1.93).
|
||||
|
||||
Terms that move with the dtype and the slot count:
|
||||
|
||||
- **Per-slot recurrent state** is dtype-independent and lives inside the pool:
|
||||
+110 MiB per slot, 440 MiB for four (already in the arithmetic above).
|
||||
- **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
|
||||
**12 rows**, still inside the 25-row bound at which the `wht6` wire is
|
||||
bit-exact — so four slots cost nothing in wire exactness.
|
||||
- **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
|
||||
58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
|
||||
bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
|
||||
- **`models.yml` is untouched**: `contextWindow` quotes the per-request window
|
||||
(200,000), which did not change; only the concurrency did.
|
||||
|
||||
Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
|
||||
(same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
|
||||
cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
|
||||
~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
|
||||
26,071 MiB slab and does not come back by quantizing KV.
|
||||
|
||||
### Boot ledger (2026-10-13 19:21, four slots, fp8 — superseded same evening)
|
||||
|
||||
The sizing law's prediction, checked against the engine's own arithmetic:
|
||||
predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
|
||||
|
||||
| Group | Page / state | Size | Tokens |
|
||||
|---|---|---|---|
|
||||
| `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
|
||||
| `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
|
||||
| `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
|
||||
| `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |
|
||||
|
||||
`kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
|
||||
the worst case` — the capacity contract, in the engine's words. The rest of the
|
||||
boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
|
||||
/ 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
|
||||
2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
|
||||
4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
|
||||
lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).
|
||||
|
||||
Two ledger lines that confirm the derived consequences: the sampler prints
|
||||
**12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
|
||||
`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
|
||||
(58.50 KiB, half of the bf16 entry).
|
||||
|
||||
### The same-day re-cuts: 4 → 3 slots, then the slab takes the card's slack
|
||||
|
||||
Frank's call, twice in one evening. Both are one-env-pair changes, and both were
|
||||
read off the boot ledger rather than assumed.
|
||||
|
||||
**4 → 3 slots (22208/6912 → 24000/5120).** Three slots need
|
||||
3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB** of pool; the 6,912 pool
|
||||
was backing 858,969 tokens — 43 % more window than the contract asks for.
|
||||
Cutting it to 5,120 (predicts 608,027; ledger **607,740**, 0.05 %) hands
|
||||
1,792 MiB to the slab and drops the all-reduce to 9 rows.
|
||||
|
||||
**24,000 → 25,600 (the card's settling slack).** The 24,000 boot printed
|
||||
`card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
|
||||
--gpu-headroom-mib and 113.19 MiB kept for the static region)` — 1,849 MiB above
|
||||
the ENGINE's own floor, with the budget table calling the surplus `throughput on
|
||||
the floor`. Claiming 1,600 MiB of it puts the pair at 30,720 and the ledger at
|
||||
`unclaimed 477.30 MiB` (cards ~336 MiB free, floor 209.19).
|
||||
|
||||
| pair | slab slots (rank 0) | pinned pool slots | A-cold 43.5k | concurrent 197k |
|
||||
|---|---|---|---|---|
|
||||
| 22208/6912, seqs 4 | 14,753 | 10,375 | 3,397 t/s | 3,307 t/s (4-way) |
|
||||
| 24000/5120, seqs 3 | 16,235 | 8,893 | 3,290 t/s | 3,051 t/s (3-way) |
|
||||
| **25600/5120, seqs 3** | **17,548** | 7,580 | **3,584 t/s** | **3,589 t/s** (3-way) |
|
||||
| 26071/3543, seqs 3 (2026-10-11 control) | — | 7,190 | 4,379 t/s | — |
|
||||
|
||||
The claim shipped with the 25,600 arm — that the slab is a THRESHOLD below
|
||||
~26 GiB so the extra 1.6 GiB buys ~0 — is **falsified by its own measurement**:
|
||||
+8.9 % single-stream, +17.6 % concurrent, both outside the ±3 % noise floor. Two
|
||||
slab points that land inside noise of *each other* (22,208 vs 24,000) are not a
|
||||
plateau; the pinned host pool is the slow path, so every MiB moved from RAM onto
|
||||
the card is throughput (the ledger shows the transfer directly: slab
|
||||
+1,313 slots, pinned pool −1,313).
|
||||
|
||||
The cliff is real, it just sits above 25,600: 26,071 measured 4,379 t/s, +22 %
|
||||
for +471 MiB. At three slots that is out of reach (26,112 + 5,120 = 31,232 > the
|
||||
30,969 ceiling). At **two** slots the pool falls to 3,072 and the slab reaches
|
||||
**27,897** — the ceiling exactly, slab above the cliff. That arm is untested and
|
||||
it is a capacity decision, not a tuning one.
|
||||
|
||||
Capacity contract, live-verified on the 3-slot shape: three concurrent prompts
|
||||
of **199,984 tokens** each (599,952 tokens) were accepted and completed on
|
||||
`/v1/completions`; a fourth full-height request is refused as "too large to ever
|
||||
fit" — radiance never truncates.
|
||||
|
||||
### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill
|
||||
|
||||
Frank's challenge — "why are you not tracking decode?" — was right, and the
|
||||
answer is a number now. 512 generated tokens (the bench law), greedy, cold
|
||||
prompts, this shape (3 x 200,000, fp8, slab 25,600):
|
||||
|
||||
```
|
||||
1-stream 43.5k: decode 99.0 t/s (prefill 4,042, accept 0.952)
|
||||
1-stream 197k: decode 93.3 t/s (prefill 4,826, accept 0.938)
|
||||
3-stream 43.5k: decode 113.9 / 43.3 / 10.9 t/s wall 83.4s
|
||||
3-stream 197k: decode 88.9 / 16.3 / 64.6 t/s wall 122.4s
|
||||
```
|
||||
|
||||
Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
|
||||
93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
|
||||
prefill, not tokens-per-second. **Concurrency is where the slot count is
|
||||
paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
|
||||
at 197k against 99 single-stream, and the split is a lottery
|
||||
(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
|
||||
to whoever it reached first, so one stream finishes at 113 t/s while its
|
||||
sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
|
||||
number shows it.
|
||||
|
||||
Corpus caveat, stated at the claim: this probe's filler is a repetitive
|
||||
ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
|
||||
table below; these rows flatter decode relative to that table and are the
|
||||
baseline for *this* corpus, not a replacement for those rows.
|
||||
`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
|
||||
tokens, which is why a day of shape work priced only prefill).
|
||||
|
||||
### Recovery evidence
|
||||
|
||||
```text
|
||||
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
|
||||
is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
|
||||
204800 MiB (52428800 pages)
|
||||
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
|
||||
204800 MiB clears the pinned-ledger floor 150000
|
||||
Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
|
||||
```
|
||||
|
||||
`cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
|
||||
the boot, i.e. the root `ExecStartPre` hook did the write the service process
|
||||
(User=dank) cannot.
|
||||
|
||||
### Capacity probe (2026-10-13, live service)
|
||||
|
||||
`scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype
|
||||
from the deployed env on the host, so the probe cannot drift from the service:
|
||||
it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats
|
||||
of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each
|
||||
carrying a marker whose first word is unique.
|
||||
|
||||
| Stream | prompt | wall | prefill | retrieval |
|
||||
|---|---|---|---|---|
|
||||
| B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT |
|
||||
| B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT |
|
||||
| B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT |
|
||||
| B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT |
|
||||
| **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 |
|
||||
|
||||
All four answered their own marker; nothing was preempted, refused, or truncated
|
||||
(`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate
|
||||
prefill equals the single-stream rate measured in the same probe (3,397 t/s at
|
||||
43.5k), i.e. the four streams share one prefill pipeline at the slab-limited
|
||||
ceiling and the engine never ran short of KV. Prefix cache intact: the same
|
||||
43.5k prompt repeated answered in 1.0 s with `cache_n=43008`.
|
||||
|
||||
The per-stream `decode` column is NOT a decode rate here — 8 generated tokens
|
||||
make it first-token dominated (the bench law: generate ≥ 128 tokens to read a
|
||||
rate). This probe answers one question only: does the endpoint owe four
|
||||
full-height conversations at once, and answer each one exactly. It does.
|
||||
|
||||
## Gates (measured at cutover)
|
||||
|
||||
- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
|
||||
|
||||
@@ -27,7 +27,7 @@ conversation slot and unquantized KV back.
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-zealot` name), `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`); `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
|
||||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=always` (raised from `on-failure` 2026-10-13 for fleet parity — exit 0 is a CLEAN stop to systemd, so the weaker policy leaves the compaction hub dark while the unit reports success; `tests/llm_units.bats` gates it), `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
|
||||
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
|
||||
| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
|
||||
|
||||
@@ -2,18 +2,14 @@
|
||||
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
|
||||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||||
driving `podman run` directly, root systemd unit, :7770):
|
||||
the **publisher's** 4-bit-codebook container of the official
|
||||
`Qwen/Qwen3.8-Flash-Next` (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||||
sha256-verified at fetch — w4nl codebook experts + int8 trunk, PLE n-gram,
|
||||
MTP head and vision tower in-container), on BOTH R9700s
|
||||
TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
|
||||
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
|
||||
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
|
||||
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
|
||||
froggeric v22.5 template. Rollback: the in-house Swift1.5 w4dn8 `.rad` (on
|
||||
disk); tags frank-archon-freetoken-pr587-20261005 and
|
||||
frank-archon-vllm-shapeG-20261003),
|
||||
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
|
||||
(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
|
||||
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
|
||||
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||||
2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005
|
||||
and frank-archon-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
|
||||
@@ -40,10 +36,10 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[radiance StillDeadcode, digest-pinned image
|
||||
podman run, frank-archon, port 7770]
|
||||
C[Publisher qwen3.8-next-flash-fp8-iq4r-moe .rad
|
||||
2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
|
||||
3 x 200000, froggeric v22.5]
|
||||
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
|
||||
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
2 x 200000, froggeric v22.5]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -156,42 +152,15 @@ frank-zealot, port 7771]
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE since 2026-10-15**:
|
||||
the engine author's published container of the OFFICIAL (base-alignment)
|
||||
Qwen3.8-Flash-Next — `qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||||
Hub-sha256-verified at fetch: 4-bit codes into a 16-level non-uniform
|
||||
codebook after a Walsh–Hadamard rotation (GPTQ + 3 coordinate-descent
|
||||
sweeps, 10M-token calibration), ten bf16-protected experts per layer,
|
||||
int8 trunk/lm_head, fp8 HC + PLE, MTP head, bf16 vision tower; card KL
|
||||
0.0789 all-tokens / top-1 92.85 % vs bf16. **MODEL_FILE is the only line
|
||||
that moved** — the 2026-10-13 shape (25600/5120 pair, 44928 pool, fp8 KV
|
||||
3 × 200,000, MTP 2, wht6, MNBT 2048, froggeric v22.5) carries verbatim.
|
||||
Rollback: revert the swap commit (w4dn8 `.rad` stays on disk). The
|
||||
fp8-native `.rad` and the raw Swift1.5 BF16 source were DELETED from
|
||||
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
|
||||
rebuild via the fetch + build scripts named in the fp8-native runbook)
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
|
||||
(superseded by the publisher iq4r swap; payload on disk as the rollback)**:
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
|
||||
pair re-cut twice** — `--kv-cache-dtype fp8` first at
|
||||
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
|
||||
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
|
||||
raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
|
||||
settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
|
||||
1 GiB reserve invented. fp8 is what funds the slots — the pool must back
|
||||
`seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
|
||||
Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
|
||||
**25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
|
||||
prediction that the slab was a threshold below 26 GiB died on its own
|
||||
measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
|
||||
arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
|
||||
and the 4-slot fp8 shape are the documented rollbacks.
|
||||
**2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
|
||||
itself** — the 2026-10-12 power failure
|
||||
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
|
||||
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
|
||||
the FP8-native payload remains the rollback. **2026-10-11: the KV cache is
|
||||
UNQUANTIZED (bf16) at 2 x 200,000** — pool 3543 → 6912 MiB (386,904 →
|
||||
462,752 tokens), weights slab 26071 → 22208, host pool 44928,
|
||||
`--max-num-seqs 2`. bf16 costs 1.89x fp8 per token (the QSA block key is
|
||||
dtype-fixed, so it is not a clean 2x); measured price is prefill (-29%),
|
||||
NOT decode, and spec-decode acceptance rose 0.661 → 0.740.
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
|
||||
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
|
||||
@@ -28,6 +28,7 @@ covers both workloads).
|
||||
- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
|
||||
- [[Storage Policy]] — bind mounts, data locations, ownership
|
||||
- [[Deployment Flow]] — service-scoped git deploy, target map, and health checks
|
||||
- [[Service Inventory]] — shared/services.yml, the one inventory of deployable services, and service-map.py
|
||||
- [[Testing Policy]] — the test seams and how to run them
|
||||
- [[E2E Suite]] — the Playwright suite that drives the live guard chain
|
||||
- [[Container Network Namespaces]] — which local services are addressable from each Podman network namespace on Dankclaw
|
||||
@@ -182,20 +183,12 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
### Deployment and Topology
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE frank-archon runbook
|
||||
(2026-10-15)**: the PUBLISHER's container of the official
|
||||
Qwen3.8-Flash-Next (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||||
w4nl 4-bit-codebook experts + int8 trunk, Hub-sha256-verified at fetch)
|
||||
swapped in as a MODEL_FILE-only change — the 2026-10-13 shape (slab
|
||||
25600 / fp8-KV 5120, 3 × 200,000, host pool 44,928, MTP depth 2) carries
|
||||
verbatim; rollback = revert the swap commit (the w4dn8 `.rad` stays on
|
||||
disk; the fp8-native `.rad` was deleted 2026-10-15, rebuildable)
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — RETIRED 2026-10-08 → 10-15 frank-archon runbook:
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — STAGED frank-archon runbook:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` (radiance
|
||||
1.0.13), gated on KLD ≤ 0.03 vs the BF16 container and decode ≥ 30.8 t/s;
|
||||
activates by flipping `MODEL_FILE` in `scripts/frank-archon.env`, rollback
|
||||
= revert that line (the FP8-native `.rad` stays on disk)
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — the RETIRED 2026-10-05 → 10-08 frank-archon
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — the ACTIVE frank-archon
|
||||
runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
|
||||
287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
|
||||
|
||||
@@ -1,99 +1,5 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-10-13
|
||||
- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
|
||||
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
|
||||
from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
|
||||
below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
|
||||
preflight named the refusal and exited 1, and `Restart=always` re-ran it
|
||||
**6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
|
||||
container, no engine log. Law adopted: **a sysfs raise is a countdown to the
|
||||
next reboot**, so the gate raises the knob itself. New
|
||||
`scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
|
||||
re-verify, then refuse by name) and is also the unit's
|
||||
`ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank`
|
||||
and the knob is root-owned 0644, which is what the first attempt (a plain
|
||||
write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800`
|
||||
records the host ceiling.
|
||||
- **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`,
|
||||
`--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the
|
||||
measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and
|
||||
at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab
|
||||
to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12
|
||||
fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact.
|
||||
`models.yml` needed no change: the per-request window is unchanged, only the
|
||||
concurrency.
|
||||
- **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB …
|
||||
backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the
|
||||
sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB
|
||||
of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe
|
||||
(now reading the shape from the deployed env): four concurrent 196,960-token
|
||||
chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s
|
||||
aggregate prefill, no preemption; prefix cache verified (repeat answered in
|
||||
1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
|
||||
homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
|
||||
- **Re-cut twice the same evening (Frank's calls), each read off the ledger.**
|
||||
4 → 3 slots, pool 6,912 → 5,120: three slots need 5,063 MiB and the 6,912 pool
|
||||
was funding 858,969 tokens — 43 % over the contract. Then slab
|
||||
24,000 → 25,600, claiming 1,600 of the 1,849 MiB the engine leaves above its
|
||||
OWN settle floor (`card 0 settled at 2.01 GiB free against 209.19 MiB (the
|
||||
96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`): the
|
||||
real ceiling is 30,969 MiB, not the 29,120 my 1 GiB "keep free" reserve
|
||||
invented on 2026-10-11. The pair now spends 30,720 and settles
|
||||
`unclaimed 477.30 MiB` (cards ~336 MiB free).
|
||||
- **The prediction shipped with that arm was falsified by its own measurement,
|
||||
which is the whole point of banking the A/B.** Claim: the slab is a THRESHOLD
|
||||
below ~26 GiB (22,208 → 3,397 and 24,000 → 3,290 t/s are inside noise of each
|
||||
other), so 1.6 GiB buys ~0. Measured at 25,600: **3,584 t/s** single-stream
|
||||
cold at 43.5k (+8.9 %) and **3,589 t/s** 3-way aggregate at 197k (wall
|
||||
193.7 → 164.5 s, +17.6 %) — slab 16,235 → 17,548 slots while the pinned host
|
||||
pool hands the same experts back (8,893 → 7,580). The slab is a SLOPE; two
|
||||
points inside noise of each other are not a plateau. The cliff is above
|
||||
25,600: 26,071 measured 4,379 t/s (+22 % for +471 MiB), unreachable at three
|
||||
slots (26,112 + 5,120 > the ceiling) and free at two.
|
||||
- **Capacity verified live at 3 slots:** three concurrent 199,984-token prompts
|
||||
accepted and completed (599,952 tokens resident), ledger
|
||||
`backs 3 sequences all at 200000 tokens at once` at 607,740 tokens against the
|
||||
law's 608,027. A fourth full-height request is refused as too large to ever
|
||||
fit — radiance never truncates. Probe law: the oversized-prompt counter is
|
||||
FREE (the refusal fires when PROMPT tokens alone exceed the window —
|
||||
`prompt: 202,777 prompt tokens does not fit the 200000 token context`) and the
|
||||
window is ~514,286 characters at the measured 2.5714 chars/token, so a
|
||||
calibration probe below that pays a real prefill instead of reporting.
|
||||
- **Open call for Frank:** two slots (pool 3,072 + slab 27,897 = the ceiling
|
||||
exactly) crosses the cliff for the +22 %, and costs the third concurrent 200k
|
||||
conversation. Untested arm; capacity decision, not tuning.
|
||||
- Seam: the pair ceiling is derived from the engine's budget table (30,969) and
|
||||
owns that arithmetic alone; the KV-pool contract (seqs × window at the dtype's
|
||||
measured rate + 110 MiB a sequence + 448 MiB of prefix checkpoints) keeps its
|
||||
own test. 371/371 tests and e2e 33/33 on both deploys.
|
||||
- **Power-failure casualties, NOT fixed (outside the archon window):**
|
||||
`mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
|
||||
raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
|
||||
media libraries on it read empty. `frank-job-digest.service` (frankbook
|
||||
job-hunt digest) is in `list-units --state=failed`.
|
||||
- **The slab is a threshold, not a slope** — measured, same method (probe A
|
||||
leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot):
|
||||
22,208 → 3,397 t/s, 24,000 → 3,290, **25,600 → 3,290**. The third step
|
||||
bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071
|
||||
(4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
|
||||
would clear it (pool 3,072 + slab 27,897) remains untested — and the
|
||||
paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
|
||||
not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
|
||||
- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
|
||||
cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
|
||||
is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
|
||||
give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate** at 43.5k and 12.5 at 197k.
|
||||
The slot count is paid in decode concurrency — a fused-step lottery, not a
|
||||
fair share — which is exactly the axis the shape hunt was blind to (probe
|
||||
leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler
|
||||
is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode
|
||||
table, so these rows are the baseline for this corpus, not a replacement.
|
||||
- **Seam:** `probe-frank-archon-kv.py` gains **leg D** (N-stream decode, 512
|
||||
tokens, per-stream + aggregate) so no future shape change can be priced on
|
||||
prefill alone.
|
||||
|
||||
|
||||
## 2026-10-10
|
||||
- **Fleet service rename: the StarCraft names became the SERVICE names.**
|
||||
The 2026-10-09 wire alias moved onto every repo artifact: units, serve
|
||||
@@ -1745,57 +1651,3 @@ every restart since the cutover.
|
||||
[[frank-archon-radiance-swift15-w4dn8]], and the LLM index entry corrected
|
||||
(the w4dn8 entry said STAGED, the fp8-native one said ACTIVE).
|
||||
|
||||
## 2026-10-15 — frank-archon: publisher iq4r payload swap (MODEL_FILE-only)
|
||||
|
||||
- Ask: replace the served model with
|
||||
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` — the engine author's
|
||||
published rad-convert of the OFFICIAL Qwen3.8-Flash-Next (w4nl codebook
|
||||
experts, ten bf16-protected/layer, int8 trunk, PLE/MTP/vision in-container,
|
||||
card KL 0.0789 / top-1 92.85 %) — **no other argument changes**, mindful
|
||||
the operator session runs on the endpoint being swapped.
|
||||
- Disk: Frank authorized one `rm` — the spent Swift1.5 BF16 source (336 GiB)
|
||||
and the two-era-old `qwen4exp-fp8-native.rad` (173 GiB) left
|
||||
`/home/dank/models`; 150 GB → 658 GB free. fp8-native is now rebuild-only
|
||||
(fetch + build scripts); rollback for THIS swap is the w4dn8 `.rad`
|
||||
(on disk) by reverting the commit.
|
||||
- Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes
|
||||
121,969,901,568 + Hub LFS sha256, `--verify-only` default) as transient
|
||||
unit `iq4r-fetch`; verified, staged sidecar files kept as provenance.
|
||||
- Shape carried VERBATIM (25600/5120 pair, 44928 pool, fp8 KV 3 × 200,000,
|
||||
MTP 2, wht6, MNBT 2048, froggeric v22.5). Tests re-keyed
|
||||
(`frank_archon_unit.bats` pins the new payload id + sha256, keeps the
|
||||
w4dn8 rollback provenance), homepage card + trio lore + both indexes
|
||||
updated; new runbook [[frank-archon-radiance-qwen38-iq4r]]. Commits
|
||||
`6dcf4a5` (swap) + `f44c398` + `c5ac072` (docs), full bats green.
|
||||
- Cutover self-severed by design (detached `deploy.sh` survived the session
|
||||
death when frank-archon restarted); engine healthy in ~100 s. First-boot
|
||||
ledger: w4nl loads on pin 1.0.13 (no pin bump); kv_attn 26.00 KiB fp8 @
|
||||
607,740 tokens (identical geometry); slab 17,610/17,658 slots per rank
|
||||
(> w4dn8's 17,548 — fewer bf16-protected experts pack more 4-bit table);
|
||||
sampler 9 rows; MTP 2/step; prefix tier RECOVERED 900,176 blocks across
|
||||
the swap (arch+tokenizer unchanged). Vision probe green (reasoning read
|
||||
the solid-red PNG; the 64-token budget went all to thinking — fleet law).
|
||||
- e2e gate went red on three dankclaw specs — NOT the swap: dankclaw rebooted
|
||||
01:41 and the gluetun boot-race dependency-failed `searxng`, `marreta`,
|
||||
and `marreta-browser` (jobs consumed while gluetun's start job retried;
|
||||
`Restart=always` never armed because the services never ran). Cascade
|
||||
restarted; skill codified (`gluetun-dependent-boot-race-recovery`).
|
||||
Gate rerunning against the repaired fleet; frank-archon's own health gate
|
||||
passed every deploy. Open: decode/prefill ladders vs the w4dn8 record,
|
||||
probe leg D — scheduled probe window.
|
||||
|
||||
## 2026-10-15 — authentik: two more human accounts (Dankclaw)
|
||||
- **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM, no
|
||||
repo objects.** Each `<user>@franksdank.cloud` in the human-user template
|
||||
shape (`is_active=True`, display name, password via `set_password` on stdin
|
||||
only — never argv, so nothing lands in `ps`), groups `Non-Admin Users` +
|
||||
`jellyfin_users` + `openwebui_users`; no admin group granted. Verified
|
||||
through the exact PasswordStage code path (`authenticate()` accepted each
|
||||
password and rejected a wrong one), `app_entitlements(open-webui)` →
|
||||
`Open WebUI Users` for both (identical to the ridley precedent), and
|
||||
`is_superuser=False` / `path=users`. Zero `proxysession` rows for either —
|
||||
groups were set before any login, so no claim purge was due. Downstream:
|
||||
Jellyfin visibility rides the LDAP outpost cache (expect lag); OpenWebUI
|
||||
mints their accounts as role `pending` on first login.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user template
|
||||
census extended with pk 22–23.
|
||||
|
||||
@@ -366,13 +366,13 @@
|
||||
"model": {
|
||||
"unit": "archon",
|
||||
"role": "Frank's strongest and fastest model runtime",
|
||||
"hf": "https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe",
|
||||
"hf": "https://huggingface.co/frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700",
|
||||
"tagline": "...Power Overwhelming...",
|
||||
"sfx": [
|
||||
"archon-power.ogg"
|
||||
],
|
||||
"equiv": "Equivalent to Claude Opus 4.8 / OpenAI-Terra (Summer 2026 frontier intelligence).",
|
||||
"spec": "The engine author's published quant: 4-bit codebook experts + int8 trunk of Qwen3.8-Flash-Next (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — 200k-token context, three submissions at a time; sees images.",
|
||||
"spec": "Our own build: a GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next, quantized and tuned in-house (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — 200k-token context, three submissions at a time; sees images.",
|
||||
"inputs": [
|
||||
"text",
|
||||
"image"
|
||||
|
||||
@@ -0,0 +1,598 @@
|
||||
# =============================================================================
|
||||
# services.yml — THE service inventory.
|
||||
#
|
||||
# One entry per deployable service, in canonical deploy order (dependencies
|
||||
# before consumers, the LLM fleet last). Four hand-maintained lists used to
|
||||
# agree with each other by discipline alone; every one of them now READS this
|
||||
# file:
|
||||
#
|
||||
# scripts/deploy-plan.py which services a changed path selects
|
||||
# scripts/deploy-remote.sh which services a target accepts (allow-list)
|
||||
# scripts/test-changed.sh which Bats seams a changed path must re-run
|
||||
# scripts/service-map.py renderers + drift gates (`check`)
|
||||
#
|
||||
# `service-map.py check` is the referee: every quadlet unit file and every
|
||||
# root unit in the tree is claimed by exactly one entry, every repo file is
|
||||
# owned by a service or listed in `unowned:` with a reason, every seam exists,
|
||||
# the rendered Pi-hole hosts file equals the render, and Prometheus' scrape
|
||||
# targets plus the homepage probes equal the `scrape:` / `probe:` facts.
|
||||
#
|
||||
# --- fields ------------------------------------------------------------------
|
||||
# name the deployer's service word == the systemd unit name for
|
||||
# quadlet / root-unit / vm-quadlet kinds
|
||||
# hosts targets that accept it (the deploy-remote.sh allow-list)
|
||||
# kind quadlet rootless user unit from a .container file
|
||||
# root-unit systemd system unit symlinked out of scripts/
|
||||
# vm-quadlet quadlet inside the FrankBook-Pro podman machine
|
||||
# host a host-level script/timer, no container
|
||||
# deployed_by this service deploys as part of another deployable's handler
|
||||
# (the trio/pair shape). It has no dirs/files of its own; the
|
||||
# owner names the deployable whose handler installs it.
|
||||
# unit_dir repo dir holding <unit>.container — a scalar, or a per-host map
|
||||
# when one service carries its unit on two hosts (node-exporter)
|
||||
# units container names this deployable owns (netns coupling is derived
|
||||
# from the `Network=` lines in those files, never declared here)
|
||||
# dirs repo dirs whose every file selects this service
|
||||
# files exact repo files that select this service (root-unit trios,
|
||||
# vendored templates, host scripts installed by this handler)
|
||||
# retires files whose DELETION selects this service (the unit the
|
||||
# handler must remove before it can claim the ports)
|
||||
# only Dankserver's media stack still has one)
|
||||
# health the deployer's readiness URL; `wait` its 2 s poll budget
|
||||
# scrape Prometheus jobs: {job, target, metrics_path?, labels?}
|
||||
# dns names this service answers at (rendered into the Pi-hole
|
||||
# addn-hosts; every vhost resolves at the Dankclaw Caddy edge)
|
||||
# probe host:port the homepage checker dials (per-host map if on 2 hosts)
|
||||
# tests the Bats seams a change to this service must re-run
|
||||
# =============================================================================
|
||||
|
||||
hosts:
|
||||
dankclaw:
|
||||
address: 192.168.0.143
|
||||
# Caddy is consolidated on Dankclaw: it fronts every service on every
|
||||
# host, so every DNS name in the fleet resolves here.
|
||||
dns_edge: 192.168.0.143
|
||||
dankserver:
|
||||
address: 192.168.0.235
|
||||
frankbook-pro:
|
||||
address: 192.168.0.112
|
||||
|
||||
# Repo files no service owns. Each entry is a fact about the tree, not a
|
||||
# fallback: the census gate fails on anything unowned that is NOT listed, so
|
||||
# retirement work shrinks this list and a new stray file cannot slip in.
|
||||
unowned:
|
||||
shared/services.yml: the inventory this tool reads
|
||||
scripts: repo tooling (build, fetch, bench, deploy, test); the deployers'
|
||||
files are claimed by name in the service entries
|
||||
autoresearch.sh: bench scratch at the repo root (retirement pending)
|
||||
echo_prompt.txt: bench fixture at the repo root (retirement pending)
|
||||
echo_request.json: bench fixture at the repo root (retirement pending)
|
||||
prose_prompt.txt: bench fixture at the repo root (retirement pending)
|
||||
prose_request.json: bench fixture at the repo root (retirement pending)
|
||||
dankclaw/apply_qwen21_patch.py: retired patch script (its target runtime is gone)
|
||||
dankclaw/comfyui-start.sh: superseded by the comfyui quadlet + Containerfile
|
||||
dankclaw/udev: host udev rules, applied by manual sudo install + udevadm trigger
|
||||
tests: the seams themselves
|
||||
report: frozen point-in-time audit reports
|
||||
old: retired scripts, reference only
|
||||
.githooks: the commit hooks that run the seams
|
||||
|
||||
services:
|
||||
# --- Search stack: one netns family, gluetun owns it ---------------------
|
||||
- name: gluetun
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/searxng/quadlet
|
||||
units: [gluetun]
|
||||
tests: [tests/searxng_quadlet.bats]
|
||||
# The tunnel's WireGuard bind — the homepage card for the tunnel itself.
|
||||
probe: 127.0.0.1:51820
|
||||
|
||||
- name: searxng-valkey
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/searxng/quadlet
|
||||
units: [searxng-valkey]
|
||||
# An app-level dependency, not a network one: the deploy handler
|
||||
# restarts SearXNG after its cache, so the cache's files select both.
|
||||
dependents: [searxng]
|
||||
tests: [tests/searxng_quadlet.bats]
|
||||
probe: 127.0.0.1:6379
|
||||
|
||||
- name: searxng
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/searxng/quadlet
|
||||
units: [searxng]
|
||||
dirs: [shared/searxng]
|
||||
tests: [tests/searxng_limiter.bats, tests/searxng_quadlet.bats]
|
||||
scrape:
|
||||
- job: services
|
||||
target: 192.168.0.143:8081
|
||||
metrics_path: /
|
||||
dns: [search.franksdank.cloud]
|
||||
probe: 127.0.0.1:8081
|
||||
|
||||
- name: marreta
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/marreta/quadlet
|
||||
units: [marreta, marreta-browser]
|
||||
tests: [tests/marreta_quadlet.bats]
|
||||
dns: [marreta.franksdank.cloud]
|
||||
probe: 127.0.0.1:8082
|
||||
|
||||
# --- DNS ------------------------------------------------------------------
|
||||
- name: pihole
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/pihole/quadlet
|
||||
units: [pihole]
|
||||
dirs: [dankclaw/pihole]
|
||||
tests: [tests/pihole.bats, tests/docs_completeness.bats]
|
||||
dns: [pihole.franksdank.cloud]
|
||||
probe: 192.168.0.143:8083
|
||||
|
||||
# --- Chat front end -------------------------------------------------------
|
||||
- name: open-webui
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/open-webui/quadlet
|
||||
units: [open-webui]
|
||||
tests: [tests/open_webui_quadlet.bats, tests/caddyfile.bats]
|
||||
scrape:
|
||||
- job: services
|
||||
target: 192.168.0.143:3005
|
||||
metrics_path: /
|
||||
dns: [ai.franksdank.cloud]
|
||||
probe: 127.0.0.1:3005
|
||||
|
||||
# --- Git hosting ----------------------------------------------------------
|
||||
- name: gitea-postgres
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/gitea/quadlet
|
||||
units: [gitea-postgres]
|
||||
deployed_by: gitea-server
|
||||
tests: [tests/caddyfile.bats, tests/docs_completeness.bats]
|
||||
probe: 127.0.0.1:5432
|
||||
|
||||
- name: gitea-server
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/gitea/quadlet
|
||||
units: [gitea-server]
|
||||
tests: [tests/caddyfile.bats, tests/docs_completeness.bats]
|
||||
scrape:
|
||||
- job: gitea
|
||||
target: 192.168.0.143:3000
|
||||
metrics_path: /metrics
|
||||
dns: [gitea.franksdank.cloud]
|
||||
probe: 127.0.0.1:3000
|
||||
|
||||
# --- Image generation -----------------------------------------------------
|
||||
- name: comfyui
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/comfyui/quadlet
|
||||
units: [comfyui]
|
||||
dirs: [dankclaw/comfyui]
|
||||
files: [dankclaw/Containerfile.comfyui]
|
||||
health: http://127.0.0.1:8188/
|
||||
wait: 300
|
||||
tests: [tests/comfyui_rocm.bats, tests/docs_completeness.bats]
|
||||
scrape:
|
||||
- job: comfyui
|
||||
target: comfyui:8188
|
||||
metrics_path: /
|
||||
dns: [comfyui.franksdank.cloud]
|
||||
probe: 127.0.0.1:8188
|
||||
|
||||
# --- Requests -------------------------------------------------------------
|
||||
- name: seerr
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/seerr/quadlet
|
||||
units: [seerr]
|
||||
tests: [tests/seerr_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [seerr.franksdank.cloud]
|
||||
probe: 127.0.0.1:5055
|
||||
|
||||
# --- Agent stack: one handler, four containers ----------------------------
|
||||
- name: hermes
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/hermes/quadlet
|
||||
units: [hermes, hermes-dashboard, hermes-webui, chrome-devtools-protocol]
|
||||
dirs: [dankclaw/hermes]
|
||||
tests: [tests/hermes_webui.bats, tests/docs_completeness.bats]
|
||||
dns: [hermes.franksdank.cloud]
|
||||
probe: 127.0.0.1:8787
|
||||
|
||||
- name: hermes-dashboard
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
deployed_by: hermes
|
||||
tests: [tests/hermes_webui.bats, tests/docs_completeness.bats]
|
||||
dns: [hermesadmin.franksdank.cloud]
|
||||
probe: 192.168.0.143:9119
|
||||
|
||||
- name: hermes-webui
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
deployed_by: hermes
|
||||
tests: [tests/hermes_webui.bats, tests/docs_completeness.bats]
|
||||
|
||||
# --- Observability --------------------------------------------------------
|
||||
- name: prometheus
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/observability/quadlet
|
||||
units: [prometheus]
|
||||
dirs: [shared/prometheus]
|
||||
tests: [tests/observability_quadlet.bats, tests/caddyfile.bats,
|
||||
tests/docs_completeness.bats]
|
||||
scrape:
|
||||
- job: prometheus
|
||||
target: localhost:9090
|
||||
dns: [prometheus.franksdank.cloud]
|
||||
probe: 127.0.0.1:9090
|
||||
|
||||
- name: grafana
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/observability/quadlet
|
||||
units: [grafana]
|
||||
dirs: [shared/grafana]
|
||||
tests: [tests/observability_quadlet.bats, tests/caddyfile.bats,
|
||||
tests/docs_completeness.bats]
|
||||
scrape:
|
||||
- job: grafana
|
||||
target: grafana:3000
|
||||
metrics_path: /metrics
|
||||
dns: [grafana.franksdank.cloud]
|
||||
probe: 127.0.0.1:3002
|
||||
|
||||
- name: loki
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/observability/quadlet
|
||||
units: [loki]
|
||||
dirs: [dankclaw/loki, shared/loki]
|
||||
tests: [tests/observability_quadlet.bats, tests/caddyfile.bats,
|
||||
tests/docs_completeness.bats]
|
||||
scrape:
|
||||
- job: loki
|
||||
target: loki:3100
|
||||
metrics_path: /metrics
|
||||
dns: [loki.franksdank.cloud]
|
||||
probe: 127.0.0.1:3100
|
||||
|
||||
- name: alloy
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/observability/quadlet
|
||||
units: [alloy]
|
||||
dirs: [dankclaw/alloy, shared/alloy]
|
||||
tests: [tests/observability_quadlet.bats, tests/caddyfile.bats,
|
||||
tests/docs_completeness.bats]
|
||||
probe: 127.0.0.1:12345
|
||||
|
||||
- name: node-exporter
|
||||
# One service on two hosts: the same exporter, its own unit per host.
|
||||
hosts: [dankclaw, dankserver]
|
||||
kind: quadlet
|
||||
unit_dir:
|
||||
dankclaw: dankclaw/observability/quadlet
|
||||
dankserver: dankserver/observability/quadlet
|
||||
units: [node-exporter]
|
||||
tests: [tests/observability_quadlet.bats, tests/caddyfile.bats,
|
||||
tests/docs_completeness.bats]
|
||||
scrape:
|
||||
- job: node-dankclaw
|
||||
target: host.containers.internal:9100
|
||||
- job: node-dankserver
|
||||
target: 192.168.0.235:9100
|
||||
probe: 127.0.0.1:9100
|
||||
|
||||
- name: alertmanager
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/observability/quadlet
|
||||
units: [alertmanager]
|
||||
# The Alertmanager config ships from Prometheus' directory, and the Signal
|
||||
# relay is a HOST unit this handler installs: a change to either reaches
|
||||
# the machine only through an alertmanager deploy.
|
||||
files:
|
||||
- shared/prometheus/alertmanager.yml
|
||||
- scripts/signal-relay.py
|
||||
- scripts/signal-relay.service
|
||||
tests: [tests/observability_quadlet.bats, tests/caddyfile.bats,
|
||||
tests/docs_completeness.bats]
|
||||
probe: 127.0.0.1:9093
|
||||
|
||||
# --- Identity -------------------------------------------------------------
|
||||
- name: authentik-postgres
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/authentik/quadlet
|
||||
units: [authentik-postgres]
|
||||
deployed_by: authentik-server
|
||||
tests: [tests/authentik.bats]
|
||||
|
||||
- name: authentik-server
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/authentik/quadlet
|
||||
units: [authentik-server]
|
||||
tests: [tests/authentik.bats]
|
||||
dns: [auth.franksdank.cloud]
|
||||
|
||||
- name: authentik-worker
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/authentik/quadlet
|
||||
units: [authentik-worker]
|
||||
deployed_by: authentik-server
|
||||
tests: [tests/authentik.bats]
|
||||
|
||||
- name: authentik-outpost
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/authentik/quadlet
|
||||
units: [authentik-outpost]
|
||||
tests: [tests/authentik.bats]
|
||||
|
||||
- name: authentik-outpost-ldap
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/authentik/quadlet
|
||||
units: [authentik-outpost-ldap]
|
||||
tests: [tests/authentik.bats]
|
||||
|
||||
# --- The auth chain's edge, then the edge itself --------------------------
|
||||
- name: anubis
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/anubis/quadlet
|
||||
units: [anubis]
|
||||
dirs: [dankclaw/anubis]
|
||||
tests: [tests/anubis.bats, tests/caddyfile.bats, tests/homepage_caddy.bats]
|
||||
|
||||
- name: caddy
|
||||
hosts: [dankclaw]
|
||||
kind: quadlet
|
||||
unit_dir: dankclaw/caddy/quadlet
|
||||
units: [caddy]
|
||||
dirs: [dankclaw/caddy]
|
||||
files: [dankclaw/Caddyfile]
|
||||
# Retiring the old split proxy selects Caddy: it must remove the installed
|
||||
# unit and container before it can claim 80/443.
|
||||
retires: [dankclaw/gitea/quadlet/gitea-caddy.container]
|
||||
tests: [tests/caddy_quadlet.bats, tests/caddyfile.bats,
|
||||
tests/homepage_caddy.bats, tests/docs_completeness.bats]
|
||||
dns:
|
||||
- franksdank.cloud
|
||||
- www.franksdank.cloud
|
||||
- dankclaw.franksdank.cloud
|
||||
- dankserver.franksdank.cloud
|
||||
probe: 127.0.0.1:80
|
||||
|
||||
- name: homepage
|
||||
hosts: [dankclaw]
|
||||
kind: host
|
||||
dirs: [shared/homepage]
|
||||
files:
|
||||
- scripts/homepage-status.py
|
||||
- scripts/homepage-status.service
|
||||
- scripts/homepage-status.timer
|
||||
tests: [tests/homepage_caddy.bats, tests/homepage_config.bats,
|
||||
tests/homepage_deploy.bats, tests/homepage_pages.bats,
|
||||
tests/homepage_checker.bats]
|
||||
|
||||
# --- Dankserver media stack ----------------------------------------------
|
||||
# The romm trio is one handler and one directory: every file in it selects
|
||||
# all three, exactly as the deployer restarts all three.
|
||||
- name: romm-mariadb
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/romm/quadlet
|
||||
units: [romm-mariadb]
|
||||
deployed_by: romm
|
||||
dirs: [dankserver/romm]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
probe: 192.168.0.235:3306
|
||||
|
||||
- name: romm-redis
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/romm/quadlet
|
||||
units: [romm-redis]
|
||||
deployed_by: romm
|
||||
dirs: [dankserver/romm]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
|
||||
- name: romm
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/romm/quadlet
|
||||
units: [romm]
|
||||
dirs: [dankserver/romm]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [romm.franksdank.cloud]
|
||||
probe: 192.168.0.235:8085
|
||||
|
||||
- name: copyparty
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/copyparty/quadlet
|
||||
units: [copyparty]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [files.franksdank.cloud]
|
||||
probe: 192.168.0.235:3923
|
||||
|
||||
- name: jellyfin
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/jellyfin/quadlet
|
||||
units: [jellyfin]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [jelly.franksdank.cloud]
|
||||
probe: 192.168.0.235:8096
|
||||
|
||||
- name: calibre-web
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/calibre-web/quadlet
|
||||
units: [calibre-web]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [books.franksdank.cloud]
|
||||
probe: 192.168.0.235:8083
|
||||
|
||||
- name: calibre-server
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/calibre-server/quadlet
|
||||
units: [calibre-server]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
|
||||
- name: plex
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/plex/quadlet
|
||||
units: [plex]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [plex.franksdank.cloud, plex.franksdank.casa]
|
||||
probe: 192.168.0.235:32400
|
||||
|
||||
- name: tautulli
|
||||
hosts: [dankserver]
|
||||
kind: quadlet
|
||||
unit_dir: dankserver/tautulli/quadlet
|
||||
units: [tautulli]
|
||||
tests: [tests/dankserver_quadlet.bats, tests/deployment_flow.bats]
|
||||
probe: 192.168.0.235:8181
|
||||
|
||||
# --- FrankBook-Pro: the *arr stack inside the podman machine --------------
|
||||
# Every container rides gluetun's netns, and that coupling is DERIVED: each
|
||||
# arr's unit file says `Network=gluetun.container`, so a gluetun unit change
|
||||
# selects the whole stack with no declaration here.
|
||||
- name: gluetun
|
||||
hosts: [frankbook-pro]
|
||||
kind: vm-quadlet
|
||||
unit_dir: frankbook-pro/gluetun/quadlet
|
||||
units: [gluetun]
|
||||
tests: [tests/frankbook_quadlet.bats, tests/deployment_flow.bats]
|
||||
|
||||
- name: prowlarr
|
||||
hosts: [frankbook-pro]
|
||||
kind: vm-quadlet
|
||||
unit_dir: frankbook-pro/prowlarr/quadlet
|
||||
units: [prowlarr]
|
||||
tests: [tests/frankbook_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [prowlarr.franksdank.cloud]
|
||||
probe: 192.168.0.112:9696
|
||||
|
||||
- name: qbittorrent
|
||||
hosts: [frankbook-pro]
|
||||
kind: vm-quadlet
|
||||
unit_dir: frankbook-pro/qbittorrent/quadlet
|
||||
units: [qbittorrent]
|
||||
tests: [tests/frankbook_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [qbittorrent.franksdank.cloud]
|
||||
probe: 192.168.0.112:8080
|
||||
|
||||
- name: radarr
|
||||
hosts: [frankbook-pro]
|
||||
kind: vm-quadlet
|
||||
unit_dir: frankbook-pro/radarr/quadlet
|
||||
units: [radarr]
|
||||
tests: [tests/frankbook_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [radarr.franksdank.cloud]
|
||||
probe: 192.168.0.112:7878
|
||||
|
||||
- name: sonarr
|
||||
hosts: [frankbook-pro]
|
||||
kind: vm-quadlet
|
||||
unit_dir: frankbook-pro/sonarr/quadlet
|
||||
units: [sonarr]
|
||||
tests: [tests/frankbook_quadlet.bats, tests/deployment_flow.bats]
|
||||
probe: 192.168.0.112:8989
|
||||
|
||||
- name: lidarr
|
||||
hosts: [frankbook-pro]
|
||||
kind: vm-quadlet
|
||||
unit_dir: frankbook-pro/lidarr/quadlet
|
||||
units: [lidarr]
|
||||
tests: [tests/frankbook_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [lidarr.franksdank.cloud]
|
||||
probe: 192.168.0.112:8686
|
||||
|
||||
- name: readarr
|
||||
hosts: [frankbook-pro]
|
||||
kind: vm-quadlet
|
||||
unit_dir: frankbook-pro/readarr/quadlet
|
||||
units: [readarr]
|
||||
tests: [tests/frankbook_quadlet.bats, tests/deployment_flow.bats]
|
||||
dns: [readarr.franksdank.cloud]
|
||||
probe: 192.168.0.112:8787
|
||||
|
||||
# --- The LLM fleet (last: never implied by a generic change) --------------
|
||||
- name: frank-dragoon
|
||||
hosts: [dankclaw]
|
||||
kind: root-unit
|
||||
files:
|
||||
- scripts/frank-dragoon.env
|
||||
- scripts/frank-dragoon.service
|
||||
- scripts/serve-frank-dragoon.sh
|
||||
health: http://127.0.0.1:7770/health
|
||||
wait: 1200
|
||||
tests: [tests/frank_dragoon_unit.bats, tests/llm_units.bats,
|
||||
tests/llm_deploy_gates.bats, tests/llm_secrets.bats,
|
||||
tests/scripts_syntax.bats, tests/docs_completeness.bats]
|
||||
scrape:
|
||||
- job: frank-dragoon
|
||||
target: host.containers.internal:7770
|
||||
metrics_path: /metrics
|
||||
labels: {service: frank-dragoon}
|
||||
probe: 127.0.0.1:7770
|
||||
|
||||
- name: frank-zealot
|
||||
hosts: [dankclaw]
|
||||
kind: root-unit
|
||||
files:
|
||||
- scripts/frank-zealot.env
|
||||
- scripts/frank-zealot.service
|
||||
- scripts/serve-frank-zealot.sh
|
||||
health: http://127.0.0.1:7771/health
|
||||
wait: 900
|
||||
tests: [tests/frank_zealot_unit.bats, tests/llm_units.bats,
|
||||
tests/llm_deploy_gates.bats, tests/llm_secrets.bats,
|
||||
tests/scripts_syntax.bats]
|
||||
scrape:
|
||||
- job: frank-zealot
|
||||
target: host.containers.internal:7771
|
||||
metrics_path: /metrics
|
||||
labels: {service: frank-zealot}
|
||||
probe: 127.0.0.1:7771
|
||||
|
||||
- name: frank-archon
|
||||
hosts: [dankserver]
|
||||
kind: root-unit
|
||||
files:
|
||||
- scripts/frank-archon.env
|
||||
- scripts/frank-archon.service
|
||||
- scripts/serve-frank-archon.sh
|
||||
# The vendored chat template the engine renders with.
|
||||
- scripts/qwen38-froggeric-v22.5.jinja
|
||||
health: http://127.0.0.1:7770/health
|
||||
wait: 900
|
||||
tests: [tests/frank_archon_unit.bats, tests/llm_units.bats,
|
||||
tests/llm_deploy_gates.bats, tests/llm_secrets.bats,
|
||||
tests/scripts_syntax.bats]
|
||||
scrape:
|
||||
- job: frank-archon
|
||||
target: 192.168.0.235:7770
|
||||
metrics_path: /metrics
|
||||
labels: {service: frank-archon}
|
||||
probe: 192.168.0.235:7770
|
||||
@@ -122,7 +122,7 @@ UNIT_DIR="$REPO_ROOT/dankclaw/authentik/quadlet"
|
||||
@test "the homepage vhost is www, guarded by Anubis" {
|
||||
local block
|
||||
block=$(awk '/^www\.franksdank\.cloud[[:space:]]*\{/,/^}/' "$CADDYFILE")
|
||||
echo "$block" | grep -q 'reverse_proxy anubis:8923'
|
||||
echo "$block" | grep -q 'import guarded'
|
||||
}
|
||||
|
||||
@test "the naked domain redirects to www at the TLS edge" {
|
||||
|
||||
@@ -251,3 +251,14 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
|
||||
grep -q -- '--server.http.listen-addr=0.0.0.0:12345' "$unit"
|
||||
grep -q '^PublishPort=127.0.0.1:12345:12345$' "$unit"
|
||||
}
|
||||
|
||||
@test "the guard chain is a named snippet the guarded vhosts import" {
|
||||
# Rank 6: the Anubis pre-auth chain (reverse_proxy anubis:8923 with the
|
||||
# X-Real-Ip header_up) existed as nine verbatim copies. One snippet,
|
||||
# per-vhost imports; a chain edit is a one-line change.
|
||||
grep -q '^(guarded) {' "$CADDYFILE"
|
||||
local imports
|
||||
imports=$(grep -c $'^\timport guarded$' "$CADDYFILE")
|
||||
[ "$imports" -ge 9 ]
|
||||
[ "$(grep -c 'reverse_proxy anubis:8923' "$CADDYFILE")" -eq 1 ]
|
||||
}
|
||||
|
||||
@@ -13,11 +13,15 @@ load helpers
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "changed-test wrapper maps a compose change to compose and environment tests" {
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths dankserver/compose.yaml
|
||||
@test "changed-test wrapper maps a shared config change to its service seams" {
|
||||
# The compose era is retired (rank 7): the former compose fixture here
|
||||
# mapped dankserver/compose.yaml, a file that no longer exists. The
|
||||
# equivalent contract now: a service config change maps to the
|
||||
# service's seams and nothing unrelated.
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths shared/searxng/settings.yml
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"tests/compose_validity.bats"* ]]
|
||||
[[ "$output" == *"tests/env_consistency.bats"* ]]
|
||||
[[ "$output" == *"tests/searxng_limiter.bats"* ]]
|
||||
[[ "$output" == *"tests/searxng_quadlet.bats"* ]]
|
||||
[[ "$output" != *"tests/homepage_pages.bats"* ]]
|
||||
}
|
||||
|
||||
@@ -47,3 +51,27 @@ load helpers
|
||||
[[ "$output" != *"tests/frank_medium_unit.bats"* ]]
|
||||
[[ "$output" != *"tests/frank_archon_unit.bats"* ]]
|
||||
}
|
||||
|
||||
@test "changed-test wrapper maps the frankbook unit to the frankbook and deploy-flow seams" {
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths frankbook-pro/sonarr/quadlet/sonarr.container
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"tests/frankbook_quadlet.bats"* ]]
|
||||
[[ "$output" == *"tests/deployment_flow.bats"* ]]
|
||||
[[ "$output" == *"tests/deployment_scoping.bats"* ]]
|
||||
[[ "$output" != *"tests/frank_archon_unit.bats"* ]]
|
||||
}
|
||||
|
||||
@test "changed-test wrapper maps the inventory itself to its seam" {
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths shared/services.yml
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"tests/service_inventory.bats"* ]]
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths scripts/service-map.py
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"tests/service_inventory.bats"* ]]
|
||||
}
|
||||
|
||||
@test "changed-test wrapper refuses an unmapped path" {
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths newdir/file.txt
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"no focused-test mapping"* ]]
|
||||
}
|
||||
|
||||
Executable
+129
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy_sandbox.bash — the deploy-handler seam's fixture builder.
|
||||
#
|
||||
# The seam (tests/llm_deploy_gates.bats) EXECUTES scripts/deploy-remote.sh's
|
||||
# LLM handlers with every host touchpoint stubbed (tests/stubs/) and every
|
||||
# artifact pointed at a fixture, so the assertions are the handler's exit
|
||||
# status, its refusal text, and what it ASKED systemd/podman to do.
|
||||
#
|
||||
# Usage:
|
||||
# source tests/deploy_sandbox.bash
|
||||
# sandbox_setup
|
||||
# run_handler dankserver frank-archon
|
||||
# sandbox_teardown
|
||||
#
|
||||
# After sandbox_setup the caller owns:
|
||||
# $STAGE the temp root (payload/, bin/, systemd/, state/)
|
||||
# $REMOTE_DIR the sandbox checkout (a git clone; the handler
|
||||
# cds into it, so relative paths resolve there)
|
||||
# $STAGE/systemctl.log every systemctl invocation
|
||||
# $STAGE/podman.log every podman invocation
|
||||
# $STAGE/systemd/<unit>.service the unit link a green deploy leaves behind
|
||||
# Env the caller sets per case: STUB_IMAGE ("" = empty storage),
|
||||
# STUB_MAIN_PID (0 = a stray process owns the port), the LLM_* overrides.
|
||||
|
||||
sandbox_setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
export HOME="$STAGE/home"
|
||||
mkdir -p "$HOME" "$STAGE/state" "$STAGE/systemd" "$STAGE/bin"
|
||||
|
||||
# Host touchpoints: the seam's stubs, on PATH under their real names.
|
||||
local stub repo_root
|
||||
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
for stub in "$repo_root"/tests/stubs/*; do
|
||||
ln -s "$stub" "$STAGE/bin/$(basename "$stub")"
|
||||
done
|
||||
export PATH="$STAGE/bin:$PATH"
|
||||
export STUB_SYSTEMCTL_LOG="$STAGE/systemctl.log"
|
||||
export STUB_PODMAN_LOG="$STAGE/podman.log"
|
||||
# A deploy marks state; a sandbox with no git identity would make the
|
||||
# identity, not the handler, the thing under test.
|
||||
export GIT_AUTHOR_NAME=seam GIT_AUTHOR_EMAIL=seam@localhost
|
||||
export GIT_COMMITTER_NAME=seam GIT_COMMITTER_EMAIL=seam@localhost
|
||||
export SYSTEMD_SYSTEM_DIR="$STAGE/systemd"
|
||||
export DEPLOY_STATE_DIR="$STAGE/state"
|
||||
# run_privileged pipes the password from FRANK_SUDO_PW into `sudo -S --`;
|
||||
# the stub drops it and execs, so no privilege gate hides a handler bug.
|
||||
export FRANK_SUDO_PW=test
|
||||
|
||||
# Fixture payload: 3 MiB where a size floor applies, 10 bytes elsewhere.
|
||||
mkdir -p "$STAGE/models/tokenizer"
|
||||
truncate -s 3145728 "$STAGE/models/payload.rad"
|
||||
truncate -s 10 "$STAGE/models/ck.hgn"
|
||||
truncate -s 10 "$STAGE/models/ngram.hgn"
|
||||
truncate -s 10 "$STAGE/models/vision.hgn"
|
||||
printf 'x\n' > "$STAGE/models/llama-server"
|
||||
chmod +x "$STAGE/models/llama-server"
|
||||
truncate -s 10 "$STAGE/models/model.gguf"
|
||||
truncate -s 10 "$STAGE/models/mmproj.gguf"
|
||||
local tf
|
||||
for tf in tokenizer.json chat_template.jinja tokenizer_config.json \
|
||||
merges.txt vocab.json generation_config.json; do
|
||||
printf 'x\n' > "$STAGE/models/tokenizer/$tf"
|
||||
done
|
||||
|
||||
# $HOME/serverAppConfigs is the checkout the handler works in: a git
|
||||
# WORKTREE of the caller's checkout, so the handler runs the EXACT files
|
||||
# under test, and it is a real git repo: the deployer refuses a revision
|
||||
# it cannot resolve. The handler cds into REMOTE_DIR, so every relative
|
||||
# path it uses resolves there.
|
||||
#
|
||||
# HEAD is committed state, so the seam asserts on what is in history —
|
||||
# the same rule as the whole suite, and the reason an uncommitted
|
||||
# deployer change is invisible here until it is committed.
|
||||
git -C "$repo_root" worktree add -q --detach "$HOME/serverAppConfigs" HEAD
|
||||
export REMOTE_DIR="$HOME/serverAppConfigs"
|
||||
|
||||
# Every endpoint variable is env-overridable, so the handlers see the
|
||||
# fixtures and never the real /home/dank paths.
|
||||
export STUB_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:deadbeef
|
||||
export LLM_DRAGOON_IMAGE="$STUB_IMAGE"
|
||||
export LLM_DRAGOON_CKPT="$STAGE/models/ck.hgn" LLM_DRAGOON_CKPT_BYTES=10
|
||||
export LLM_DRAGOON_NGRAM="$STAGE/models/ngram.hgn" LLM_DRAGOON_NGRAM_BYTES=10
|
||||
export LLM_DRAGOON_VISION="$STAGE/models/vision.hgn" LLM_DRAGOON_VISION_BYTES=10
|
||||
export LLM_DRAGOON_TOK_DIR="$STAGE/models/tokenizer"
|
||||
export LLM_ZEALOT_RUNTIME="$STAGE/models/llama-server"
|
||||
export LLM_ZEALOT_MODEL_FILE="$STAGE/models/model.gguf"
|
||||
export LLM_ZEALOT_MMPROJ="$STAGE/models/mmproj.gguf"
|
||||
export LLM_ARCHON_ENV_FILE="$REMOTE_DIR/scripts/frank-archon.env"
|
||||
# The archon handler reads the served image out of the unit's own env
|
||||
# file, so the stub's storage must hold exactly that ref.
|
||||
ARCHON_IMAGE="$(sed -nE 's/^FRANK_ARCHON_IMAGE=(.+)$/\1/p' "$LLM_ARCHON_ENV_FILE")"
|
||||
}
|
||||
|
||||
sandbox_teardown() {
|
||||
[[ -z "${STAGE:-}" ]] && return 0
|
||||
# A worktree leaves a registration in the caller's .git/worktrees, so
|
||||
# remove it through git, not just the directory.
|
||||
[[ -d "$STAGE/home/serverAppConfigs" ]] &&
|
||||
git -C "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" \
|
||||
worktree remove --force "$STAGE/home/serverAppConfigs"
|
||||
rm -rf "$STAGE"
|
||||
}
|
||||
|
||||
run_handler() {
|
||||
local target="$1" service="$2"
|
||||
run bash "$REMOTE_DIR/scripts/deploy-remote.sh" \
|
||||
"$target" "$(git -C "$REMOTE_DIR" rev-parse HEAD)" "$service"
|
||||
}
|
||||
|
||||
# archon_payload_env — the unit's env file with the payload coordinates
|
||||
# redirected into the sandbox (and the size floor optionally lowered, which
|
||||
# is how a test builds the truncated-.rad case).
|
||||
archon_payload_env() {
|
||||
local re="s#^FRANK_ARCHON_MODEL_DIR=.*#FRANK_ARCHON_MODEL_DIR=$STAGE/models#;"
|
||||
re+=" s#^FRANK_ARCHON_MODEL_FILE=.*#FRANK_ARCHON_MODEL_FILE=payload.rad#"
|
||||
if [[ -n "${1:-}" ]]; then
|
||||
re+="; s#^FRANK_ARCHON_RAD_MIN_MIB=.*#FRANK_ARCHON_RAD_MIN_MIB=$1#"
|
||||
fi
|
||||
sed -E "$re" "$LLM_ARCHON_ENV_FILE" > "$STAGE/archon.env"
|
||||
export LLM_ARCHON_ENV_FILE="$STAGE/archon.env"
|
||||
}
|
||||
|
||||
restarts_requested() {
|
||||
# grep -c prints 0 and exits 1 on no match; the count, not the status,
|
||||
# is what the assertions read.
|
||||
local n=0
|
||||
[[ -f "$STAGE/systemctl.log" ]] && n="$(grep -c 'restart ' "$STAGE/systemctl.log")"
|
||||
printf '%s\n' "$n"
|
||||
}
|
||||
@@ -78,7 +78,10 @@ load helpers
|
||||
}
|
||||
|
||||
@test "remote runner health-checks an inference endpoint" {
|
||||
grep -q '127.0.0.1:7770/health' "$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
# start_root_unit carries the health gate for all three LLM units; the
|
||||
# contract is that each endpoint's start names its port, and 7770 is
|
||||
# frank-dragoon's.
|
||||
grep -q 'start_root_unit frank-dragoon 7770' "$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
}
|
||||
|
||||
@test "deploy.sh pulls the merged checkout on the target" {
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: documentation completeness.
|
||||
# (compose era retired 2026-10-13; the sweep below now skips trash so a
|
||||
# revived compose file stays covered without resurrecting retired services)
|
||||
# Every service defined in a compose.yaml must have a matching documentation
|
||||
# page in serverAppConfigs-docs/ and must be reachable from the index. This is the "docs first"
|
||||
# contract of this repo: no service ships without documentation.
|
||||
@@ -18,7 +20,7 @@ load helpers
|
||||
return 1
|
||||
fi
|
||||
done < <(services_in "$f")
|
||||
done < <(find "$REPO_ROOT" -name 'compose.yaml' -not -path '*/.git/*' | sort)
|
||||
done < <(find "$REPO_ROOT" -name 'compose.yaml' -not -path '*/.git/*' -not -path '*/.AGENTTRASH/*' | sort)
|
||||
}
|
||||
|
||||
@test "docs index links every docs page" {
|
||||
|
||||
@@ -1,33 +1,12 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: environment variable consistency and secret hygiene.
|
||||
# Every ${VAR} referenced in a compose.yaml or Caddyfile must be defined in the
|
||||
# matching .env.example. .env.example files must contain only placeholders, and
|
||||
# Every ${VAR} referenced in a Caddyfile must be defined in the matching
|
||||
# .env.example. .env.example files must contain only placeholders, and
|
||||
# the live gitea database password must never appear in tracked files.
|
||||
# (The compose.yaml sweeps retired with the last compose file on 2026-10-13.)
|
||||
|
||||
load helpers
|
||||
|
||||
@test "every compose.yaml has a sibling .env.example" {
|
||||
while IFS= read -r f; do
|
||||
local envfile
|
||||
envfile="$(dirname "$f")/.env.example"
|
||||
[ -f "$envfile" ] || { echo "missing $envfile for $f"; return 1; }
|
||||
done < <(find "$REPO_ROOT" -name 'compose.yaml' -not -path '*/.git/*' | sort)
|
||||
}
|
||||
|
||||
@test "every \${VAR} in compose.yaml is defined in the sibling .env.example" {
|
||||
while IFS= read -r f; do
|
||||
local envfile
|
||||
envfile="$(dirname "$f")/.env.example"
|
||||
while IFS= read -r var; do
|
||||
[ -z "$var" ] && continue
|
||||
if ! env_vars_defined "$envfile" | grep -qx "$var"; then
|
||||
echo "$f references \${$var} but $envfile does not define it"
|
||||
return 1
|
||||
fi
|
||||
done < <(env_vars_in "$f")
|
||||
done < <(find "$REPO_ROOT" -name 'compose.yaml' -not -path '*/.git/*' | sort)
|
||||
}
|
||||
|
||||
@test ".env.example files contain only placeholder values" {
|
||||
while IFS= read -r f; do
|
||||
has_placeholder_secret "$f"
|
||||
|
||||
+1773
File diff suppressed because it is too large
Load Diff
+168
-191
@@ -3,52 +3,168 @@
|
||||
#
|
||||
# This file asserts the RADIANCE shape (StillDeadcode engine, digest-pinned
|
||||
# stilldeadcode/radiance image, BOTH R9700s TP=2 on the wht6 wire, serving
|
||||
# the publisher's 4-bit-codebook .rad of the official Qwen3.8-Flash-Next
|
||||
# (fetched by scripts/download-frank-archon-iq4r.sh; the in-house
|
||||
# Swift1.5 w4dn8 .rad of scripts/convert-swift15-w4dn8.sh is the on-disk
|
||||
# rollback). It replaces the FreeToken seam of
|
||||
# the Swift1.5 Qwen3.8-Flash-Next GPTQ-w4 .rad built by
|
||||
# scripts/convert-swift15-w4dn8.sh). It replaces the FreeToken seam of
|
||||
# 2026-10-05 (git tag frank-archon-freetoken-pr587-20261005) and pins what
|
||||
# flags alone do not say: the served window clears the harness worst case,
|
||||
# the KV pool backs seqs x max-model-len at the declared cache dtype (an
|
||||
# engine HARD limit, not a ratio), the per-card VRAM pair stays inside the
|
||||
# the pinned-host ledger has a ttm pages cap floor, the serve script ENFORCES
|
||||
# it and RAISES the knob itself when a reboot reset it (the cap's default is
|
||||
# half of RAM), the served window is what models.yml must
|
||||
# measured card ceiling, the pinned-host ledger has a ttm pages cap floor and
|
||||
# the serve script ENFORCES it, the served window is what models.yml must
|
||||
# quote, the host-networking contract (no --publish), the image gate on a
|
||||
# name@sha256 ref, the .rad payload gate with its size floor, the
|
||||
# chat-template override mount, the self-healing law (the run returns before
|
||||
# the engine is up, so the serve script must wait on the container for
|
||||
# Restart=always to mean anything), the deployer's false-green MainPID check,
|
||||
# the vendored hfget fetch path (no token for this ungated repo), and the
|
||||
# chat-template override mount, the self-healing contract (`podman run`
|
||||
# returns before the engine is up, so the serve script must stay attached
|
||||
# for Restart=always to mean anything), the deployer's false-green MainPID
|
||||
# check, the vendored hfget fetch path (no token for this ungated repo), and the
|
||||
# client contract (reasoning auto -> reasoning_content; models.yml maps the
|
||||
# field on the harness side, not here).
|
||||
|
||||
# The serve script is EXECUTED against fixtures (stub podman, stub device
|
||||
# nodes, stub ttm sysfs knobs, a temp payload dir) so the gates are observed
|
||||
# as the operator observes them: exit status and the words on stderr. The
|
||||
# knobs that make that possible (FRANK_ARCHON_DEV_KFD, _DEV_DRI,
|
||||
# _TTM_SYSFS_DIR, _LOG) default to the production paths.
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/scripts/frank-archon.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-archon.sh"
|
||||
TTM_LIB="$REPO_ROOT/scripts/ttm-preflight.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-archon.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-iq4r.sh"
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-swift15.sh"
|
||||
BUILD="$REPO_ROOT/scripts/convert-swift15-w4dn8.sh"
|
||||
HFGET="$REPO_ROOT/scripts/hfget.sh"
|
||||
TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
|
||||
#
|
||||
# Fixtures: a full env derived from the SHIPPED frank-archon.env with only
|
||||
# the host paths redirected, so a payload rename moves the env, not this file.
|
||||
setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
MODELS="$STAGE/models"
|
||||
STATE="$STAGE/state"
|
||||
TTM="$STAGE/ttm"
|
||||
mkdir -p "$MODELS" "$STATE" "$TTM" "$STAGE/bin" "$STAGE/dri"
|
||||
touch "$STAGE/kfd" "$STAGE/template.jinja"
|
||||
# A 3 MiB .rad clears the 2 MiB floor the fixture env pins.
|
||||
dd if=/dev/zero of="$MODELS/payload.rad" bs=1048576 count=3 status=none
|
||||
# 200 GiB of pages, the shipped cap's value, in 4 KiB pages.
|
||||
printf '52428800\n' > "$TTM/pages_limit"
|
||||
cat > "$STAGE/bin/podman" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
image)
|
||||
case " ${STUB_IMAGE:-} " in
|
||||
*" $3 "*) exit 0 ;;
|
||||
*) exit 1 ;;
|
||||
esac ;;
|
||||
rm) exit 0 ;;
|
||||
run) echo "PODMAN $*" >> "${STUB_ARGV:-/dev/null}"; exit 0 ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
STUB
|
||||
chmod +x "$STAGE/bin/podman"
|
||||
IMAGE="$(sed -nE 's/^FRANK_ARCHON_IMAGE=(.+)$/\1/p' "$ENV_FILE")"
|
||||
# The fixture env IS the shipped env with only the host paths and the
|
||||
# size floor redirected, so a payload rename moves frank-archon.env and
|
||||
# this file follows with it.
|
||||
local re="s#^FRANK_ARCHON_MODEL_DIR=.*#FRANK_ARCHON_MODEL_DIR=$MODELS#;"
|
||||
re+=" s#^FRANK_ARCHON_MODEL_FILE=.*#FRANK_ARCHON_MODEL_FILE=payload.rad#;"
|
||||
re+=" s#^FRANK_ARCHON_RAD_MIN_MIB=.*#FRANK_ARCHON_RAD_MIN_MIB=2#;"
|
||||
re+=" s#^FRANK_ARCHON_TEMPLATE=.*#FRANK_ARCHON_TEMPLATE=$STAGE/template.jinja#;"
|
||||
re+=" s#^FRANK_ARCHON_STATE_DIR=.*#FRANK_ARCHON_STATE_DIR=$STATE#;"
|
||||
re+=" s#^FRANK_ARCHON_TTM_MIN_MIB=.*#FRANK_ARCHON_TTM_MIN_MIB=150000#"
|
||||
sed -E "$re" "$ENV_FILE" > "$STAGE/serve.env"
|
||||
{
|
||||
echo "FRANK_ARCHON_TTM_SYSFS=$TTM"
|
||||
echo "FRANK_ARCHON_DEV_KFD=$STAGE/kfd"
|
||||
echo "FRANK_ARCHON_DEV_DRI=$STAGE/dri"
|
||||
echo "FRANK_ARCHON_LOG=$STAGE/serve.log"
|
||||
echo "STUB_IMAGE=$IMAGE"
|
||||
echo "STUB_ARGV=$STAGE/argv.txt"
|
||||
} >> "$STAGE/serve.env"
|
||||
}
|
||||
|
||||
# A PATH with `cat` (the gate reads knobs with it) and NO `sudo`, so the
|
||||
# privileged fallback cannot rescue a write the test means to fail.
|
||||
nosudo_bin() {
|
||||
local bin="$BATS_TEST_TMPDIR/nosudo-bin"
|
||||
mkdir -p "$bin"
|
||||
ln -sf "$(command -v cat)" "$bin/cat"
|
||||
ln -sf "$(command -v bash)" "$bin/bash"
|
||||
echo "$bin"
|
||||
teardown() {
|
||||
rm -rf "$STAGE"
|
||||
}
|
||||
|
||||
# serve_run — execute the serve script against the fixtures.
|
||||
serve_run() {
|
||||
run env PATH="$STAGE/bin:$PATH" FRANK_ARCHON_ENV_FILE="$STAGE/serve.env" \
|
||||
bash "$SERVE"
|
||||
}
|
||||
|
||||
@test "the serve script refuses when the image is not in storage" {
|
||||
# The env file wins over the ambient environment (the serve script
|
||||
# sources it with `set -a`), so the stub's storage is emptied in the file.
|
||||
sed -i 's#^STUB_IMAGE=.*#STUB_IMAGE=#' "$STAGE/serve.env"
|
||||
serve_run
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"not in podman storage"* ]]
|
||||
[[ "$output" == *"podman pull"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing .rad and names its builder" {
|
||||
rm -f "$MODELS/payload.rad"
|
||||
serve_run
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"container: missing"*"$MODELS/payload.rad"* ]]
|
||||
[[ "$output" == *"scripts/convert-swift15-w4dn8.sh"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script refuses a .rad below the size floor, naming both sizes" {
|
||||
# A truncated rad-convert is a guaranteed long-timeout boot; the floor is
|
||||
# the fast failure (FRANK_ARCHON_RAD_MIN_MIB).
|
||||
dd if=/dev/zero of="$MODELS/payload.rad" bs=1048576 count=1 status=none
|
||||
serve_run
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"1 MiB"* ]]
|
||||
[[ "$output" == *"2 MiB"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script refuses an absent chat template" {
|
||||
# The engine is started with --override-chat-template; a missing file is
|
||||
# a container that dies at argv parse, not a fast refusal.
|
||||
rm -f "$STAGE/template.jinja"
|
||||
serve_run
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"template: missing"*"$STAGE/template.jinja"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script refuses when the ttm pages cap is below the pinned ledger" {
|
||||
printf '1048576\n' > "$TTM/pages_limit" # 4 GiB, under the ~147 GiB ledger
|
||||
serve_run
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"ttm pages cap is 4096 MiB"* ]]
|
||||
[[ "$output" == *"150000"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script refuses when the GPU device nodes are absent" {
|
||||
rm -f "$STAGE/kfd"
|
||||
serve_run
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"GPU device nodes absent"*"$STAGE/kfd"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script runs the engine with the .rad, the template and TP=2" {
|
||||
serve_run
|
||||
[ "$status" -eq 0 ]
|
||||
argv="$(cat "$STAGE/argv.txt")"
|
||||
grep -q -- "--model /models/payload.rad" <<<"$argv"
|
||||
grep -q -- "-v $MODELS:/models:ro" <<<"$argv"
|
||||
grep -q -- "--override-chat-template /templates/template.jinja" <<<"$argv"
|
||||
grep -q -- "--tp $(sed -nE 's/^FRANK_ARCHON_TP=([0-9]+)$/\1/p' "$ENV_FILE")" <<<"$argv"
|
||||
grep -q -- "--tp-wire $(sed -nE 's/^FRANK_ARCHON_TP_WIRE=(.+)$/\1/p' "$ENV_FILE")" <<<"$argv"
|
||||
grep -q -- "--kv-cache-dtype $(sed -nE 's/^FRANK_ARCHON_KV_DTYPE=(.+)$/\1/p' "$ENV_FILE")" <<<"$argv"
|
||||
# The container name is the unit's ExecStop target, and the host-network
|
||||
# contract publishes no port.
|
||||
grep -q -- "--name $(sed -nE 's/^FRANK_ARCHON_CONTAINER=(.+)$/\1/p' "$ENV_FILE")" <<<"$argv"
|
||||
grep -q -- "--network host" <<<"$argv"
|
||||
! grep -q -- "--publish" <<<"$argv"
|
||||
}
|
||||
|
||||
@test "frank-archon unit, serve script, env, fetch and build scripts and template exist" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -x "$SERVE" ]
|
||||
[ -f "$TTM_LIB" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$DOWNLOAD" ]
|
||||
[ -x "$BUILD" ]
|
||||
@@ -135,40 +251,37 @@ nosudo_bin() {
|
||||
grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE"
|
||||
ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
|
||||
[ "$ml" -ge 177356 ]
|
||||
# Concurrency: 3 slots x the served window is the capacity contract, and
|
||||
# the fused TP all-reduce is declared over seqs x (1 + spec) = 9 rows,
|
||||
# under the 25-row bound at which the wht6 wire stays bit-exact. The
|
||||
# "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork
|
||||
# (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
|
||||
# seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
|
||||
# tied inside the +-3% noise floor — slots buy concurrency, not speed.
|
||||
grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE"
|
||||
# Concurrency: 2 slots x the served window is the capacity contract, and
|
||||
# the fused TP all-reduce is declared over seqs x (1 + spec) = 6 rows,
|
||||
# under the 25-row bound at which the wht6 wire stays bit-exact. A
|
||||
# "this slot count cannot boot" refusal is per-runtime (compiler shape
|
||||
# specialization rejects counts the engine running here accepts), so it
|
||||
# must be re-tested, not inherited; measurements here put seqs 2 and 3
|
||||
# inside each other's noise floor.
|
||||
grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
|
||||
# expert_tiered placement: vram_weights + the pinned host pool serve the
|
||||
# per-rank resident set TOGETHER, so cutting the VRAM slab to pay for the
|
||||
# KV pool moves the pool with it. KV is QUANTIZED fp8 (the engine takes
|
||||
# only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table is pinned
|
||||
# in host RAM per Frank's law.
|
||||
# KV pool moves the pool with it. KV is UNQUANTIZED bf16 (the engine takes
|
||||
# only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table rides
|
||||
# pinned host RAM per the placement knob.
|
||||
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_KV_DTYPE=bf16$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
|
||||
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
|
||||
grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_PREFIX_DISK_MIB=65536$' "$ENV_FILE"
|
||||
# The pinned ledger (2 x pool + 47.7 GiB PLE + prefix host) must be
|
||||
# under the ttm pages cap, and the env states the floor the serve
|
||||
# script enforces (>= 140,000 MiB — the kernel's own default cap, half of
|
||||
# RAM, fails it, which is the point) plus the value it raises the knob to
|
||||
# when a reboot reset it (the target must clear the floor with slack).
|
||||
# script enforces (>= 140,000 MiB — the current cap ~123 GiB fails it,
|
||||
# which is the point).
|
||||
grep -q '^FRANK_ARCHON_TTM_MIN_MIB=' "$ENV_FILE"
|
||||
ttmm=$(awk -F= '/^FRANK_ARCHON_TTM_MIN_MIB=/{print $2}' "$ENV_FILE")
|
||||
ttmt=$(awk -F= '/^FRANK_ARCHON_TTM_TARGET_MIB=/{print $2}' "$ENV_FILE")
|
||||
[ "$ttmt" -gt "$ttmm" ]
|
||||
pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
|
||||
ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_HOST_MIB=/{print $2}' "$ENV_FILE")
|
||||
# 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
|
||||
@@ -206,20 +319,15 @@ nosudo_bin() {
|
||||
}
|
||||
|
||||
@test "the per-card VRAM pair stays inside the measured card ceiling" {
|
||||
# The engine's OWN budget table, not a hand-picked reserve. Boot of
|
||||
# 2026-10-13 at the pair 24000/5120: card 32,624 MiB (31.86 GiB), budgets
|
||||
# claimed 29,120, `3.42 GiB is neither budgeted nor accounted — the
|
||||
# driver's context, the activation arena and --gpu-headroom-mib come out of
|
||||
# it; anything past those is throughput on the floor`, and `card 0 settled
|
||||
# at 2.01 GiB free against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and
|
||||
# 113.19 MiB kept for the static region)`. Claimable reserve above the
|
||||
# engine's floor: 2,058 - 209 = 1,849 MiB, so the ceiling is 30,969 MiB.
|
||||
# The 2026-10-11 bound of 29,120 ("keep >= 1 GiB free", card read as 31,856
|
||||
# MiB) was MY reserve, which the engine never asked for; the shipped pair
|
||||
# spends the difference and lands ~458 MiB above the floor.
|
||||
# Measured on Dankserver 2026-10-11: card 31,856 MiB, a boot consumes
|
||||
# 1,710 MiB OUTSIDE both budgets (driver context, the 990 MiB activation
|
||||
# arena, hardware queues, headroom, step staging) and the fp8 pair settled
|
||||
# at 1,379 MiB free. Claiming more than 29,120 MiB in total is an
|
||||
# allocation failure at graph capture, not a warning — so the KV pool and
|
||||
# the weights slab are one budget, and this seam is the guard.
|
||||
w=$(awk -F= '/^FRANK_ARCHON_VRAM_WEIGHTS_MIB=/{print $2}' "$ENV_FILE")
|
||||
kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
|
||||
[ $(( w + kv )) -le 30969 ]
|
||||
[ $(( w + kv )) -le 29120 ]
|
||||
}
|
||||
|
||||
@test "the serve script wires the engine flags from the env" {
|
||||
@@ -278,99 +386,6 @@ nosudo_bin() {
|
||||
! grep -q 'FRANK_ARCHON_TOOL_PARSER' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve script gates the image, the .rad payload, the template and the GPU nodes" {
|
||||
# The image gate is the digest ref itself; the payload gate checks the
|
||||
# .rad file and its size floor — a truncated convert would otherwise be
|
||||
# a guaranteed long-timeout boot instead of a fast failure. The ttm
|
||||
# preflight refuses to start under a pages cap below the pinned ledger.
|
||||
grep -q 'FRANK_ARCHON_IMAGE' "$SERVE"
|
||||
grep -q 'podman image exists' "$SERVE"
|
||||
grep -q 'FRANK_ARCHON_MODEL_FILE' "$SERVE"
|
||||
grep -q 'FRANK_ARCHON_RAD_MIN_MIB' "$SERVE"
|
||||
grep -q 'FRANK_ARCHON_TEMPLATE' "$SERVE"
|
||||
grep -q 'FRANK_ARCHON_TTM_MIN_MIB' "$SERVE"
|
||||
grep -q 'pages_limit' "$SERVE"
|
||||
grep -q 'pages_max' "$SERVE"
|
||||
grep -q '/dev/kfd' "$SERVE"
|
||||
grep -q -- '--device /dev/dri' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the ttm gate raises a reset pages cap and refuses when the write fails" {
|
||||
# The cap is a RUNTIME knob: the kernel default is half of RAM, so every
|
||||
# reboot resets it (measured: the 2026-10-12 23:17 power failure on
|
||||
# dankserver took it 200 GiB -> 125,988 MiB, and the preflight refusal was
|
||||
# re-run 6,919 times by Restart=always over ~20 h). Behaviour required of
|
||||
# the gate, in both directions:
|
||||
# knob below the floor + writable -> cap ends at the target, start passes
|
||||
# knob below the floor + unwritable -> cap stays low, start REFUSES by name
|
||||
local dir="$BATS_TEST_TMPDIR/ttm"
|
||||
mkdir -p "$dir"
|
||||
export FRANK_ARCHON_TTM_SYS_DIR="$dir"
|
||||
export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||
|
||||
echo 32253039 > "$dir/pages_limit" # 125,988 MiB: the kernel default here
|
||||
run bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(cat "$dir/pages_limit")" = "52428800" ] # 204,800 MiB in 4 KiB pages
|
||||
|
||||
echo 32253039 > "$dir/pages_limit"
|
||||
chmod 444 "$dir/pages_limit" # the write refused: cap cannot move
|
||||
# The refusal case must not have `sudo` on PATH: the gate's privileged
|
||||
# fallback runs `sudo -n`, and on a passwordless host root can write a 444
|
||||
# file, which would silently test nothing.
|
||||
run env PATH="$(nosudo_bin)" bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"ERROR: ttm pages cap is 125988 MiB"* ]]
|
||||
[ "$(cat "$dir/pages_limit")" = "32253039" ]
|
||||
}
|
||||
|
||||
@test "the ttm cap counts only the knobs the kernel exposes" {
|
||||
# An absent knob is not a zero: a zero-cap reading would refuse a healthy
|
||||
# host. The effective cap is the MIN over the knobs that exist; pages_max,
|
||||
# when a kernel exposes one, is the binding one.
|
||||
local dir="$BATS_TEST_TMPDIR/ttm2"
|
||||
mkdir -p "$dir"
|
||||
export FRANK_ARCHON_TTM_SYS_DIR="$dir"
|
||||
export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||
|
||||
echo 51200000 > "$dir/pages_limit" # 200,000 MiB, no pages_max knob
|
||||
run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "200000" ]
|
||||
|
||||
echo 30000000 > "$dir/pages_max" # 117,187 MiB: now the binding knob
|
||||
run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
|
||||
[ "$output" = "117187" ]
|
||||
|
||||
chmod 444 "$dir/pages_limit" "$dir/pages_max" # both writes refused
|
||||
run env PATH="$(nosudo_bin)" bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"ERROR: ttm pages cap is 117187 MiB"* ]]
|
||||
[ "$(cat "$dir/pages_max")" = "30000000" ]
|
||||
}
|
||||
|
||||
@test "the gate runs as a root pre-start hook, where the knob write is legal" {
|
||||
# The knob is root-owned 0644 and the service runs as User=dank, so the
|
||||
# gate CANNOT raise it from the service process. systemd's `+` prefix on
|
||||
# ExecStartPre runs as root even under User= — the canonical privileged
|
||||
# path — and the gate's own --raise mode is what that line calls.
|
||||
grep -q '^ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise$' "$UNIT"
|
||||
|
||||
# Behaviour of that entry point: sourced env values, knob raised, exit 0.
|
||||
local dir="$BATS_TEST_TMPDIR/ttm3" envf="$BATS_TEST_TMPDIR/ttm3.env"
|
||||
mkdir -p "$dir"
|
||||
printf 'FRANK_ARCHON_TTM_MIN_MIB=150000\nFRANK_ARCHON_TTM_TARGET_MIB=204800\n' > "$envf"
|
||||
echo 32253039 > "$dir/pages_limit"
|
||||
run env FRANK_ARCHON_ENV_FILE="$envf" FRANK_ARCHON_TTM_SYS_DIR="$dir" \
|
||||
bash "$TTM_LIB" --raise
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(cat "$dir/pages_limit")" = "52428800" ]
|
||||
|
||||
# Usage is a named failure, not a silent no-op.
|
||||
run bash "$TTM_LIB"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"--raise"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script clears stale containers and self-heals via podman wait" {
|
||||
# A crashed unit can leave the container name registered forever after;
|
||||
@@ -383,23 +398,18 @@ nosudo_bin() {
|
||||
grep -q 'exec podman wait "$FRANK_ARCHON_CONTAINER"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the model id matches the payload's fetch script (and the rollback's build script)" {
|
||||
# The deployed payload is the PUBLISHER's .rad: the fetch script names
|
||||
# the repo, the file, and the Hub's LFS sha256; nothing is converted on
|
||||
# the host for it. The in-house w4dn8 rollback keeps its own trio
|
||||
# (convert-swift15-w4dn8.sh + download-frank-archon-swift15.sh, and the
|
||||
# fp8-native era keeps build-frank-archon-rad.sh +
|
||||
# download-frank-archon.sh).
|
||||
@test "the model id matches the payload's build and fetch scripts" {
|
||||
# The payload dir/file are the convert script's output layout; the fetch
|
||||
# script must name the repo the convert stages from. (FP8-native era,
|
||||
# the rollback payload, keeps its own trio: build-frank-archon-rad.sh +
|
||||
# download-frank-archon.sh.)
|
||||
grep -q '^FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad$' "$ENV_FILE"
|
||||
grep -q 'StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe' "$DOWNLOAD"
|
||||
grep -q 'qwen3.8-next-flash-fp8-iq4r-moe.rad' "$DOWNLOAD"
|
||||
grep -q '0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20' "$DOWNLOAD"
|
||||
grep -q '^FRANK_ARCHON_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad$' "$ENV_FILE"
|
||||
grep -q 'qwen4exp-swift15-w4dn8-i8-hc8m.rad' "$BUILD"
|
||||
grep -q 'ukisai/Swift1.5-Qwen3.8-Flash-Next' "$DOWNLOAD"
|
||||
grep -q 'hfget.sh' "$DOWNLOAD"
|
||||
# The repo is UNGATED — no token may be required on the fetch path.
|
||||
! grep -q 'HF_TOKEN' "$DOWNLOAD"
|
||||
# Rollback provenance: the w4dn8 build script names its payload.
|
||||
grep -q 'qwen4exp-swift15-w4dn8-i8-hc8m.rad' "$BUILD"
|
||||
}
|
||||
|
||||
@test "no bare HIP-reserved env var is defined" {
|
||||
@@ -409,36 +419,3 @@ nosudo_bin() {
|
||||
! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
|
||||
! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the deployer gates the image, payload and containers, then health-checks" {
|
||||
# The deployer refuses when the image is absent from dank's rootless
|
||||
# storage or the .rad is missing/too small — then clears stale
|
||||
# containers (including the legacy identities), waits out the port,
|
||||
# relinks/enables/restarts the root unit and health-gates it.
|
||||
grep -q 'LLM_ARCHON_CONTAINER=frank-archon$' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_ARCHON_IMAGE' "$REMOTE_DEPLOY"
|
||||
grep -q 'podman image exists' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_ARCHON_MODEL_FILE' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_ARCHON_RAD_MIN_MIB' "$REMOTE_DEPLOY"
|
||||
grep -q 'frank-archon-qwen' "$REMOTE_DEPLOY"
|
||||
grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
|
||||
grep -q 'glm-probe' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-archon' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-archon.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-archon.service' "$REMOTE_DEPLOY"
|
||||
# The health gate is a bare curl: the HTTP listener binds AFTER the
|
||||
# full load (main.cpp — engine.start() then engine.serve()), so during
|
||||
# load the port refuses connections and a bare curl is honest. The
|
||||
# FreeToken-era body read ('status ok') is gone with that engine.
|
||||
grep -q 'curl -fsS http://127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
! grep -q '"status":"ok"' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deployer will not report green on a stray port-holder" {
|
||||
# The 2026-09-26 false green: a leftover probe held :7770 and satisfied
|
||||
# the health curl while the real unit was stopped. The deploy is only
|
||||
# green when the unit's own MainPID is live behind the health answer.
|
||||
grep -q 'systemctl show -p MainPID --value frank-archon.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'stray process holds the port' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@@ -10,11 +10,11 @@
|
||||
# exact strings the engine reads: the image DIGEST, the checkpoint/n-gram/
|
||||
# vision file names AND byte counts inside the /models mount, and the
|
||||
# tokenizer's flat-file shape (a HF cache snapshot's symlinks dangle inside
|
||||
# the container). A serve gate and a deployer gate that disagree is the
|
||||
# 2026-10-01 crash-loop; both sides are pinned here, plus the one engine
|
||||
# law the card is emphatic about: HALOGEN_SPEC_ADAPT=0 — the default
|
||||
# adaptive policy silently disables the MTP head per request and HALVES
|
||||
# decode.
|
||||
# the container). A serve gate and a deployer gate that disagree let a
|
||||
# deploy pass what the unit would refuse, so both sides are pinned here —
|
||||
# plus the one engine behavior the card is emphatic about:
|
||||
# HALOGEN_SPEC_ADAPT=0 — the default adaptive policy silently disables the
|
||||
# MTP head per request and HALVES decode.
|
||||
#
|
||||
# Everything is tested from what an operator or supervisor can observe —
|
||||
# the command line podman would run, the exit code, the deployer's error
|
||||
@@ -45,6 +45,13 @@ podman_stub() {
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
images) printf '%s\n' "${STUB_DIGESTS:-}" ;;
|
||||
image)
|
||||
# `podman image exists REF` — exit 1 when storage lacks the ref,
|
||||
# which is exactly the primitive the shared gate library calls.
|
||||
case " ${STUB_DIGESTS:-} " in
|
||||
*" $3 "*) exit 0 ;;
|
||||
*) exit 1 ;;
|
||||
esac ;;
|
||||
rm) exit 0 ;;
|
||||
run|wait) echo "PODMAN $*"; exit "${STUB_WAIT_RC:-0}" ;;
|
||||
*) exit 0 ;;
|
||||
@@ -110,10 +117,11 @@ good_env() {
|
||||
|
||||
@test "the head stays on: adaptive spec must stay 0 and the window shape pinned" {
|
||||
# The card is emphatic: the default adaptive policy turns the MTP head
|
||||
# off for whole requests after low acceptance and HALVES decode. The
|
||||
# two-slot law does NOT carry over from the 27B engine — on this line
|
||||
# batching and speculation coexist (the card's quickstart ships 2
|
||||
# slots WITH the head), so 2 is concurrent capacity, not a tax.
|
||||
# off for whole requests after low acceptance and HALVES decode. A
|
||||
# "slots > 1 disables drafting" restriction belongs to the engine
|
||||
# generation that imposed it — on this line batching and speculation
|
||||
# coexist (the card's quickstart ships 2 slots WITH the head), so 2 is
|
||||
# concurrent capacity, not a tax.
|
||||
grep -qE '^HALOGEN_SPEC_ADAPT=0$' "$ENV_FILE"
|
||||
grep -qE '^HALOGEN_KV_SLOTS=2$' "$ENV_FILE"
|
||||
# The cap/timeout pair is coupled (FLAGS.md table): a request at the
|
||||
@@ -179,7 +187,7 @@ good_env() {
|
||||
|
||||
@test "the serve script refuses a checkpoint of the wrong byte count" {
|
||||
# A truncated curl exit-0 artifact must not boot; the Hub's byte
|
||||
# count is the contract (fleet fetch law).
|
||||
# count is the contract, any other size is a broken artifact.
|
||||
podman_stub
|
||||
good_env
|
||||
run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
|
||||
@@ -197,7 +205,7 @@ good_env() {
|
||||
run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
|
||||
FRANK_DRAGOON_NGRAM_TABLE=absent.hgn "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"n-gram-table"*"missing"* ]]
|
||||
[[ "$output" == *"frank-dragoon: n-gram: missing at $TEST_TMP/absent.hgn"* ]]
|
||||
run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
|
||||
FRANK_DRAGOON_NGRAM_BYTES=42 "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
@@ -213,7 +221,7 @@ good_env() {
|
||||
run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
|
||||
FRANK_DRAGOON_VISION_TOWER=absent.hgn "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"vision-tower"*"missing"* ]]
|
||||
[[ "$output" == *"frank-dragoon: vision: missing at $TEST_TMP/absent.hgn"* ]]
|
||||
run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
|
||||
FRANK_DRAGOON_VISION_BYTES=42 "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
@@ -280,16 +288,16 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
! grep -q 'PODMAN wait' "$TEST_TMP/serve.log"
|
||||
}
|
||||
|
||||
@test "the shipped env keeps the harness fit law inside the ceiling" {
|
||||
@test "the shipped env keeps the harness fit margin inside the ceiling" {
|
||||
# 262144 (the checkpoint's native window) is the per-slot hard ceiling
|
||||
# (a longer prompt is a 400 naming the limit, never a silent
|
||||
# truncation). The harness clamps max_tokens = min(45056, 217088 − est
|
||||
# − 4096), compacts at 85 % of its declared 217088 window, and its
|
||||
# chars/4 est UNDER-counts the rendered prompt by ~3 % on code/agent
|
||||
# traffic (measured 4,450 tokens at est 152,938 on 2026-10-07 — with
|
||||
# window == ceiling, every request over ~146k est was a guaranteed
|
||||
# 400 before compaction could fire at 85 %). The 45k gap between the
|
||||
# harness window and the engine ceiling must absorb that error at
|
||||
# traffic — so a harness window equal to the engine ceiling would make
|
||||
# every long request a guaranteed 400 before compaction could fire.
|
||||
# The 45k gap between the harness window and the engine ceiling must
|
||||
# absorb that error at
|
||||
# both monotone maxima of est + budget (est ≤ 217088 − 4096):
|
||||
# full budget at the compaction threshold: 1.03*184524 + 45056
|
||||
# clamped budget at the clamp's zero point: 1.03*212992 + 0
|
||||
@@ -328,11 +336,6 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
grep -qF "LLM_DRAGOON_CKPT_BYTES=\"\${LLM_DRAGOON_CKPT_BYTES:-$bytes}\"" "$DEPLOYER"
|
||||
}
|
||||
|
||||
@test "the deployer points a missing artifact at the flashnext fetch script" {
|
||||
grep -qE 'download-frank-dragoon-flashnext\.sh' "$DEPLOYER"
|
||||
grep -qE 'podman pull' "$DEPLOYER"
|
||||
}
|
||||
|
||||
@test "the stray sweep cannot take frank-zealot or the deployer down" {
|
||||
# frank-zealot also listens on this host (7771, llama-server) and the
|
||||
# deployer's own command line contains "frank-dragoon"; only the FULL
|
||||
@@ -344,8 +347,9 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
}
|
||||
|
||||
@test "the deployed unit keeps the fleet's restart and boot-persistence shape" {
|
||||
grep -qE '^Restart=(always|on-failure)$' "$UNIT"
|
||||
grep -qE '^WantedBy=' "$UNIT"
|
||||
# Restart=always and the WantedBy value are the FLEET parity seam
|
||||
# (tests/llm_units.bats); what is endpoint-specific is which script the
|
||||
# unit runs, under whom, and what a stop must reap.
|
||||
grep -qE '^ExecStart=.*/serve-frank-dragoon\.sh$' "$UNIT"
|
||||
grep -qE '^User=dank$' "$UNIT"
|
||||
grep -qE '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
|
||||
@@ -85,10 +85,12 @@ run_stage() {
|
||||
[ -f "$ENV_FILE" ]
|
||||
}
|
||||
|
||||
@test "frank-zealot is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
@test "frank-zealot runs unprivileged with the memlock budget and its env file" {
|
||||
# Restart policy, RestartSec and WantedBy are the FLEET parity seam
|
||||
# (tests/llm_units.bats). What is endpoint-specific: CUDA pins its
|
||||
# staging buffers, so the memlock budget is load-bearing even though the
|
||||
# weights are copied into VRAM, and the unit must name the serve script
|
||||
# and the env file that sizes it.
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-zealot.sh$' "$UNIT"
|
||||
@@ -253,26 +255,20 @@ run_stage() {
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing runtime, GGUF or projector" {
|
||||
# The refusal is the shared gate library's: it names the artifact class,
|
||||
# the exact path, and the script that produces it.
|
||||
rm -f "$MODEL_DIR/$MM_STUB"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-zealot: missing"* ]]
|
||||
[[ "$output" == *"frank-zealot:"* ]]
|
||||
[[ "$output" == *"mmproj"*"missing"*"$MM_STUB"* ]]
|
||||
[[ "$output" == *"download-frank-zealot.sh"* ]]
|
||||
|
||||
rm -f "$RUNTIME_DIR/bin/llama-server"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-zealot: missing"* ]]
|
||||
}
|
||||
|
||||
@test "the deploy handler gates on binary, GGUF and projector, then MainPID-hardens health" {
|
||||
grep -q 'deploy_llm_zealot' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_ZEALOT_RUNTIME' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_ZEALOT_MODEL_FILE' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_ZEALOT_MMPROJ' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-zealot' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-zealot.service' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'frank-zealot.service has no MainPID' "$REMOTE_DEPLOY"
|
||||
[[ "$output" == *"runtime"*"missing"*"llama-server"* ]]
|
||||
[[ "$output" == *"build-llama-master-cuda.sh"* ]]
|
||||
}
|
||||
|
||||
@test "the two endpoints keep separate ports and the large handler never pkills llama-server" {
|
||||
|
||||
@@ -4,6 +4,12 @@
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
# skip_if_no_podman — skip the test when podman is absent (the image gate
|
||||
# asks real storage a real question)
|
||||
skip_if_no_podman() {
|
||||
command -v podman >/dev/null 2>&1 || skip "podman is not installed"
|
||||
}
|
||||
|
||||
# yaml_parse_ok <file> — assert a compose file parses as valid YAML
|
||||
yaml_parse_ok() {
|
||||
python3 - "$1" <<'PYEOF'
|
||||
|
||||
@@ -101,7 +101,10 @@ webui_service_yaml() {
|
||||
}
|
||||
|
||||
@test "remote deployer accepts and dispatches hermes-webui" {
|
||||
grep -q 'dankclaw:hermes-webui' "$REMOTE_DEPLOY"
|
||||
# Acceptance is the inventory (rank 3): the old literal allow-list in
|
||||
# the remote runner is retired.
|
||||
run python3 "$REPO_ROOT/scripts/service-map.py" allows --host dankclaw --service hermes-webui
|
||||
[ "$status" -eq 0 ]
|
||||
# Dispatched to the unit handler, never to compose: deploy_hermes installs
|
||||
# the quadlets and retires compose-era containers.
|
||||
grep -q 'hermes|hermes-dashboard|hermes-webui) deploy_hermes' "$REMOTE_DEPLOY"
|
||||
@@ -111,5 +114,7 @@ webui_service_yaml() {
|
||||
}
|
||||
|
||||
@test "top-level deployer accepts hermes-webui as a Dankclaw service" {
|
||||
grep -q 'dankclaw:hermes-webui' "$DEPLOY"
|
||||
# deploy.sh validates --service through the inventory's allows.
|
||||
run python3 "$REPO_ROOT/scripts/service-map.py" allows --host dankclaw --service hermes-webui
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@@ -51,7 +51,7 @@ setup() {
|
||||
# domain never enters the chain (it redirects at the TLS edge first).
|
||||
local block
|
||||
block=$(awk '/^www\.franksdank\.cloud[[:space:]]*\{/,/^}/' "$CADDY")
|
||||
echo "$block" | grep -q 'reverse_proxy anubis:8923'
|
||||
echo "$block" | grep -q 'import guarded'
|
||||
# forwards accepted traffic to; Host survives the hop, so per-vhost
|
||||
# routing resumes there — guarded vhosts proxy the whole request
|
||||
# through Authentik, which forwards authenticated traffic back on the
|
||||
|
||||
Executable
+136
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the deployer's LLM handlers are the twin of the serve scripts'
|
||||
# artifact gates (the 2026-10-01 crash-loop: a green deploy against a
|
||||
# missing payload left a restart-looping unit and a dark endpoint).
|
||||
#
|
||||
# These tests EXECUTE scripts/deploy-remote.sh's LLM handlers in a sandbox
|
||||
# (tests/deploy_sandbox.bash): every host touchpoint is a stub, the artifacts
|
||||
# are fixtures, and the assertions are exit status, the refusal on stderr,
|
||||
# and what the handler asked systemd to do. A handler that started the unit
|
||||
# before checking its artifacts would pass a text-grep seam and fail here.
|
||||
|
||||
load helpers
|
||||
load deploy_sandbox
|
||||
|
||||
setup() { sandbox_setup; }
|
||||
teardown() { sandbox_teardown; }
|
||||
|
||||
@test "the dragoon handler refuses when the engine image is not in storage" {
|
||||
export STUB_IMAGE=
|
||||
run_handler dankclaw frank-dragoon
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-dragoon: image:"*"not in podman storage"* ]]
|
||||
[ "$(restarts_requested)" = "0" ]
|
||||
}
|
||||
|
||||
@test "the dragoon handler refuses a truncated checkpoint, pointing at the fetch" {
|
||||
truncate -s 9 "$STAGE/models/ck.hgn"
|
||||
run_handler dankclaw frank-dragoon
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"checkpoint: is 9 bytes, expected 10"* ]]
|
||||
# The refusal is only useful if it names the remedy: the fetch script
|
||||
# re-verifies byte counts and hashes.
|
||||
[[ "$output" == *"download-frank-dragoon-flashnext.sh"* ]]
|
||||
[ "$(restarts_requested)" = "0" ]
|
||||
}
|
||||
|
||||
@test "the dragoon handler refuses a symlinked tokenizer file" {
|
||||
# Snapshot layouts link their members into place, and the link dangles
|
||||
# inside the container mount — only regular files are servable.
|
||||
rm "$STAGE/models/tokenizer/vocab.json"
|
||||
ln -s /nowhere/vocab.json "$STAGE/models/tokenizer/vocab.json"
|
||||
run_handler dankclaw frank-dragoon
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-dragoon: tokenizer: $STAGE/models/tokenizer/vocab.json is missing or a symlink"* ]]
|
||||
[ "$(restarts_requested)" = "0" ]
|
||||
}
|
||||
|
||||
@test "the dragoon handler starts the unit once every artifact gate passes" {
|
||||
run_handler dankclaw frank-dragoon
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'restart frank-dragoon.service' "$STAGE/systemctl.log"
|
||||
grep -qx 'rm -f frank-dragoon frank-dragoon-halogen' "$STAGE/podman.log"
|
||||
}
|
||||
|
||||
@test "the zealot handler refuses when the llama.cpp runtime is not executable" {
|
||||
# The runtime is an operator-built binary; a tree that compiled but lost
|
||||
# its exec bit must refuse exactly like a missing one.
|
||||
rm "$STAGE/models/llama-server"
|
||||
run_handler dankclaw frank-zealot
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-zealot: runtime: missing at $STAGE/models/llama-server"* ]]
|
||||
[[ "$output" == *"cmake"* ]]
|
||||
[ "$(restarts_requested)" = "0" ]
|
||||
}
|
||||
|
||||
@test "the zealot handler refuses a missing projector — a text-only endpoint is not deployable" {
|
||||
# The projector IS the vision capability: deploying without it silently
|
||||
# ships a text-only endpoint.
|
||||
rm "$STAGE/models/mmproj.gguf"
|
||||
run_handler dankclaw frank-zealot
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-zealot: mmproj: missing at $STAGE/models/mmproj.gguf"* ]]
|
||||
[ "$(restarts_requested)" = "0" ]
|
||||
}
|
||||
|
||||
@test "the zealot handler starts the unit once every artifact gate passes" {
|
||||
run_handler dankclaw frank-zealot
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'restart frank-zealot.service' "$STAGE/systemctl.log"
|
||||
}
|
||||
|
||||
@test "the archon handler refuses when the radiance image is not in storage" {
|
||||
export STUB_IMAGE=
|
||||
run_handler dankserver frank-archon
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-archon: image:"*"not in podman storage"* ]]
|
||||
[[ "$output" == *"podman pull"* ]]
|
||||
[ "$(restarts_requested)" = "0" ]
|
||||
}
|
||||
|
||||
@test "the archon handler refuses a .rad below the env's size floor" {
|
||||
truncate -s 1048576 "$STAGE/models/payload.rad" # 1 MiB, far under the floor
|
||||
archon_payload_env
|
||||
export STUB_IMAGE="$ARCHON_IMAGE"
|
||||
run_handler dankserver frank-archon
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-archon: .rad: is 1 MiB, expected >= "* ]]
|
||||
[[ "$output" == *"convert-swift15-w4dn8.sh"* ]]
|
||||
[ "$(restarts_requested)" = "0" ]
|
||||
}
|
||||
|
||||
@test "the archon handler clears the legacy container identities and relinks the unit" {
|
||||
# The retired identities (the R9V container, the ad hoc probe, the glm
|
||||
# experiment) hold the same GPUs, so a deploy that leaves one running
|
||||
# leaves the new engine unable to bind. The unit link is the root-unit
|
||||
# shape's whole install.
|
||||
archon_payload_env 2
|
||||
export STUB_IMAGE="$ARCHON_IMAGE"
|
||||
run_handler dankserver frank-archon
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'restart frank-archon.service' "$STAGE/systemctl.log"
|
||||
[ -L "$STAGE/systemd/frank-archon.service" ]
|
||||
local ctr
|
||||
for ctr in frank-archon frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe; do
|
||||
grep -qx "rm -f $ctr" "$STAGE/podman.log"
|
||||
done
|
||||
}
|
||||
|
||||
@test "no LLM handler reports green when its unit has no MainPID behind the port" {
|
||||
# The 2026-09-26 false green: a leftover probe held :7770 and satisfied
|
||||
# the health curl while the real unit was stopped.
|
||||
archon_payload_env 2
|
||||
export STUB_IMAGE="$ARCHON_IMAGE"
|
||||
export STUB_MAIN_PID=0
|
||||
run_handler dankserver frank-archon
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-archon.service has no MainPID"* ]]
|
||||
}
|
||||
|
||||
@test "a refused LLM handler leaves its service pending in the deploy state" {
|
||||
export STUB_IMAGE=
|
||||
run_handler dankserver frank-archon
|
||||
[ "$status" -ne 0 ]
|
||||
grep -q 'status=pending' "$STAGE/state/frank-archon"
|
||||
[ ! -e "$STAGE/state/frank-zealot" ]
|
||||
}
|
||||
Executable
+172
@@ -0,0 +1,172 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the shared artifact gates every LLM endpoint runs before it starts.
|
||||
#
|
||||
# The rule these encode: a serve script and the deployer must refuse the
|
||||
# SAME missing/truncated artifact, and they must refuse it the same way.
|
||||
# Two hand-written copies of the same gates drift apart until a deploy
|
||||
# passes what the unit would refuse; the library is the one place the
|
||||
# refusal lives, and both callers source it.
|
||||
#
|
||||
# Everything here is observable: arguments in, exit status and stderr out.
|
||||
# Nothing inspects the library's internals.
|
||||
|
||||
load helpers
|
||||
|
||||
LIB="$REPO_ROOT/scripts/lib/llm-gates.sh"
|
||||
|
||||
setup() {
|
||||
TMP="$(mktemp -d)"
|
||||
# A fixture with a known, independent size: 12 bytes.
|
||||
printf '0123456789ab' > "$TMP/artifact.bin"
|
||||
mkdir -p "$TMP/flat"
|
||||
touch "$TMP/flat/tokenizer.json" "$TMP/flat/config.json"
|
||||
}
|
||||
|
||||
teardown() {
|
||||
rm -rf "$TMP"
|
||||
}
|
||||
|
||||
# call <invocation> — run one gate in a fresh shell under `set -e`, exactly
|
||||
# as a serve script sources the library, and capture status + both streams.
|
||||
call() {
|
||||
run env GATE_PREFIX="${GATE_PREFIX:-testgate}" PATH="$PATH" \
|
||||
bash -c "set -euo pipefail; source '$LIB'; $1"
|
||||
}
|
||||
|
||||
@test "a present artifact passes the file gate with no output" {
|
||||
GATE_PREFIX=testgate call "require_file '$TMP/artifact.bin' checkpoint 'fetch it'"
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "an absent artifact fails the file gate naming the path and the remedy" {
|
||||
call "require_file '$TMP/absent.bin' checkpoint 'run scripts/download-frank-archon-swift15.sh'"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"checkpoint: missing"* ]]
|
||||
[[ "$output" == *"$TMP/absent.bin"* ]]
|
||||
[[ "$output" == *"run scripts/download-frank-archon-swift15.sh"* ]]
|
||||
}
|
||||
|
||||
@test "the file gate refuses a directory where an artifact is required" {
|
||||
# The serve gate is `-f`, not `-e`: a directory named like the payload
|
||||
# (an unpacked staging dir) must not boot the engine.
|
||||
call "require_file '$TMP/flat' checkpoint 'fetch it'"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"checkpoint"* ]]
|
||||
}
|
||||
|
||||
@test "a symlinked artifact is refused: a HF snapshot link dangles in the container mount" {
|
||||
ln -s "$TMP/artifact.bin" "$TMP/linked.bin"
|
||||
call "require_file '$TMP/linked.bin' checkpoint 'fetch it'"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"checkpoint"*"missing"* ]]
|
||||
}
|
||||
|
||||
@test "the byte gate accepts the exact contracted size" {
|
||||
call "require_bytes '$TMP/artifact.bin' checkpoint 12 'fetch it'"
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "the byte gate refuses a size mismatch quoting both the real and the expected size" {
|
||||
# The Hub's declared byte count is the contract; curl writes the error
|
||||
# body and exits 0, so a truncation is invisible without this gate.
|
||||
call "require_bytes '$TMP/artifact.bin' checkpoint 42 'run the fetch script'"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"12 bytes"* ]]
|
||||
[[ "$output" == *"expected 42"* ]]
|
||||
[[ "$output" == *"run the fetch script"* ]]
|
||||
}
|
||||
|
||||
@test "the byte gate is an equality contract, not a floor" {
|
||||
# A LARGER file is not a better artifact: it is a different artifact
|
||||
# (the compacted r2 of a checkpoint is byte-exact, not "at least").
|
||||
call "require_bytes '$TMP/artifact.bin' checkpoint 6 'fetch it'"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"expected 6"* ]]
|
||||
}
|
||||
|
||||
@test "the minimum-size gate accepts the threshold itself" {
|
||||
dd if=/dev/zero of="$TMP/two-mib.rad" bs=1048576 count=2 status=none
|
||||
call "require_min_mib '$TMP/two-mib.rad' container 2 'rebuild it'"
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "the minimum-size gate refuses below the threshold naming both sizes" {
|
||||
dd if=/dev/zero of="$TMP/one-mib.rad" bs=1048576 count=1 status=none
|
||||
call "require_min_mib '$TMP/one-mib.rad' container 2 'rebuild it'"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"1 MiB"* ]]
|
||||
[[ "$output" == *"2 MiB"* ]]
|
||||
[[ "$output" == *"rebuild it"* ]]
|
||||
}
|
||||
|
||||
@test "the flat-directory gate accepts a directory of regular files" {
|
||||
call "require_dir_files '$TMP/flat' tokenizer 'fetch the tokenizer' tokenizer.json config.json"
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "the flat-directory gate names the member that is absent" {
|
||||
call "require_dir_files '$TMP/flat' tokenizer 'fetch the tokenizer' tokenizer.json vocab.json"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"tokenizer"*"missing"* ]]
|
||||
[[ "$output" == *"vocab.json"* ]]
|
||||
}
|
||||
|
||||
@test "the flat-directory gate refuses a symlinked member" {
|
||||
# A HuggingFace cache snapshot's entries are symlinks into blobs/ and
|
||||
# dangle inside the container mount even when the directory is real.
|
||||
ln -s /nowhere/x "$TMP/flat/vocab.json"
|
||||
call "require_dir_files '$TMP/flat' tokenizer 'fetch the tokenizer' tokenizer.json vocab.json"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"vocab.json"*"missing or a symlink"* ]]
|
||||
}
|
||||
|
||||
@test "the flat-directory gate refuses a missing directory" {
|
||||
call "require_dir_files '$TMP/nope' tokenizer 'fetch the tokenizer' tokenizer.json"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"tokenizer: missing"* ]]
|
||||
}
|
||||
|
||||
@test "the flat-directory gate refuses a symlinked directory" {
|
||||
# The snapshot directory itself can be the link (a HF cache blobs/
|
||||
# sibling), and a link into the container mount dangles there too.
|
||||
ln -s "$TMP/flat" "$TMP/linked-dir"
|
||||
call "require_dir_files '$TMP/linked-dir' tokenizer 'fetch the tokenizer' tokenizer.json"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"tokenizer: missing"* ]]
|
||||
}
|
||||
|
||||
@test "the image gate refuses a ref the storage does not hold" {
|
||||
# A name@sha256 ref exists only if THAT manifest was pulled, so the
|
||||
# gate is the ref itself — one command, no grep race.
|
||||
skip_if_no_podman
|
||||
call "require_image 'example.invalid/nothing@sha256:0000000000000000000000000000000000000000000000000000000000000000' 'podman pull it'"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"not in podman storage"* ]]
|
||||
[[ "$output" == *"podman pull it"* ]]
|
||||
}
|
||||
|
||||
@test "every failure names the endpoint the operator is standing in front of" {
|
||||
GATE_PREFIX=frank-dragoon call "require_file '$TMP/absent.bin' checkpoint 'fetch it'"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == frank-dragoon:* ]]
|
||||
}
|
||||
|
||||
@test "a failing gate stops the caller from reaching the runtime" {
|
||||
# The composition property that matters: the gate is not advisory. A
|
||||
# serve script that calls it and then execs the engine must die first.
|
||||
cat > "$TMP/serve-like.sh" <<EOF
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
source '$LIB'
|
||||
require_file '$TMP/absent.bin' checkpoint 'fetch it'
|
||||
echo RUNTIME-EXECUTED
|
||||
EOF
|
||||
run bash "$TMP/serve-like.sh"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" != *"RUNTIME-EXECUTED"* ]]
|
||||
[[ "$output" == *"checkpoint: missing"* ]]
|
||||
}
|
||||
@@ -80,3 +80,38 @@ unit_repo_path() {
|
||||
}
|
||||
done < <(find "$REPO_ROOT/scripts" -name "frank-*.service" -type f | grep -v AGENTTRASH | sort)
|
||||
}
|
||||
|
||||
@test "every LLM root unit is self-healing against a voluntary exit" {
|
||||
# Restart=always, not on-failure. Every endpoint here is a container
|
||||
# runner (podman run) or a server whose front end can exit 0 on a
|
||||
# control-plane event; systemd treats exit 0 as a CLEAN stop and
|
||||
# on-failure leaves the endpoint dark until the next boot, with the
|
||||
# unit reporting success. No LLM unit may ship the weaker policy: a
|
||||
# single endpoint on on-failure is a daemon that goes dark silently.
|
||||
local unit
|
||||
while IFS= read -r unit; do
|
||||
grep -qx 'Restart=always' "$unit" || {
|
||||
echo "$(basename "$unit"): $(grep '^Restart=' "$unit") (must be always)"
|
||||
return 1
|
||||
}
|
||||
done < <(find "$REPO_ROOT/scripts" -name "frank-*.service" -type f | grep -v AGENTTRASH | sort)
|
||||
}
|
||||
|
||||
@test "every LLM root unit restarts promptly and starts at boot" {
|
||||
# RestartSec=10 is the fleet's cadence: fast enough that a transient
|
||||
# fault is invisible to traffic, slow enough that a hard fault does not
|
||||
# spin the engine (each radiance boot re-pins ~147 GiB).
|
||||
# WantedBy=multi-user.target is boot persistence for a ROOT unit —
|
||||
# default.target is the USER-manager's target and means nothing here.
|
||||
local unit
|
||||
while IFS= read -r unit; do
|
||||
grep -qE '^RestartSec=([1-9]|1[0-9]|20)$' "$unit" || {
|
||||
echo "$(basename "$unit"): no RestartSec in 1..20"
|
||||
return 1
|
||||
}
|
||||
grep -qx 'WantedBy=multi-user.target' "$unit" || {
|
||||
echo "$(basename "$unit"): $(grep '^WantedBy=' "$unit")"
|
||||
return 1
|
||||
}
|
||||
done < <(find "$REPO_ROOT/scripts" -name "frank-*.service" -type f | grep -v AGENTTRASH | sort)
|
||||
}
|
||||
|
||||
@@ -112,11 +112,11 @@ GLUETUN="$REPO_ROOT/dankclaw/searxng/quadlet/gluetun.container"
|
||||
# Public edge: the vhost must proxy to the challenge proxy, not straight to the app.
|
||||
awk '
|
||||
/^marreta\.franksdank\.cloud \{/ { inblk=1 }
|
||||
inblk && /reverse_proxy anubis:8923/ { challenge=1 }
|
||||
inblk && /import guarded/ { challenge=1 }
|
||||
inblk && /^\}/ { inblk=0 }
|
||||
END { exit challenge ? 0 : 1 }
|
||||
' "$REPO_ROOT/dankclaw/Caddyfile" \
|
||||
|| { echo "marreta vhost must reverse_proxy anubis:8923"; return 1; }
|
||||
|| { echo "marreta vhost must import the guarded chain (the one named Anubis snippet)"; return 1; }
|
||||
}
|
||||
|
||||
@test "the Caddyfile post-auth listener routes marreta to the published 8082" {
|
||||
|
||||
@@ -433,11 +433,14 @@ assert any('ALERTS' in e for e in exprs), exprs
|
||||
grep -q -- '--web.listen-address=127.0.0.1:9100' "$unit"
|
||||
grep -q -- '--web.listen-address=192.168.0.235:9100' "$unit"
|
||||
! grep -qE -- '--web\.listen-address=(0\.0\.0\.0|:9100|\[::\])' "$unit"
|
||||
# One owner per container name: the compose service must be gone, or two
|
||||
# definitions fight over the same container.
|
||||
! grep -qE '^\s+node-exporter:' "$REPO_ROOT/dankserver/compose.yaml"
|
||||
# The deploy path has to know about it on this target.
|
||||
grep -q 'dankserver:node-exporter' "$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
# One owner per container name: Dankserver has no compose file at all
|
||||
# since the 2026-10-13 retirement (it lived beside diverged quadlet
|
||||
# mounts for weeks — the two-owners trap by construction).
|
||||
[ ! -e "$REPO_ROOT/dankserver/compose.yaml" ]
|
||||
# The deploy path has to know about it on this target — acceptance
|
||||
# is the inventory (rank 3), not a literal list in the runner.
|
||||
run python3 "$REPO_ROOT/scripts/service-map.py" allows --host dankserver --service node-exporter
|
||||
[ "$status" -eq 0 ]
|
||||
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver \
|
||||
--paths dankserver/observability/quadlet/node-exporter.container
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
Executable
+138
@@ -0,0 +1,138 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the service inventory (rank 3). shared/services.yml is the one
|
||||
# inventory of deployable services; scripts/service-map.py is its only
|
||||
# reader, and every consumer (deploy-plan.py, deploy-remote.sh's
|
||||
# allow-list, deploy.sh's --service gate, test-changed.sh's seam mapper,
|
||||
# the Pi-hole hosts render, the Prometheus scrape facts, the homepage
|
||||
# probes) goes through it. These tests pin the inventory's integrity,
|
||||
# the planner's selection against a captured fixture, and the rendered
|
||||
# artifacts' drift against the tracked copies.
|
||||
|
||||
load helpers
|
||||
|
||||
@test "inventory: schema, unit claims, path claims, compose ownership, and render drift are sound" {
|
||||
run python3 "$REPO_ROOT/scripts/service-map.py" check
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "inventory: the deployers' allow-list is the inventory" {
|
||||
# Every service the inventory places on a host must pass "allows" for
|
||||
# that host, and a name the inventory never places must be refused.
|
||||
for host in dankclaw dankserver frankbook-pro; do
|
||||
while IFS= read -r svc; do
|
||||
[ -n "$svc" ] || continue
|
||||
run python3 "$REPO_ROOT/scripts/service-map.py" allows --host "$host" --service "$svc"
|
||||
[ "$status" -eq 0 ] || { echo "allows refused $host:$svc"; return 1; }
|
||||
done < <(python3 "$REPO_ROOT/scripts/service-map.py" services --host "$host")
|
||||
done
|
||||
run python3 "$REPO_ROOT/scripts/service-map.py" allows --host dankserver --service sonarr
|
||||
[ "$status" -ne 0 ] # the *arr stack lives on frankbook-pro, not dankserver
|
||||
run python3 "$REPO_ROOT/scripts/service-map.py" allows --host dankclaw --service frank-archon
|
||||
[ "$status" -ne 0 ] # frank-archon lives on dankserver
|
||||
}
|
||||
|
||||
@test "inventory: the planner reproduces the pinned selection fixture" {
|
||||
# tests/fixtures/service-plan-golden.tsv is the characterization of
|
||||
# path -> service selection across every tracked path, every target,
|
||||
# and the multi-path combos the old hand-written planner was fed.
|
||||
python3 - "$REPO_ROOT" <<'PYEOF'
|
||||
import importlib.util
|
||||
import sys
|
||||
|
||||
root = sys.argv[1]
|
||||
spec = importlib.util.spec_from_file_location("sm", root + "/scripts/service-map.py")
|
||||
sm = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(sm)
|
||||
inv = sm.Inventory.load(root + "/shared/services.yml")
|
||||
bad = []
|
||||
with open(root + "/tests/fixtures/service-plan-golden.tsv") as fh:
|
||||
for line in fh:
|
||||
host, paths, want = line.rstrip("\n").split("\t")
|
||||
got = ",".join(sm.select(inv, host, paths.split("+")))
|
||||
if got != want:
|
||||
bad.append(f"{host} {paths}\n want: {want}\n got: {got}")
|
||||
if bad:
|
||||
print(f"{len(bad)} selection mismatches:", *bad[:20], sep="\n")
|
||||
sys.exit(1)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
@test "inventory: deploy-plan.py keeps its CLI contract on top of service-map" {
|
||||
# stdin paths, positional paths, and the --repo/--base/--head diff mode
|
||||
# all route to the same selection.
|
||||
run bash -c "cd '$REPO_ROOT' && printf '%s\n' dankclaw/searxng/quadlet/gluetun.container | \
|
||||
scripts/deploy-plan.py --target dankclaw"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "${lines[0]}" = "gluetun" ]
|
||||
run "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver scripts/frank-archon.env
|
||||
[ "$status" -eq 0 ]
|
||||
[ "${lines[0]}" = "frank-archon" ]
|
||||
run "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw --repo "$REPO_ROOT" --base HEAD --head HEAD
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "inventory: the Pi-hole hosts file is a rendered artifact" {
|
||||
# The tracked addn-hosts must be byte-identical to the render; editing
|
||||
# it by hand is drift the inventory check also catches.
|
||||
run bash -c "cd '$REPO_ROOT' && python3 scripts/service-map.py render pihole-hosts | \
|
||||
diff -u dankclaw/pihole/addn-hosts -"
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "inventory: changed-test seams for a service unit are the service's own" {
|
||||
# The rank-3 contract: a unit change selects the OWNING service's
|
||||
# declared seams, not the old shotgun of unrelated quadlet seams.
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths dankclaw/gitea/quadlet/gitea.network
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"tests/caddyfile.bats"* ]]
|
||||
[[ "$output" == *"tests/deployment_scoping.bats"* ]]
|
||||
[[ "$output" != *"tests/open_webui_quadlet.bats"* ]]
|
||||
[[ "$output" != *"tests/searxng_quadlet.bats"* ]]
|
||||
}
|
||||
|
||||
@test "inventory: every tracked quadlet unit is claimed by exactly one service" {
|
||||
# check units covers claim conflicts; this pins the census size so a
|
||||
# unit added WITHOUT an inventory entry (unclaimed) fails the seam.
|
||||
python3 - "$REPO_ROOT" <<'PYEOF'
|
||||
import importlib.util
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import PurePosixPath
|
||||
|
||||
root = sys.argv[1]
|
||||
spec = importlib.util.spec_from_file_location("sm", root + "/scripts/service-map.py")
|
||||
sm = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(sm)
|
||||
inv = sm.Inventory.load(root + "/shared/services.yml")
|
||||
graph = sm.graph_for(tuple(inv.hosts))
|
||||
owned = {u for u in graph.units}
|
||||
problems = []
|
||||
tracked_units = [
|
||||
p for p in sm.tracked_files()
|
||||
if PurePosixPath(p).name.endswith((".container", ".network")) and not sm.is_trash(p)
|
||||
]
|
||||
for path in tracked_units:
|
||||
parts = PurePosixPath(path).parts
|
||||
name = parts[-1]
|
||||
key = (parts[0], name.rsplit(".", 1)[0])
|
||||
if key not in graph.units and key not in graph.networks:
|
||||
problems.append(f"unclaimed unit-file path: {path}")
|
||||
if problems:
|
||||
print("\n".join(problems))
|
||||
sys.exit(1)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
@test "inventory: the node-exporter unit is a per-host rendered template" {
|
||||
# Both quadlets are rendered artifacts: the render — not a hand-edited
|
||||
# mirror kept in step by a comment — defines the unit on every host.
|
||||
for host in dankclaw dankserver; do
|
||||
run bash -c "cd '$REPO_ROOT' && python3 scripts/service-map.py render node-exporter-unit --host $host | \
|
||||
diff -u $host/observability/quadlet/node-exporter.container -"
|
||||
[ "$status" -eq 0 ] || { echo "$host unit drift"; return 1; }
|
||||
done
|
||||
# Hand-edited copies fail the drift gate:
|
||||
grep -q "Keep this file in step" dankserver/observability/quadlet/node-exporter.container && return 1
|
||||
true
|
||||
}
|
||||
Executable
+2
@@ -0,0 +1,2 @@
|
||||
#!/usr/bin/env bash
|
||||
exit 0
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stub podman for the deploy-handler seam: `image exists` succeeds only for
|
||||
# $STUB_IMAGE (empty = nothing in storage); every other verb is a no-op.
|
||||
# The invocations are logged so a test can read what the handler executed.
|
||||
printf '%s\n' "$*" >> "${STUB_PODMAN_LOG:-/dev/null}"
|
||||
case "$1 $2" in
|
||||
"image exists")
|
||||
[[ -n "${STUB_IMAGE:-}" && "$3" == "$STUB_IMAGE" ]] || exit 1
|
||||
;;
|
||||
esac
|
||||
exit 0
|
||||
Executable
+2
@@ -0,0 +1,2 @@
|
||||
#!/usr/bin/env bash
|
||||
exit 0
|
||||
Executable
+2
@@ -0,0 +1,2 @@
|
||||
#!/usr/bin/env bash
|
||||
exit 0
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stub sudo for the deploy-handler seam: run_privileged pipes the password
|
||||
# on stdin and calls `sudo -S -- <cmd>`. The stub drops the password and the
|
||||
# flags and execs the command, so no privilege gate hides a handler bug.
|
||||
cat >/dev/null 2>&1 || true
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
-S|--prompt) shift ;;
|
||||
--) shift; break ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
exec "$@"
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stub systemctl for the deploy-handler seam: every invocation is logged, so
|
||||
# a test asserts on what the handler ASKED systemd to do. `is-active` answers
|
||||
# "inactive" (the state a cutover finds the unit in) and MainPID answers
|
||||
# $STUB_MAIN_PID — 0 is the stray-port-holder shape, the false-green gate
|
||||
# every LLM handler reads after a health answer.
|
||||
printf '%s\n' "$*" >> "${STUB_SYSTEMCTL_LOG:-/dev/null}"
|
||||
case "$*" in
|
||||
"is-active "*) echo inactive ;;
|
||||
*"MainPID --value "*) echo "${STUB_MAIN_PID:-4242}" ;;
|
||||
esac
|
||||
exit 0
|
||||
Reference in New Issue
Block a user