Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Frank
|
8617b79a18 | ||
|
Frank
|
33eae71d33 | ||
|
Frank
|
8618ec6dbe | ||
|
Frank
|
e2308dff71 | ||
|
Frank
|
6ba19f6f48 | ||
|
Frank
|
0ff016a6b2 | ||
|
Frank
|
bd5c28c4e9 |
No files matched your search
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
|
||||
| Dankserver | `frank-primary` | **FreeToken PR #587** (branch `rocm714_torch214`, head `77a7201`; local image `frank-primary-freetoken:pr587-77a7201` built on the host from `scripts/docker/frank-primary-freetoken.Dockerfile` — digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards, modelopt block-FP8, loaded straight from safetensors), **ONE R9700 (second card idle)**, all experts in **pinned host RAM** (~120 GiB) behind a GPU LRU (`--moe-cache-rate 0.10`), PLE table 47.7 GiB pinned (root unit + `LimitMEMLOCK=infinity`), **KV pool `--num-tokens 299968`** (24 KiB/token, multiple-of-64 law of the `qsa_sparse` auto backend), 2 slots, `--max-output-tokens 36864`, **parsers AUTO** (thinking → `reasoning_content`, tools qwen3.8 dialect), vision tower host-streamed. Measured: boot ~2.3 min cold / 92 s warm, decode **17.1 tok/s** single-stream; **TP=2 does not boot on ROCm** (pynccl JIT); `/metrics` 404 (Prometheus scrape reads down). Deploy gate reads the `/health` BODY (`status:"ok"`) — FreeToken 200s while loading. Rollback tag `frank-primary-vllm-shapeG-20261003` |7770|
|
||||
| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
|
||||
|
||||
@@ -40,7 +40,11 @@ the [runbook index](#runbook-index). The full swap history lives in
|
||||
- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
|
||||
scripts, systemd units, env tunables, bench harnesses
|
||||
- `old/` — retired LLM scripts (reference only)
|
||||
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format)
|
||||
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
|
||||
The managed skill library is distilled there one page per skill-cluster:
|
||||
`LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
|
||||
`Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
|
||||
pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
|
||||
- `tests/` — bats suite + Playwright e2e
|
||||
|
||||
## Git Workflow
|
||||
@@ -260,7 +264,7 @@ merely wedged kernel usually still answers ARP.
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
|
||||
- Dankserver `frank-primary` (root unit, FreeToken on one R9700):
|
||||
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
|
||||
2. Bench.
|
||||
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
|
||||
@@ -295,8 +299,9 @@ a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
|
||||
image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
|
||||
ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
|
||||
`frank-primary` takes them through a BF16 mmproj loaded explicitly (this build never
|
||||
autoloads projectors), and `frank-small` through
|
||||
`frank-primary` takes them natively (FreeToken builds the qwen3.8 vision
|
||||
tower and streams its block weights from host RAM two at a time by default),
|
||||
and `frank-small` through
|
||||
`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
|
||||
GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
|
||||
**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the
|
||||
@@ -443,122 +448,63 @@ pairs with Q5 only; SGLang rollback in
|
||||
`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).
|
||||
|
||||
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
|
||||
### frank-primary — FreeToken (PR #587 ROCm), Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
|
||||
|
||||
The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
|
||||
digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
|
||||
(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
|
||||
**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
|
||||
2026-10-03 the **shape-G** geometry (shape E + MNBT 2048 + the CUDA-graph plan
|
||||
term at the measured floor): `--max-model-len 200704` × 3 slots with
|
||||
`--kv-cache-memory-bytes 3,166,500,000`/rank (~400,700 KV tokens — the engine
|
||||
prices this pool at 7,891.8 B/token here), util **0.982** (the law — see the
|
||||
unbootable-1.0 lesson below), `--expert-precise-memory`,
|
||||
`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`,
|
||||
`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries
|
||||
`contextWindow: 200704` for this endpoint — the served window, not the model's
|
||||
native 262144, so the harness compacts before vLLM answers a longer prompt
|
||||
with a 400.
|
||||
The **FreeToken engine at PR #587 head `77a7201`** (branch
|
||||
`rocm714_torch214`, built on the host into the local image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201` by
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — the PR's own
|
||||
docs/install_amd.md stack: digest-pinned
|
||||
`rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0` base =
|
||||
torch 2.11.0+rocm7.14/triton 3.7, `--no-build-isolation --no-deps -e .`,
|
||||
curated portable deps, `flashlib==0.3.0 --no-deps`, and a `libhiprtc.so`
|
||||
dev-symlink the pip-ROCm wheel lacks) serving
|
||||
**`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards,
|
||||
standard modelopt block-FP8 — 75,264 `weight_scale_inv` tensors, no
|
||||
`input_scale`, loaded directly from HF safetensors, no FTW conversion) on
|
||||
**ONE R9700 (gfx1201)** — the second card sits idle. Plain `podman run`
|
||||
through `scripts/serve-frank-primary.sh` under the root systemd unit
|
||||
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
|
||||
container name is the unit name (`frank-primary`; the vLLM-era
|
||||
`frank-primary-qwen` is in the legacy cleanup list). Cutover 2026-10-05;
|
||||
rollback tag `frank-primary-vllm-shapeG-20261003` (the tcclaviger vLLM trio
|
||||
and its 362-line env lore live in that tag; the tcclaviger image stays in
|
||||
dank's rootless storage).
|
||||
|
||||
**`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by
|
||||
single-variable boot probe against the then-deployed shape: SEQS 2 fails at
|
||||
BOTH MNBT 1024 and 2048 with `torch.fx.experimental.symbolic_shapes.Constraint
|
||||
ViolationError: Constraints violated (L['query_start_loc'].size()[0]) …
|
||||
specialized it to be a constant (3)`; SEQS 3 passes at both. `model_runner.py`
|
||||
sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
|
||||
`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
|
||||
`SEQS + 1` for any `MNBT ≥ SEQS` — MNBT only moves the compile RANGE
|
||||
((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
|
||||
graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
|
||||
**Shape**: all expert weights in **pinned host RAM** (~120 GiB) behind a
|
||||
GPU LRU (`--moe-cache-rate 0.10`, ~12 GiB — the single-card ceiling given
|
||||
the KV contract), the ~47.7 GiB PLE ngram table pinned in host RAM
|
||||
(→ root unit + memlock law), **KV pool `--num-tokens 299968`**
|
||||
(24 KiB/token f16 across the 12 full-attention layers ≈ 7.0 GiB; **must be
|
||||
a multiple of 64** — the auto attention backend is `qsa_sparse`, which
|
||||
overrides page size to 64; 300,000 was refused at first boot), 2 decode
|
||||
slots, `--max-output-tokens 36864` (the harness maxTokens; requests that
|
||||
omit one get this default). **Parsers are AUTO**: thinking lands in
|
||||
`reasoning_content` natively, tool calls in the qwen3.8 dialect — models.yml
|
||||
flips `reasoningContentField` to `reasoning_content` on the harness side.
|
||||
Vision works (models.yml keeps `input: [text, image]`); the vision tower's
|
||||
blocks stream from host RAM two at a time by default.
|
||||
|
||||
**The expert arena has THREE reachable levers: `max_model_len`, the overhead
|
||||
term, and the CUDA-graph estimate.**
|
||||
The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
|
||||
graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
|
||||
`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
|
||||
200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
|
||||
0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
|
||||
headroom (the shape it replaced died with `0 bytes free` on a real 995-token
|
||||
request). The second lever is `--expert-precise-memory`, **ON since
|
||||
2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted +
|
||||
compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB
|
||||
difference lands in the arena — measured plan lines at 200704 × 3: overhead
|
||||
5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region
|
||||
13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9
|
||||
deliberately: because the pool is not an arena lever, cutting it would cost
|
||||
concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots.
|
||||
The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
|
||||
headroom collapses the offload hit rate.
|
||||
**Measured at cutover**: boot ~2.3 min cold-ish / 92 s warm → /health
|
||||
status ok; decode **17.1 tok/s** single-stream temp 0 (the vLLM incumbent
|
||||
measured ~30 tok/s on TWO cards; the KV contract caps the single-card LRU,
|
||||
and **TP=2 exists but does not boot on ROCm** — the TP allreduce JIT
|
||||
(`pynccl.cu`) can't resolve `cuda_runtime.h` for gfx1201; probed
|
||||
2026-10-06, signature in the runbook). `/metrics` is a 404 — the Dankclaw
|
||||
Prometheus llm-fleet scrape of :7770 reads down until repointed.
|
||||
`ft ctl cache --kv/--moe` resizes pools live (grow only; shrink needs a
|
||||
restart); `ft ctl stats` reports occupancy.
|
||||
|
||||
**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).**
|
||||
`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the
|
||||
planner charges `budget = total × util` against terms that overstate reality:
|
||||
the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while
|
||||
the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25
|
||||
GiB) is far above the idle-measured figure. So the visible free space is the
|
||||
estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena
|
||||
lever (each +0.01 = +0.32 GiB) but **0.982 is the law, and 1.00 is UNBOOTABLE
|
||||
on this fork — measured 2026-10-03**: the worker init demands
|
||||
`free > budget`, and at `le=1` budget == device total, so both ranks raise
|
||||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` before a
|
||||
single weight loads, twice (2-restart crash loop, zero payload progress). The
|
||||
CLI field accepting `le=1` is the trap; Frank's 0.982 is where it always
|
||||
ships. What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
|
||||
`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the
|
||||
analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's
|
||||
own post-capture line reported 0.31 GiB, and a second boot (with the plan
|
||||
already charging 0.32 GiB) reported **-0.06 GiB**, so the 256 MiB base pool is
|
||||
pure over-charge: shape G charges `0 base + 12 MiB/graph` = 0.07 GiB, which
|
||||
still prices per graph for a future 18-graph spec-decode shape (0.21 GiB).
|
||||
Verify every deploy by comparing the `Estimated CUDA graph memory`
|
||||
line with `Graph capturing finished … took X GiB`; if capture exceeds the
|
||||
estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO
|
||||
arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)`
|
||||
anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the
|
||||
fitted figure was fitted AT 4096.
|
||||
**Deploy gate law (changed for FreeToken)**: `/health` returns **200 with
|
||||
`{"status":"loading"}`** while the engine loads — the FastAPI front end
|
||||
binds before the scheduler is up, so a bare `curl` false-greens (measured:
|
||||
"healthy after 6 s" mid-page-in). The deployer loops until the body says
|
||||
`"status":"ok"`; the MainPID stray-port guard is unchanged. The image gate
|
||||
is `podman image exists` (a local build carries no RepoDigests — the old
|
||||
digest grep refused exactly this shape; pinning lives in the Dockerfile).
|
||||
|
||||
**A container's compile cache does NOT survive `podman rm -f`, and on this
|
||||
endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
|
||||
`/root/.cache` at all** (fresh container: `No such file or directory`); after a
|
||||
boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus
|
||||
`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine
|
||||
prints `torch.compile took 181.20 s in total` (173.19 s of it one graph,
|
||||
`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and
|
||||
reboot re-pays it. The serve script now mounts
|
||||
`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT /
|
||||
torch_compile tree (`VLLM_CACHE_ROOT`, whose default is
|
||||
`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and
|
||||
nothing in the image is hidden because the image has no such directory. The
|
||||
compile RANGE is part of the cache key, so an MNBT change costs one
|
||||
recompilation, not a permanent tax. Same class of fix as the recipe's
|
||||
`/cache/{triton,vllm,inductor}` volumes. **Proven live 2026-10-02 on
|
||||
consecutive boots:** cold `Compiling a graph for compile range (1, 1024) takes
|
||||
179.24 s` / `torch.compile took 188.26 s`; warm `Directly load the compiled
|
||||
graph(s) for compile range (1, 1024) from the cache, took 2.143 s` /
|
||||
`torch.compile took 3.97 s in total` / `init engine (profile, create kv cache,
|
||||
warmup model) took 41.13 s (compilation: 3.98 s)`. Time to `/health` is
|
||||
dominated by the 172.8 GiB payload load, so the visible win is the ~184 s of
|
||||
inductor/AOT build, not a proportional boot cut.
|
||||
|
||||
Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
|
||||
**37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;
|
||||
2-way concurrency sums BELOW parity (25.49 vs 26.8 single) — slots are a
|
||||
capacity feature, not a throughput one. V-only q8 KV is NOT possible via
|
||||
config: stock `--kv-cache-dtype` is K+V together; a V-only quant needs a fork
|
||||
patch.
|
||||
|
||||
Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
|
||||
`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
|
||||
frank-primary-qwen`. The serve script drives `podman run` directly, gates
|
||||
the digest-pinned image, the payload's `config.json` and its exact
|
||||
131-shard count and the GPU device nodes, clears stale container
|
||||
identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
|
||||
same, waits out :7770, relinks/enables/restarts the unit and health-gates
|
||||
with the MainPID check. Boot cost ~7 min warm (the 172.8 GiB payload load; the
|
||||
~184 s inductor/AOT build inside it is what the mounted cache now skips — see
|
||||
the compile-cache lesson below); health returns 503 while loading.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
|
||||
|
||||
## Key Lessons Learned
|
||||
@@ -1005,7 +951,7 @@ Active:
|
||||
|---------|---------|
|
||||
| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
|
||||
| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
|
||||
| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| frank-primary (FreeToken PR #587 ROCm, Qwen3.8-Flash-Next FP8, 1×R9700, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
@@ -1014,6 +960,7 @@ Retired (rollback targets and history):
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
||||
|
||||
+48
-27
@@ -1250,24 +1250,25 @@ deploy_llm_small() {
|
||||
echo "frank-small: health answered by unit MainPID $main_pid"
|
||||
}
|
||||
|
||||
# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
|
||||
# (digest-pinned image, gfx1201) serving the orcarouter
|
||||
# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
|
||||
# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
|
||||
# script drives `podman run` directly (the image entrypoint execs
|
||||
# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
|
||||
# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
|
||||
# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
|
||||
# alias as the endpoints it replaced; the harness models.yml id is unchanged.
|
||||
# deploy_llm_primary — frank-primary is the FreeToken engine (PR #587 ROCm
|
||||
# build, gfx1201) serving the orcarouter Qwen3.8-Flash-Next-Uncensored-FP8
|
||||
# payload (131 safetensors shards) on ONE R9700 as a ROOT unit whose serve
|
||||
# script drives `podman run` directly (the image carries `ft serve`).
|
||||
# 2026-10-05 cutover from the tcclaviger vLLM fork (pre-cutover tag
|
||||
# frank-primary-vllm-shapeG-20261003); before that, 2026-09-29 cutover from
|
||||
# the R9V radiance fork, whose launcher/shim/setup-state machinery is
|
||||
# retired. Same port and alias as the endpoints it replaced; the harness
|
||||
# models.yml id is unchanged.
|
||||
#
|
||||
# Gates: the pinned runtime image must be loadable from dank's rootless
|
||||
# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
|
||||
# the payload dir must carry config.json and the expected shard count
|
||||
# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
|
||||
# sha256 verification was the fetch script's job at download time, never
|
||||
# re-hashed here.
|
||||
LLM_PRIMARY_CONTAINER=frank-primary-qwen
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
|
||||
# Gates: the runtime image must exist in dank's rootless storage
|
||||
# (FRANK_PRIMARY_IMAGE — a LOCAL tag: the image is built on the host from
|
||||
# scripts/docker/frank-primary-freetoken.Dockerfile, which pins the
|
||||
# FreeToken PR-head SHA and the base image), the payload dir must carry
|
||||
# config.json and the expected shard count (FRANK_PRIMARY_MODEL_SHARDS),
|
||||
# and the unit env must be complete. Per-file sha256 verification was the
|
||||
# fetch script's job at download time, never re-hashed here.
|
||||
LLM_PRIMARY_CONTAINER=frank-primary
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
||||
|
||||
deploy_llm_primary() {
|
||||
# Refuse rather than restart-loop: a unit whose image or payload is
|
||||
@@ -1286,15 +1287,16 @@ deploy_llm_primary() {
|
||||
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
|
||||
return 1
|
||||
fi
|
||||
# Capture podman's output BEFORE grepping: under set -o pipefail, the
|
||||
# classic `podman images | grep -q` races (grep exits on match, podman
|
||||
# dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
|
||||
# 2026-09-30 deploy refusal).
|
||||
local digests
|
||||
digests="$(podman images --format '{{.RepoDigests}}')"
|
||||
if ! grep -qF "${image#*@}" <<<"$digests"; then
|
||||
echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
|
||||
echo "frank-primary: fetch it (podman pull $image)" >&2
|
||||
# The image gate is EXISTENCE IN STORAGE, not a registry digest: the
|
||||
# FreeToken image is built ON THE HOST (a local tag carries NO
|
||||
# RepoDigests entry — the old grep refused exactly this shape), and
|
||||
# pinning lives in the Dockerfile (pinned PR-head SHA + digest-pinned
|
||||
# base image). `podman image exists` accepts a local tag and a
|
||||
# name@digest ref alike, in one command — no pipefail/grep race (the
|
||||
# 2026-09-30 lesson) for either shape.
|
||||
if ! podman image exists "$image"; then
|
||||
echo "ERROR: runtime image absent from rootless storage: $image" >&2
|
||||
echo "frank-primary: build it (scripts/docker/frank-primary-freetoken.Dockerfile) or pull it" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$model_dir/config.json" ]]; then
|
||||
@@ -1337,7 +1339,26 @@ deploy_llm_primary() {
|
||||
run_privileged systemctl restart frank-primary.service
|
||||
# Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
|
||||
# measured 2026-09-29). Parity with every predecessor on this port.
|
||||
wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
|
||||
# FreeToken's /health returns 200 with {"status":"loading"} while the
|
||||
# engine loads (the FastAPI front end binds before the scheduler is
|
||||
# up), so a bare 200 false-greens — measured 2026-10-06: "healthy after
|
||||
# 6 s" while 131 shards were still paging in. vLLM 503'd during load,
|
||||
# which made the bare curl honest there; FreeToken needs the BODY read.
|
||||
# Only status:"ok" counts. (The MainPID check below still guards the
|
||||
# stray-port case.)
|
||||
local i ok=0
|
||||
for ((i=1; i<=900; i++)); do
|
||||
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null | grep -q '"status":"ok"'; then
|
||||
echo "frank-primary engine healthy after approximately $((i * 2))s"
|
||||
ok=1
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
if [[ "$ok" != 1 ]]; then
|
||||
echo "FAIL: frank-primary never reached /health status ok in 1800s" >&2
|
||||
return 1
|
||||
fi
|
||||
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
|
||||
# port cannot tell WHO answers — a stray probe holding :7770 satisfies
|
||||
# it while the unit is stopped. The deploy is only green if the unit
|
||||
|
||||
@@ -31,8 +31,9 @@
|
||||
# Triton kernels on ROCm.
|
||||
#
|
||||
# Base is digest-pinned (2026-10-05): sha256:a223aee1… is the manifest this
|
||||
# build was qualified against; a re-pushed tag cannot change the stack.
|
||||
FROM rocm/pytorch@sha256:a223aee17aef5d21c3b9f63436dd19d27d1c665ec8b2f40011c9546cabae2a80
|
||||
# build was qualified against; a re-pushed tag cannot change the stack. The
|
||||
# readable tag rides along for humans (podman verifies the digest matches).
|
||||
FROM rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee17aef5d21c3b9f63436dd19d27d1c665ec8b2f40011c9546cabae2a80
|
||||
|
||||
# Pin the PR head, not the branch: a future push to rocm714_torch214 would
|
||||
# otherwise silently change the deployed engine.
|
||||
|
||||
+73
-353
@@ -1,362 +1,82 @@
|
||||
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
||||
# frank-primary.env — FreeToken (PR #587 ROCm) serving the orcarouter
|
||||
# Qwen3.8-Flash-Next-Uncensored-FP8 payload on dankserver :7770.
|
||||
#
|
||||
# 2026-10-03 shape G (as shipped, after the 1.00 revert): the CUDA-graph plan
|
||||
# term dropped from the analytic 0.55 GiB to the measured 0.07 GiB (0 base +
|
||||
# 12 MiB/graph), and UTIL at the 0.982 law. MEASURED on the live deploy: arena
|
||||
# 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x concurrency (the first
|
||||
# shape holding two full-length conversations with real spare), driver used
|
||||
# 30.97 GiB of 31.86 — 0.89 GiB of real free VRAM per card, held through a
|
||||
# live 4,096-token generation. UTIL stays 0.982 ALWAYS: 1.00 is unbootable on
|
||||
# this fork (worker init refuses budget >= device total) and 0.13-GiB-margin
|
||||
# boots die under load.
|
||||
# `budget - non-expert - KV reserve - graphs - overhead` and the KV reserve is a
|
||||
# registry constant, so those two terms are the whole flag surface.
|
||||
# 2026-10-03 shape F (absorbed): MNBT 2048 + the graph term pinned. Measured on
|
||||
# the live deploy: arena 17.52 GiB / 7,653 slots, budget 31.29, graphs 0.32.
|
||||
# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
|
||||
# --expert-precise-memory ON. The window shrink from the native 262144 and the
|
||||
# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
|
||||
# the measured 2x2 is below); shape E adds the precise-overhead plan, which
|
||||
# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
|
||||
# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
|
||||
# The pool is deliberately NOT cut (option A): the pool is not an arena lever
|
||||
# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
|
||||
# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
|
||||
# tag frank-primary-fp8-mtp-20261001.
|
||||
# 2026-10-05 cutover from the tcclaviger RDNA4 vLLM fork (git tag
|
||||
# frank-primary-vllm-shapeG-20261003 is the pre-cutover state; the 362-line
|
||||
# vLLM-era env is in that tag). The engine is FreeToken at PR head
|
||||
# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214), built
|
||||
# on the host into a local image by
|
||||
# scripts/docker/frank-primary-freetoken.Dockerfile — the PR's own
|
||||
# docs/install_amd.md stack (ROCm 7.14 base image, torch 2.11.0+rocm7.14,
|
||||
# triton 3.7) for gfx1201.
|
||||
#
|
||||
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
||||
# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
|
||||
# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
|
||||
# run line as `-e GPU_MAX_HW_QUEUES=…`.
|
||||
# WHY FREETOKEN: the vLLM fork's expert offload planned VRAM/host tiers per
|
||||
# boot from an estimator (util 0.982 law, arena 17.77 GiB / 7,762 slots,
|
||||
# ~40k-token GDN state faults) and 1.0 was unbootable. FreeToken keeps ALL
|
||||
# expert weights in pinned HOST RAM (~120 GiB) with an LRU GPU cache, sizes
|
||||
# KV from an explicit token pool, and serves on ONE R9700 (32 GB) with the
|
||||
# second card idle. RAM fit: 246 GiB total, service stopped frees the vLLM
|
||||
# container's ~170 GiB; weights ~120 GiB + PLE 47.7 GiB + overhead ≈ 190 GiB.
|
||||
#
|
||||
# PAYLOAD: unchanged — /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8,
|
||||
# 131 safetensors shards. Quant is standard modelopt block-FP8: 75,264
|
||||
# weight_scale_inv tensors (block 128x128), zero input_scale (FreeToken #462
|
||||
# covers exactly this export shape), one weight_scale on the PLE ngram
|
||||
# embedding. FreeToken loads HF safetensors directly; no FTW conversion.
|
||||
#
|
||||
# --- runtime ----------------------------------------------------------------
|
||||
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
||||
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
|
||||
# bytes in rootless storage.
|
||||
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
||||
|
||||
# --- payload ----------------------------------------------------------------
|
||||
# SERVING FACTS (docs/cli.md, verified against main+PR587):
|
||||
# - `ft serve` loads the checkpoint directly; `--served-model-name` keeps the
|
||||
# harness id; reasoning/tool parsers are AUTO (reasoning lands in
|
||||
# reasoning_content — models.yml flips reasoningContentField to match).
|
||||
# - KV: --page-size must stay 1 for GDN/SWA models (default); --cache-type
|
||||
# radix auto-picks the GDN-aware variant.
|
||||
# - MoE: --moe-strategy auto -> offload (fp8 experts CANNOT use
|
||||
# --moe-cpu-layers; only bf16/nvfp4/mxfp4 have a CPU executor).
|
||||
# --moe-cache-rate holds the GPU LRU fraction of all experts.
|
||||
# - `ft ctl cache --kv --moe` grows the pools LIVE on a running server
|
||||
# (shrinking needs a restart); `ft ctl stats` reports pool occupancy.
|
||||
# - KV MATH: 12 full-attention layers x 2 KV heads x 256 head_dim x 2 (K+V)
|
||||
# x 2 B = 24 KiB/token (f16 KV; the 36 GDN layers use fixed per-seq state,
|
||||
# not per-token KV). See FRANK_PRIMARY_FT_NUM_TOKENS below for the pool
|
||||
# size and its measured page-size law.
|
||||
FRANK_PRIMARY_IMAGE=localhost/frank-primary-freetoken:pr587-77a7201
|
||||
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
||||
FRANK_PRIMARY_MODEL_SHARDS=131
|
||||
|
||||
# --- identity ---------------------------------------------------------------
|
||||
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
|
||||
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
|
||||
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
||||
# name across systemctl and podman ps. The vLLM-era name frank-primary-qwen
|
||||
# moved into the legacy list so every deploy cleans it up.
|
||||
FRANK_PRIMARY_CONTAINER=frank-primary
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
||||
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
||||
FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# Containers removed before every start: the retired R9V identity and the ad
|
||||
# hoc experiment identity (both hold the GPUs / name registrations).
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape (2026-10-02, shape E) ---------------------------
|
||||
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
|
||||
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
|
||||
# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
|
||||
# GPU KV cache size: 398,658 tokens,
|
||||
# Maximum concurrency for 200,704 tokens per request: 1.99x
|
||||
# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
|
||||
# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
|
||||
# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
|
||||
# concurrency figure across geometries. Shorter requests use the spare blocks,
|
||||
# and a third LONG request queues rather than being refused. Verified: three
|
||||
# concurrent 67-token requests answered in parallel, ~4.1 s each.
|
||||
#
|
||||
# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
|
||||
# against the then-deployed known-good shape, production stopped for each):
|
||||
# SEQS 2 + MNBT 1024 -> Engine core initialization failed
|
||||
# SEQS 2 + MNBT 2048 -> Engine core initialization failed
|
||||
# SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
|
||||
# SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
|
||||
# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
|
||||
# "marked dynamic but your code specialized it to be a constant (3)", raised
|
||||
# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
|
||||
# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
|
||||
# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
|
||||
# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
|
||||
# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
|
||||
# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
|
||||
# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
|
||||
#
|
||||
# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
|
||||
# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
|
||||
# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
|
||||
# MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
|
||||
# MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
|
||||
# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
||||
# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
|
||||
# died with 0 bytes free on a real 995-token request). To grow the arena,
|
||||
# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
|
||||
# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
|
||||
# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
|
||||
#
|
||||
# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
|
||||
# measured +18.7% decode vs default geometry and the win grows with depth.
|
||||
# Do not drop it.
|
||||
# CHUNKED PREFILL: the fork exposes the standard scheduler flags. --max-num-batched-tokens
|
||||
# (MNBT) is the lever; --enable-chunked-prefill is an auto-flag (True in SchedulerConfig)
|
||||
# and --prefill-schedule-interval is the gap between iterations. The running shape carries
|
||||
# --max-num-batched-tokens 2048 with --max-model-len 200704, so chunked prefill is ON and
|
||||
# already active: a 200704-token prompt is prefilled in ~98 iterations of 2048 tokens.
|
||||
# MEASURED 2026-09-29 on the ad hoc qwen-fp8 container (same fork, same payload, TP=2):
|
||||
# MNBT 8192 vs the default 2048 is NEUTRAL on decode at every depth, and small values
|
||||
# re-chunk prefill only — they hurt TTFT and never touch decode, because a decode step is
|
||||
# max_num_seqs x (spec+1) tokens, independent of MNBT. There is NO --ubatch-size
|
||||
# equivalent in vLLM: that is a llama.cpp compute-buffer concept (the frank-small pair
|
||||
# doubled on 2026-10-05 has no counterpart here). MNBT is therefore NOT a lever on this
|
||||
# shape — the deployed 2048 is the fork's default — and every change re-enters the
|
||||
# compile RANGE path, so FRANK_PRIMARY_CACHE_DIR pays one recompilation.
|
||||
# MNBT 2048 (Frank's call, 2026-10-03; the lock was 1024). The measured law
|
||||
# still holds: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s), 1024-4096
|
||||
# are interchangeable on decode, 12288 costs 2.5% plus arena. 2048 is ARENA-FREE
|
||||
# by construction: the registry's overhead model is
|
||||
# base + 0.22 GiB x max(0, (nbt - 4096)/1024) with the anchor at nbt 4096, so
|
||||
# every rung up to and including 4096 prices the SAME 5.25 GiB overhead — and
|
||||
# the fitted 5.25 was fitted at nbt 4096, i.e. our 2048 sits UNDER the anchor,
|
||||
# so the activation term is covered by a figure measured at a bigger batch.
|
||||
# SEQS 3 boots at both 1024 and 2048 (measured 2026-10-02; the SEQS=2
|
||||
# ConstraintViolationError is a SEQS property, not an MNBT one). Costs exactly
|
||||
# one recompilation: the compile RANGE is part of the cache key, so
|
||||
# FRANK_PRIMARY_CACHE_DIR pays for whichever rung is chosen once.
|
||||
# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
|
||||
# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
|
||||
# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
|
||||
# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
|
||||
# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
|
||||
# --expert-precise-memory is ON (shape E). It selects the fitted overhead
|
||||
# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
|
||||
# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
|
||||
# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
|
||||
# booted AS the service, plan read off the engine's own log):
|
||||
# runtime overhead 7.60 -> 5.25 GiB (registry Qwen4Exp…, precise
|
||||
# gfx1201 TP2 nbt4096/seqs16/len262144)
|
||||
# expert cache 14.87 -> 17.22 GiB
|
||||
# slots 6,496 -> 7,522 (+1,026 = +16%, 2.344 MiB each)
|
||||
# swap region 13.7 GiB / 5,986 slots, 133-134 per layer
|
||||
# boot healthy in 420 s, no OOM, no knife-edge
|
||||
# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
|
||||
# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
|
||||
# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
|
||||
# the cause. At 200704 x 3 it fits with room; the run line also reports
|
||||
# 'expert_precise_memory': True in non-default args, so a silent argparse drop
|
||||
# would be visible in the boot log.
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1
|
||||
# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
|
||||
# guide recommends 2 for this class of workload; the runtime default on Linux
|
||||
# is 4). Verified a REAL knob, not a no-op: the string is present in
|
||||
# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
|
||||
# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
|
||||
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
|
||||
# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
|
||||
# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
|
||||
# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
|
||||
# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
|
||||
# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
|
||||
# and reboot) throws them away and re-pays the build. The engine's own log
|
||||
# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
|
||||
# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
|
||||
# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
|
||||
# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
|
||||
# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
|
||||
# PROVEN LIVE 2026-10-02, two boots of this exact shape:
|
||||
# cold `Compiling a graph for compile range (1, 1024) takes 179.24 s`
|
||||
# `torch.compile took 188.26 s in total`
|
||||
# warm `Directly load the compiled graph(s) for compile range (1, 1024)
|
||||
# from the cache, took 2.143 s`
|
||||
# `torch.compile took 3.97 s in total`
|
||||
# `init engine (profile, create kv cache, warmup model) took 41.13 s
|
||||
# (compilation: 3.98 s)`
|
||||
# (Total time to /health is dominated by the 172.8 GiB payload load, not the
|
||||
# compile — the saving is the ~184 s of inductor/AOT build.)
|
||||
# Safe to grow/prune: entries are keyed by version + config hash, so a stale
|
||||
# entry is a miss, never a wrong answer.
|
||||
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
|
||||
FRANK_PRIMARY_MAXLEN=200704
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
|
||||
# is not an arena lever, raising it back from a cut buys concurrency only, and
|
||||
# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
|
||||
# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
|
||||
# as the shipped value; it remains the conservative arm if free VRAM ever binds.
|
||||
# KV forced to 400,700 tokens (Frank, 2026-10-03). The engine prices this pool
|
||||
# at 7,891.8 B/token at this shape and loses ~3.87 MB to block alignment, so
|
||||
# 3,166,500,000 B is the ask that lands at/above 400,700 (verify the reported
|
||||
# "GPU KV cache size" line on every deploy; it is not a constant ratio across
|
||||
# geometries — 7,701 B/token at 262144).
|
||||
FRANK_PRIMARY_KV_BYTES=3166500000
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_MNBT=2048
|
||||
FRANK_PRIMARY_GATHER_CHUNKS=32
|
||||
FRANK_PRIMARY_GATHER_LANES=64
|
||||
# PREFIX CACHING: ON. An OFF experiment ran 2026-10-05 (below) and FAILED
|
||||
# to stop the corruption, so the historical flag is back.
|
||||
# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
|
||||
# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
|
||||
# burst, then the process-lifetime silence cap:
|
||||
# [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
|
||||
# with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
|
||||
# [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
|
||||
# [gdn_hip] ERROR: state fault report limit (32) reached; further faults
|
||||
# are counted silently
|
||||
# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
|
||||
# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
|
||||
# (QFN) flaw under prefix caching and is being actively worked; output
|
||||
# quality for this request may degrade. Mitigation:
|
||||
# --no-enable-prefix-caching".
|
||||
# MECHANISM: with prefix caching on, this fork puts QFN on
|
||||
# mamba_cache_mode 'align' (the default-for-this-arch branch in
|
||||
# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
|
||||
# mode, so align is the only sound mode WITH the flag). Align mode
|
||||
# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
|
||||
# mamba page every block, where block = 3136 tokens (the attention block
|
||||
# size is raised to 3136 so the page sizes match exactly — the boot log's
|
||||
# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
|
||||
# page whenever a request resumes from a cached prefix. The corrupt state
|
||||
# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
|
||||
# where the request's output turned repetitive/corrupted. The NaN stayed
|
||||
# in the mamba page, so every later turn of that conversation re-seeded the
|
||||
# same non-finite state for those two heads (silent: the fault-report cap
|
||||
# was already spent), while fresh short sessions looked fine.
|
||||
# RAM WAS "RULED OUT" THE SAME DAY — AND THAT WAS WRONG (RESOLVED below).
|
||||
# What the evidence actually supported: zero MCE/EDAC records is the
|
||||
# EXPECTED silence of consumer non-ECC memory (a silent flip raises no
|
||||
# machine check); an 8 GiB pattern check over 60 s covers ~0.5 % of
|
||||
# 246 GiB — no verdict on a speed setting; "4800 is the JEDEC floor on
|
||||
# 5600-rated DIMMs" is a rating, not a stability proof at four-DIMM
|
||||
# loading. Only the two-head NaN signature argument was load-bearing,
|
||||
# and the revert test outvoted it.
|
||||
# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
|
||||
# history instead of at most 3135 tail tokens (a full 40k context costs
|
||||
# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
|
||||
# corrupted conversation is worse than a slower first token.
|
||||
# RESULT (2026-10-05, later that day): THE MITIGATION DID NOT WORK. With
|
||||
# --no-enable-prefix-caching deployed and mechanically verified (zero
|
||||
# gdn_hip faults, a 40,771-token single-request boundary probe coherent),
|
||||
# live long conversations STILL degraded at ~40k on the running endpoint
|
||||
# (Frank, direct). The container serving that window logged ZERO faults
|
||||
# while the corruption reproduced, so the align-mode page re-seed is not
|
||||
# the (only) mechanism — see the runbook's RESULT note for the open
|
||||
# hypotheses (re-fed poisoned transcript vs a fault-free long-context
|
||||
# path such as fp8 KV at depth). Cost bought nothing: caching is back ON.
|
||||
# The knob stays commented for a future fork build with a fixed GDN state
|
||||
# path: set 1, redeploy, re-verify past 40,768 tokens (one conversation
|
||||
# coherent AND zero gdn_hip lines).
|
||||
# RESOLVED 2026-10-05 EVENING (Frank): IT WAS THE RAM. 4800 MT/s (in
|
||||
# effect since the 2026-10-04 17:17 boot) was marginal; 4600 MT/s (since
|
||||
# the 2026-10-05 12:21 boot) and the endpoint is clean — zero gdn_hip
|
||||
# ERROR lines, zero MCE, and the live agent session on it past 140k
|
||||
# context with no degradation. Both failure eras sit under the RAM: the
|
||||
# 2026-10-04 NaN burst (corrupted weight/activation traffic poisoning a
|
||||
# GDN state write — the guard-visible path) and the fault-free ~40k
|
||||
# degradation (silent flips in KV/expert data, which no guard covers on
|
||||
# non-ECC). CONSEQUENCE FOR THE KNOB: the OFF experiment was CONFOUNDED
|
||||
# — both arms ran on the marginal speed — so it neither proved nor
|
||||
# disproved the fork's align-mode flaw. Prefix caching stays ON (zero
|
||||
# gdn_hip lines under heavy re-send traffic since the speed drop); flip
|
||||
# the knob only if a fault recurs on stable RAM or the fork ships a fix.
|
||||
# FRANK_PRIMARY_NO_PREFIX_CACHING=1
|
||||
# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
|
||||
# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
|
||||
# 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
|
||||
# shape G v1) died in worker init BEFORE loading a byte of weights — the fork
|
||||
# refuses budget >= device total with
|
||||
# ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)
|
||||
# on BOTH ranks at 09:53 and 09:54, NRestarts=2 — so 1.0 is not even a runtime
|
||||
# knife-edge here, it is unbootable. The two historical ceiling deaths (the
|
||||
# 2x262144 bf16 boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a
|
||||
# real 995-token request) remain the reason the margin exists. Rollback note
|
||||
# for the other direction: if a request ever dies with
|
||||
# `Tried to allocate ... 0 bytes free`, the first knob back is THIS number
|
||||
# down a notch, not the window.
|
||||
FRANK_PRIMARY_UTIL=0.982
|
||||
# CUDA-graph plan term, priced at the measured capture. expert_plan.py charges
|
||||
# estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs (defaults
|
||||
# 256/52 = 0.55 GiB for our 6-graph plan), and the estimator's own docstring
|
||||
# says to tune these against the post-capture line. MEASURED twice on this
|
||||
# shape: shape E reported "Graph capturing finished ... took 0.31 GiB" (TP0) and
|
||||
# 0.17 GiB (TP1); shape F, with the 0.32 GiB we then charged, reported
|
||||
# "took -0.06 GiB" — a NEGATIVE delta, i.e. capture fits inside memory the
|
||||
# profiler had already counted. So the base pool (256 MiB) is the over-charge:
|
||||
# charge 0 base + 12 MiB/graph = 0.07 GiB for the shipped no-spec 6-graph plan,
|
||||
# which returns 0.25 GiB more to the arena while still paying per graph, so a
|
||||
# future spec-decode shape (18 graphs) is charged 0.21 GiB rather than zero.
|
||||
# Verify every deploy against the "Estimated CUDA graph memory" vs "Graph
|
||||
# capturing finished ... took X GiB" lines; if capture ever goes positive and
|
||||
# exceeds the charge, raise FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52.
|
||||
# A THIRD env var exists and is the only way to state the graph term outright:
|
||||
# VLLM_CUDAGRAPH_MEM_ESTIMATE_MB pins the WHOLE term to an exact MiB total in
|
||||
# one knob — estimate_cudagraph_memory() checks it FIRST and returns base/
|
||||
# per-graph untouched when set (verified in the shipped image,
|
||||
# v1/worker/gpu/cudagraph_mem.py). We ship the BASE/PER_GRAPH pair instead so
|
||||
# a future spec-decode shape re-prices automatically; switch to ESTIMATE_MB
|
||||
# only if the drift check below ever shows the pair cannot track capture.
|
||||
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0
|
||||
FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12
|
||||
|
||||
# HOST-region cap — the last arena-plan term that still autofit per boot. With
|
||||
# no cap the planner's host bound is MemAvailable AT BOOT minus PLE 47.68 -
|
||||
# 2 x 7.0 process - 5.0 api/engine - 5% margin (measured boot 2026-10-05:
|
||||
# MemAvailable 237.97 -> 159.39 GiB usable across 2 ranks; the demand is 105.48
|
||||
# = 52.74/rank x 2 ranks, 23,040 swap-target experts). The same host reads 75.4
|
||||
# GiB available while running — the bound tracks whatever the co-tenants hold at
|
||||
# boot, and a boot where that drops near the demand silently pins next-hottest
|
||||
# experts VRAM-only instead (smaller swap region; the pinned_extra warning is
|
||||
# the only trace). Pin 120 GiB: measured demand + 14% headroom, on a host with
|
||||
# 246.1 GiB total. If a boot ever logs "expert-offload-mem … exceeds
|
||||
# MemAvailable at plan time", THAT is the contract being broken — look at the
|
||||
# co-tenant RAM before touching this number. Plan semantics verified in the
|
||||
# shipped image (offloader/expert_plan.py): the cap binds as the bound, an
|
||||
# exceedance only warns.
|
||||
FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120
|
||||
|
||||
# --- optional engine knobs (unset == the flag is absent) ---------------------
|
||||
# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
|
||||
# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
|
||||
# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the
|
||||
# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came
|
||||
# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the
|
||||
# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the
|
||||
# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our
|
||||
# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass
|
||||
# draft_sample_method=probabilistic (greedy drafts are rejected more often under
|
||||
# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace:
|
||||
# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and
|
||||
# --expert-precise-memory spends that same budget, so an MTP arm must be tested
|
||||
# with precise memory BOTH on and off.
|
||||
# MTP3: '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
|
||||
FRANK_PRIMARY_SPEC=
|
||||
# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The
|
||||
# capture-ladder law: the piecewise cudagraph ladder must contain
|
||||
# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15;
|
||||
# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default
|
||||
# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can
|
||||
# miss its own step size. Pinning the ladder is ALSO the candidate fix for the
|
||||
# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must
|
||||
# land on a captured graph size (3 is not in the default ladder, 4 is —
|
||||
# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2).
|
||||
# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law.
|
||||
# MTP3 at SEQS 3: '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}'
|
||||
FRANK_PRIMARY_COMPCONFIG=
|
||||
# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON.
|
||||
# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 /
|
||||
# top_k 20 and the engine WARNS about it on every boot, so a client that sends
|
||||
# no sampling fields gets creativity it never asked for. This is the one knob
|
||||
# that changes ANSWERS, not speed.
|
||||
# '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}'
|
||||
FRANK_PRIMARY_GEN_CONFIG=
|
||||
|
||||
# --- client contract: reasoning split, tools, preserve_thinking ----------------
|
||||
# Without --reasoning-parser the engine leaves the thinking text INLINE in
|
||||
# `content` with a dangling `</think>` and a null `reasoning` field (measured
|
||||
# 2026-10-01: content = 'We need answer...Brief.\n\n391') — the
|
||||
# malformed thinking block clients choke on. The qwen3 parser splits it
|
||||
# properly. The GLM rollback (d91379d) had dropped all three of these; the
|
||||
# working values are the ones ecaad36 shipped for this model.
|
||||
FRANK_PRIMARY_REASONING_PARSER=qwen3
|
||||
# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
|
||||
# tool_calls work here (without it, tool_choice auto returns 400).
|
||||
FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
|
||||
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
|
||||
# turns; enable_thinking makes thinking the server-side default. Verified the
|
||||
# payload's tokenizer_config template consumes both keys.
|
||||
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
|
||||
# --- FreeToken engine knobs (first-boot conservative, resize live) -----
|
||||
# Single GPU: the R9700 at index 0. HIP_VISIBLE_DEVICES=0 hides card 1 so
|
||||
# the in-container index 0 IS the right card; --gpu 0 selects it.
|
||||
FRANK_PRIMARY_FT_GPU=0
|
||||
# KV pool in TOKENS (not bytes — vLLM-era FRANK_PRIMARY_KV_BYTES is gone).
|
||||
# 299,968 tokens x 24 KiB = 7.0 GiB; clears the harness worst case
|
||||
# (177,356 tokens) and one NATIVE 262,144-token conversation + full budget.
|
||||
# MEASURED FIRST BOOT (2026-10-05): the auto attention backend is
|
||||
# qsa_sparse, which OVERRIDES page size to 64, so --num-tokens must be a
|
||||
# MULTIPLE OF 64 — 300,000 was refused ("nearest valid values: 299968 or
|
||||
# 300032"). Grows live with `ft ctl cache --kv <n>`; shrink needs a
|
||||
# restart.
|
||||
FRANK_PRIMARY_FT_NUM_TOKENS=299968
|
||||
# Concurrent decode slots. 2 is the first-boot conservative value; the
|
||||
# radix pool admits ~2 full-length 200,704-token conversations. Raise with
|
||||
# --max-running-requests only after a real-traffic look at slot occupancy.
|
||||
FRANK_PRIMARY_FT_MAX_RUNNING=2
|
||||
# Per-request output budget ceiling: the harness's maxTokens (36,864).
|
||||
# Requests that omit max_tokens get this default instead of the engine's
|
||||
# 32,768.
|
||||
FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS=36864
|
||||
# GPU expert LRU cache as a FRACTION of all experts (~120 GiB fp8 total ->
|
||||
# ~12 GiB VRAM at 0.10). First-boot conservative; resize live with
|
||||
# `ft ctl cache --moe` after reading the boot ledger + ft ctl stats.
|
||||
FRANK_PRIMARY_FT_MOE_RATE=0.10
|
||||
# Escape hatch for boot-ladder experiments (YaRN, --max-prefill-length,
|
||||
# --moe-cpu-threads, --sampling-defaults none, ...). Unset -> no extra
|
||||
# argv. Keep multi-flag values as one string; the serve script expands it
|
||||
# unquoted on purpose.
|
||||
FRANK_PRIMARY_FT_EXTRA_ARGS=
|
||||
@@ -1,29 +1,26 @@
|
||||
# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
|
||||
# Qwen3.8-Flash-Next-Uncensored-FP8, fp8 KV, NO spec decode, 3 slots x 200704
|
||||
# context with the pool sized to ~2 full-length conversations, TP=2, tiered
|
||||
# experts, 131-shard payload) on the 2x R9700 pair, dankserver :7770.
|
||||
# 2026-09-29 cutover from the R9V radiance fork (git tag
|
||||
# frank-primary-r9v-q8-20260929 is the pre-cutover state; its
|
||||
# launcher/shim/setup-state machinery is retired and the state dirs stay on
|
||||
# disk as rollback).
|
||||
# frank-primary.service — FreeToken (PR #587 ROCm) endpoint (orcarouter
|
||||
# Qwen3.8-Flash-Next-Uncensored-FP8, 131-shard payload) on ONE R9700
|
||||
# (gfx1201), dankserver :7770. 2026-10-05 cutover from the tcclaviger vLLM
|
||||
# fork (git tag frank-primary-vllm-shapeG-20261003 is the pre-cutover
|
||||
# state; the retired unit/env/serve trio is in that tag).
|
||||
#
|
||||
# Same port, same served alias; the harness models.yml id is unchanged.
|
||||
# The serve script drives `podman run` directly (the image entrypoint execs
|
||||
# `vllm serve`) and then waits on the container. NOTE: that wait can return
|
||||
# 0 even when the engine died mid-request (vLLM's API server drains and
|
||||
# exits clean on a worker fault), which is why the restart policy is
|
||||
# `always` — see the [Service] note.
|
||||
# Same port, same served alias, same container identity as the unit name;
|
||||
# the harness models.yml id is unchanged. The serve script drives
|
||||
# `podman run` directly (the image carries `ft serve`) and then waits on
|
||||
# the container. The image is built ON THE HOST from
|
||||
# scripts/docker/frank-primary-freetoken.Dockerfile (FreeToken PR head
|
||||
# 77a7201 in the PR's own ROCm 7.14 base image).
|
||||
#
|
||||
# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
|
||||
# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
|
||||
# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
|
||||
# login session on this host carries 8 MiB and a user quadlet is capped by the
|
||||
# user manager's own hard limit — raising that means restarting
|
||||
# user@1000.service and bouncing every user container on the host. A root unit
|
||||
# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
|
||||
# stays in dank's rootless podman storage.
|
||||
# WHY A ROOT UNIT (not a rootless quadlet): FreeToken pins ~120 GiB of
|
||||
# expert weights plus the ~47.7 GiB PLE ngram table in HOST RAM. Pinned
|
||||
# pages are charged against RLIMIT_MEMLOCK; dank's login session on this
|
||||
# host carries 8 MiB and a user quadlet is capped by the user manager's own
|
||||
# hard limit — raising that means restarting user@1000.service and bouncing
|
||||
# every user container on the host. A root unit grants
|
||||
# LimitMEMLOCK=infinity to the dank process via PID1, and the image stays
|
||||
# in dank's rootless podman storage.
|
||||
[Unit]
|
||||
Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
|
||||
Description=frank-primary — FreeToken (PR587) Qwen3.8-Flash-Next-Uncensored-FP8 on 1x R9700
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
||||
@@ -33,34 +30,35 @@ StartLimitIntervalSec=0
|
||||
[Service]
|
||||
User=dank
|
||||
Group=dank
|
||||
# The tiered-expert offloader pins host RAM; an RLIMIT_MEMLOCK smaller than
|
||||
# the pinned set fails the load mid-start. PID1 grants infinity here —
|
||||
# The expert LRU + PLE pin ~170 GiB of host RAM; an RLIMIT_MEMLOCK smaller
|
||||
# than the pinned set fails the load mid-start. PID1 grants infinity here —
|
||||
# parity with the fleet's other LLM units.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
||||
# systemd stop: wait alone cannot stop the container child; stop it
|
||||
# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
|
||||
ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
|
||||
# Boot cost: FP8 payload page-in plus torch compile/graph capture —
|
||||
# ~15 min cold (measured 2026-09-29), faster warm.
|
||||
# explicitly. ft serve drops connections on SIGTERM; no graceful drain
|
||||
# contract is claimed (the OpenAI front end holds no queue to drain —
|
||||
# in-flight decodes are lost, which a stop during a bench window accepts).
|
||||
ExecStop=/usr/bin/podman stop -t 120 frank-primary
|
||||
# Boot cost: 131-shard FP8 page-in + PLE pin + expert upload. vLLM-era
|
||||
# budget (1800 s) carried forward; the FreeToken first boot measured here
|
||||
# once it ran (see the runbook for the real number).
|
||||
TimeoutStartSec=1800
|
||||
# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
|
||||
# torch.OutOfMemoryError kills the engine while vLLM's API server still
|
||||
# exits 0, so ExecStart returned clean, systemd logged "Deactivated
|
||||
# successfully", and on-failure NEVER fired — the endpoint stayed down until
|
||||
# a human noticed. `always` restarts on any container death; an explicit
|
||||
# `systemctl stop` still wins (systemd does not restart a unit the operator
|
||||
# stopped), so bench windows and deploys are unaffected.
|
||||
# Restart=always, NOT on-failure (the 2026-10-01 vLLM lesson carries over:
|
||||
# a clean API-server exit after a worker fault reads as success to
|
||||
# systemd). `always` restarts on any container death; an explicit
|
||||
# `systemctl stop` still wins, so bench windows and deploys are unaffected.
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
|
||||
# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
|
||||
# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
|
||||
# Must exceed the podman grace with margin for the engine's own shutdown.
|
||||
# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without
|
||||
# an explicit TimeoutStopSec systemd's 90 s default cuts the stop short,
|
||||
# kills it, and records 'Failed with result timeout' (measured 2026-10-01
|
||||
# 13:27:17). Must exceed the podman grace with margin.
|
||||
TimeoutStopSec=180
|
||||
# The engine pins tens of GiB of host RAM (expert host slabs); it is the
|
||||
# last process on this host that should be chosen by the kernel OOM killer.
|
||||
# The engine pins tens of GiB of host RAM (expert rows + PLE table); it is
|
||||
# the last process on this host that should be chosen by the kernel OOM
|
||||
# killer.
|
||||
OOMScoreAdjust=-500
|
||||
|
||||
[Install]
|
||||
|
||||
+65
-129
@@ -1,58 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
|
||||
# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
|
||||
# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
|
||||
# frank-primary-r9v-q8-20260929 is the pre-cutover state).
|
||||
#
|
||||
# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
|
||||
# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
|
||||
# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
|
||||
# a 104-key on-host setup state; this stack is fully expressed by flags, and
|
||||
# every flag here was qualified by boot probe on 2026-09-29 — see
|
||||
# frank-primary.env for the qualification record).
|
||||
#
|
||||
# Shape (2026-10-03 shape F, on the 2026-10-02 shape E base): MAXLEN 200704 x 3
|
||||
# slots, fp8 KV, KV_BYTES
|
||||
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
|
||||
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
|
||||
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
|
||||
# CANNOT boot this model
|
||||
# (torch.fx ConstraintViolationError on query_start_loc, measured at both
|
||||
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
|
||||
# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
|
||||
# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
|
||||
# retirement of MTP compared the wrong pair — see frank-primary.env.
|
||||
# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
|
||||
# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
|
||||
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
|
||||
# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
|
||||
# state fault — the evidence block is in frank-primary.env), expert prefetch
|
||||
# OFF (36% precision measured),
|
||||
# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
|
||||
# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
|
||||
# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
|
||||
# measured capture cost (VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB, container
|
||||
# env — the analytic default charges 0.55 GiB where capture took 0.31 GiB, so
|
||||
# the 0.24 GiB difference goes to the arena) and raises util 0.98 -> 0.982,
|
||||
# Frank's cap. The native-262144 and fp8+MTP4 shapes are retired to
|
||||
# git tag frank-primary-fp8-mtp-20261001.
|
||||
#
|
||||
# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
|
||||
# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
|
||||
# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
|
||||
# container's ephemeral layer, and `podman rm -f` deletes them — so every
|
||||
# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
|
||||
# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
|
||||
# later boots load the compiled artifact instead, and hides nothing. PROVEN
|
||||
# LIVE 2026-10-02 on consecutive boots of this shape: cold `torch.compile took
|
||||
# 188.26 s`, warm `Directly load the compiled graph(s) for compile range
|
||||
# (1, 1024) from the cache, took 2.143 s` / `torch.compile took 3.97 s in
|
||||
# total` / `init engine … took 41.13 s (compilation: 3.98 s)`.
|
||||
# serve-frank-primary.sh — frank-primary endpoint (FreeToken PR #587 ROCm
|
||||
# build, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 gfx1201,
|
||||
# dankserver :7770; 2026-10-05 cutover from the tcclaviger vLLM fork — git
|
||||
# tag frank-primary-vllm-shapeG-20261003 is the pre-cutover state).
|
||||
#
|
||||
# The engine is driven by `podman run` DIRECTLY and the run line below IS
|
||||
# the engine argv: the image carries FreeToken installed at pinned PR head
|
||||
# 77a7201 (scripts/docker/frank-primary-freetoken.Dockerfile builds it from
|
||||
# the PR's own docs/install_amd.md stack). Unlike the vLLM fork there is no
|
||||
# per-boot planner: experts live in pinned host RAM with an LRU GPU cache,
|
||||
# KV is an explicit token pool, and the boot ledger in the log names
|
||||
# Shape: 1 GPU (first-boot conservative — the second R9700 sits idle;
|
||||
# --tensor-parallel-size EXISTS in FreeToken, so a 2-card TP shape is future
|
||||
# tuning once this single-card shape has a decode baseline), KV pool 300,000
|
||||
# tokens (24 KiB/token f16 across the 12 full-attention layers = 7.2 GiB;
|
||||
# clears the harness worst case of 177,356 tokens and admits one NATIVE
|
||||
# 262,144-token conversation), MoE LRU at 10% of all experts (~12 GiB),
|
||||
# 2 decode slots. First-boot conservative; `ft ctl cache --kv/--moe` grows
|
||||
# pools live, and the env comment block carries the resize law. Flag
|
||||
# surface verified against docs/cli.md of the pinned head AND against
|
||||
# `ft serve --help` inside the built image; every value here is env-driven
|
||||
# (frank-primary.env).
|
||||
|
||||
# The root unit (scripts/frank-primary.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
||||
# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
|
||||
# also works, but the unit is the fleet-parity boot-persistent shape.
|
||||
# LimitMEMLOCK=infinity) grants the pin budget: ~120 GiB of expert weights
|
||||
# plus the ~47.7 GiB PLE table pin in host RAM. A plain shell session also
|
||||
# works, but the unit is the fleet-parity boot-persistent shape.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -67,27 +40,30 @@ set -a
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
|
||||
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
|
||||
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the FreeToken image}"
|
||||
: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
|
||||
: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
|
||||
: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
|
||||
: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
|
||||
: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
|
||||
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
|
||||
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
|
||||
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
|
||||
: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
|
||||
: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
|
||||
: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
|
||||
# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
|
||||
# classic `podman images | grep -q` pattern races — grep exits on its first
|
||||
# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
|
||||
# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
|
||||
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
|
||||
if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
|
||||
echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
|
||||
echo " fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
|
||||
: "${FRANK_PRIMARY_FT_GPU:?FRANK_PRIMARY_FT_GPU must name the GPU index}"
|
||||
: "${FRANK_PRIMARY_FT_NUM_TOKENS:?FRANK_PRIMARY_FT_NUM_TOKENS must size the KV pool}"
|
||||
: "${FRANK_PRIMARY_FT_MAX_RUNNING:?FRANK_PRIMARY_FT_MAX_RUNNING must be set}"
|
||||
: "${FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS:?FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS must be set}"
|
||||
: "${FRANK_PRIMARY_FT_MOE_RATE:?FRANK_PRIMARY_FT_MOE_RATE must be set}"
|
||||
|
||||
# The image gate is EXISTENCE IN STORAGE, not a registry digest: the tag is
|
||||
# built ON THE HOST (podman build from
|
||||
# scripts/docker/frank-primary-freetoken.Dockerfile) and local builds carry
|
||||
# no RepoDigests entry — the 362-line vLLM-era RepoDigests grep refused
|
||||
# exactly this shape. Pinning lives in the Dockerfile instead: the FreeToken
|
||||
# source is checked out at a hardcoded PR-head SHA and the base image is
|
||||
# digest-pinned there. `podman image exists` accepts both a local tag and a
|
||||
# name@digest ref, and it is one command — no pipefail/grep race (the
|
||||
# 2026-09-30 lesson) for either shape.
|
||||
if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
|
||||
echo "serve-frank-primary: image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
|
||||
echo " build it: podman build -t $FRANK_PRIMARY_IMAGE -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -114,8 +90,8 @@ fi
|
||||
|
||||
# A crashed/killed unit can leave the container name registered (systemd
|
||||
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
||||
# identities (the retired R9V container and the ad hoc experiment container
|
||||
# both hold these GPUs); all idempotent.
|
||||
# identities (the vLLM-era frank-primary-qwen, the retired R9V container and
|
||||
# the ad hoc experiment containers all hold this GPU); all idempotent.
|
||||
for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
|
||||
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
||||
done
|
||||
@@ -123,74 +99,34 @@ done
|
||||
LOG=/home/dank/logs/frank_primary.log
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
|
||||
# Persistent compile cache (see the header). Create it before the mount so even
|
||||
# the first, cold boot has a target; unset -> no mount, exactly the old
|
||||
# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
|
||||
# /root/.cache/comgr (HIP kernel JIT).
|
||||
CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
|
||||
if [ -n "$CACHE_DIR" ]; then
|
||||
mkdir -p "$CACHE_DIR"
|
||||
fi
|
||||
|
||||
# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
|
||||
# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
|
||||
# available" while device_count() says 2). --ipc=host for the engine's shared
|
||||
# memory; memlock unbounded for the pinned host slabs.
|
||||
# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
|
||||
# Prefix caching: ON — the shipped contract. FRANK_PRIMARY_NO_PREFIX_-
|
||||
# CACHING=1 flips to the fork-named mitigation (--no-enable-prefix-caching)
|
||||
# for the QFN align-mode GDN state fault; it was tried 2026-10-05 and
|
||||
# REVERTED the same day (live conversations still degraded at ~40k with
|
||||
# caching off and zero faults — see the evidence block in
|
||||
# frank-primary.env). NO comment lines inside the backslash chain below —
|
||||
# a comment there terminates the continuation and the next line runs as
|
||||
# its own command (the 2026-10-05 exit-127 crash loop).
|
||||
# The ${VAR:-word} / ${VAR:+word} double-expansion for the two directions is
|
||||
# AMBIGUOUS bash (${VAR:--flag} parses as a substring offset when set —
|
||||
# measured 2026-10-05: rendered "--no-enable-prefix-caching1"), so the flag
|
||||
# is branched explicitly: exactly one of the two reaches the engine.
|
||||
if [ -n "${FRANK_PRIMARY_NO_PREFIX_CACHING:-}" ]; then
|
||||
FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching
|
||||
else
|
||||
FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
|
||||
fi
|
||||
# without the card nodes and HIP refuses contexts). --ipc=host for the
|
||||
# engine's shared memory; memlock unbounded for the ~120 GiB pinned expert
|
||||
# rows + the 47.7 GiB PLE pin (the unit's LimitMEMLOCK=infinity is the other
|
||||
# half of that law).
|
||||
# HIP_VISIBLE_DEVICES pins visibility to ONE card so in-container index 0 is
|
||||
# the right R9700; --gpu 0 selects it inside the visible set.
|
||||
podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--publish "$FRANK_PRIMARY_PUBLISH" \
|
||||
--ipc=host --ulimit memlock=-1:-1 \
|
||||
--cap-add SYS_PTRACE \
|
||||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||||
${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
|
||||
${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
|
||||
${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
|
||||
${FRANK_PRIMARY_CUDAGRAPH_BASE_MB:+-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"} \
|
||||
${FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB:+-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"} \
|
||||
${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
|
||||
-e HIP_VISIBLE_DEVICES="$FRANK_PRIMARY_FT_GPU" \
|
||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
||||
"$FRANK_PRIMARY_IMAGE" /models \
|
||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
||||
--tensor-parallel-size 2 \
|
||||
${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
|
||||
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
||||
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
||||
${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
|
||||
${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"} \
|
||||
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
||||
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
"$FRANK_PRIMARY_PREFIX_CACHE_FLAG" \
|
||||
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
|
||||
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
|
||||
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
|
||||
${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
|
||||
${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
|
||||
${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
|
||||
--trust-remote-code \
|
||||
"$FRANK_PRIMARY_IMAGE" \
|
||||
ft serve --model /models \
|
||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
|
||||
--host 0.0.0.0 --port 7770 \
|
||||
--gpu "$FRANK_PRIMARY_FT_GPU" \
|
||||
--num-tokens "$FRANK_PRIMARY_FT_NUM_TOKENS" \
|
||||
--max-running-requests "$FRANK_PRIMARY_FT_MAX_RUNNING" \
|
||||
--max-output-tokens "$FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS" \
|
||||
--moe-cache-rate "$FRANK_PRIMARY_FT_MOE_RATE" \
|
||||
${FRANK_PRIMARY_FT_EXTRA_ARGS:+$FRANK_PRIMARY_FT_EXTRA_ARGS} \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
# From here on, this script's lifetime IS the container's: wait returns the
|
||||
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
||||
# Restart=on-failure is real. (systemd stop goes through ExecStop in the
|
||||
# unit — podman wait alone cannot stop the container child.)
|
||||
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
|
||||
# podman wait alone cannot stop the container child.)
|
||||
exec podman wait "$FRANK_PRIMARY_CONTAINER"
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
type: reference
|
||||
title: Agent Docs and Edits
|
||||
description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
|
||||
tags: [agents, docs, markdown, hashline, edit-recovery]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Agent Docs and Edits
|
||||
|
||||
## Doc-structure audit (after parallel agent edits)
|
||||
|
||||
Use after one or more subagents have edited markdown docs (OKF/Obsidian
|
||||
vault with `---` frontmatter), before committing. Link checkers and
|
||||
completeness seams validate *references*, not *structure* — agents reliably
|
||||
produce structurally broken markdown that still links cleanly.
|
||||
|
||||
Damage catalogue:
|
||||
|
||||
1. Banner/prose prepended **above** the opening `---` → the frontmatter is
|
||||
no longer frontmatter (title/tags/description become body text).
|
||||
2. A rewrite **appended** instead of merged → duplicate headings and a stray
|
||||
second `---` mid-file.
|
||||
3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
|
||||
`+- `, `- - `.
|
||||
4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
|
||||
5. Line-range replacement that dropped a sentence tail → dangling fragment,
|
||||
often mid-parenthesis.
|
||||
|
||||
Checks (fast, run all):
|
||||
|
||||
```bash
|
||||
head -3 <file> # must be `---` if the page has frontmatter
|
||||
grep -c '^---$' <file> # exactly 2 (opener + closer)
|
||||
grep -n '^## ' <file> | sort -t: -k2 | uniq -f1 -d # duplicate sections
|
||||
grep -rnE '^\+- |^- - |<\|' <files> # artifacts
|
||||
python3 scripts/check-vault-links.py <vault-dir>
|
||||
./tests/run.sh # docs completeness seam etc.
|
||||
```
|
||||
|
||||
Repair order (matters):
|
||||
|
||||
1. If the file was destroyed rather than edited: stash the agent version to
|
||||
`.AGENTTRASH/<YYYYMMDD-topic>/`, then `git checkout HEAD -- <file>` and
|
||||
re-apply only the intended changes by hand.
|
||||
2. If it merely duplicated: **salvage unique content first** — cut the
|
||||
one-of-a-kind section, paste it into the copy you keep, THEN delete the
|
||||
duplicate ranges. Never delete first.
|
||||
3. Re-add missing sentence tails (insert before the orphaned fragment),
|
||||
keeping list continuation lines indented.
|
||||
4. Re-run all checks above.
|
||||
|
||||
Gotchas: agents can write AFTER they appear finished (job settles, file
|
||||
changes) — re-verify structure after the last background job reports, not
|
||||
before. Editing an agent's already-damaged file with fresh line numbers is
|
||||
safe, but its snapshot tag changes on every edit — re-read before each
|
||||
subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so
|
||||
nothing moved there reaches the repo — that is intended (temporary hold, not
|
||||
archive).
|
||||
|
||||
## Hashline edit recovery
|
||||
|
||||
Failure mode: multi-op edits resolve every `PUT N.=M:` / `CUT N.=M:` index
|
||||
against the **original** file numbering, and the post-apply diff view
|
||||
**elides lines** (it shows an anchor window, not the whole file). Two traps
|
||||
follow:
|
||||
|
||||
1. A second edit call built from a diff-view snapshot uses line numbers from
|
||||
a *partial* view. Lines outside the window are invisible but still
|
||||
counted.
|
||||
2. If the first apply was itself wrong, the file shifted. The diff view
|
||||
tells you the damage, but not the true offsets.
|
||||
|
||||
Result: the repair pastes fresh text **beside** stale text instead of over
|
||||
it.
|
||||
|
||||
Procedure:
|
||||
|
||||
1. **Never carry line numbers across calls.** Re-read the target range for a
|
||||
fresh tag before every subsequent edit. A read is cheap; a duplicated
|
||||
paragraph costs a commit and a revert.
|
||||
2. **One op per call when the previous apply was imperfect.** Fixed offsets
|
||||
are only trustworthy when every preceding apply landed exactly as
|
||||
specified.
|
||||
3. **Re-read after every apply**, and verify the whole affected region — the
|
||||
diff view is a sample, not proof. Check the *boundaries*: the line before
|
||||
the edit and the line after it. Damage shows up as a repeated sentence
|
||||
straddling the seam.
|
||||
4. **Repair by CUT, not by re-PUT.** Once you know the exact stale lines
|
||||
from a fresh read, delete them. Do not try to PUT a corrected superset
|
||||
over a range whose extent you are guessing at.
|
||||
5. Verify with a content assertion, not a diff review: `grep -c` for the
|
||||
duplicated sentence, plus a syntax/compile check on any touched script.
|
||||
|
||||
Same rule for any imperative `edit` tool with an old/new match: the matched
|
||||
text must be the **entire** text being replaced. Matching a bare fragment
|
||||
inside a paragraph splices the replacement into the middle of the sentence
|
||||
and silently corrupts the file. Re-read the file afterwards to confirm.
|
||||
|
||||
### Anchored in-page links
|
||||
|
||||
Renaming a Markdown heading breaks `[[#Old Heading]]` anchors. When a
|
||||
section is retitled, grep for the old heading text and fix every anchor in
|
||||
the same pass — doc link-checkers frequently do not validate `#fragment`
|
||||
targets, so a green suite will not catch a dangling one.
|
||||
|
||||
## Related
|
||||
|
||||
- [[OMP Harness]] — the session context these repairs protect
|
||||
- [[Command Center Dotfiles]] — the repo these edits commit to
|
||||
- [[Testing Policy]] — the seams that catch broken vault structure
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
type: reference
|
||||
title: Command Center Dotfiles
|
||||
description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
|
||||
tags: [dotfiles, omp, deploy, rsync, blankpad]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Command Center Dotfiles
|
||||
|
||||
Repo `~/.omp` is itself the deployment: packages are home-relative trees
|
||||
under `dotfiles/<pkg>/`; `scripts/deploy.sh` symlinks live files into the
|
||||
repo checkout. `git pull` + `deploy.sh` is the whole propagation story.
|
||||
|
||||
## Transport decision (agreed with Frank, executed 2026-09-09)
|
||||
|
||||
- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to
|
||||
`<user>@<host>:/home/<user>/.omp/`, NOT clone-from-Gitea — `main` can be
|
||||
ahead of `origin/main`; push is a user-only trigger, so clone would ship
|
||||
stale state.
|
||||
- Deploy runs on-target as the target user: `bash
|
||||
/home/<user>/.omp/scripts/deploy.sh` (resolves the repo via `git -C
|
||||
$(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be
|
||||
present at the target path — full-replica transport guarantees this).
|
||||
|
||||
## Pre-overwrite audit (MANDATORY before first sync on a non-blank host)
|
||||
|
||||
Diff every file that `deploy.sh --check` reports as DRIFT against the repo
|
||||
copy BEFORE letting quarantine-and-replace run. Host-specific divergence
|
||||
belongs in an override package, not in quarantine:
|
||||
|
||||
1. Pull divergent live files from target
|
||||
(`ssh ... 'cat <path>' > local-staged-copy`).
|
||||
2. Secret-audit staged copies (grep token/secret/password/key shapes)
|
||||
before anything enters git.
|
||||
3. Compose `dotfiles/zz-<hostname>/` on the source machine from those live
|
||||
copies; commit to repo first.
|
||||
4. Ship the package dir via rsync, then run deploy sync — override deploys
|
||||
LAST and win over base packages.
|
||||
|
||||
## deploy.sh mechanics (verified on blankpad)
|
||||
|
||||
- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it
|
||||
first, always.
|
||||
- Default mode = sync: adopt identicals silently, link new,
|
||||
quarantine-and-replace divergents into `.quarantine/<host>/<ts>/`
|
||||
(gitignored; never rm).
|
||||
- **Override-target bug (fixed cc2c652):** `zz-<host>` packages nest
|
||||
base-package dirs internally (`zz-blankpad/niri/.config/...`); target
|
||||
derivation must strip the nested package segment, else overrides deploy to
|
||||
phantom `$HOME/niri/.config/...` while real targets keep pointing at
|
||||
canonical copies (visible symptom: check shows 0 actions but
|
||||
`readlink ~/.zshrc` points at base `zsh/.zshrc`).
|
||||
- systemd units auto-enable only when ExecStart binary resolves on that
|
||||
host; dconf packages load via `dconf load /`, never symlinks.
|
||||
- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once
|
||||
post-transfer even before first sync.
|
||||
|
||||
## ssh/sshpass transport gotchas (all bit live)
|
||||
|
||||
- Resolve the target via mDNS FQDN (`avahi-resolve -4 -n <host>.local`); a
|
||||
bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or
|
||||
firewall default-drop, not auth failure — needs a local/console fix on
|
||||
target.
|
||||
- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; an env passed
|
||||
via the tool's env parameter did not propagate to the sshpass child
|
||||
process.
|
||||
- The remote login shell may be **fish**: never send bash syntax
|
||||
(`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash -c
|
||||
"..."` or ship a script. See [[SSH and Remote Access]].
|
||||
- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A
|
||||
here-doc attached to the ssh command is consumed by `cat` — the intended
|
||||
script body becomes the file content, silently. Verify shipped bytes with
|
||||
an md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat`
|
||||
drained it (false DIVERGED).
|
||||
|
||||
## Verification of a completed deploy
|
||||
|
||||
Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped
|
||||
= idempotent steady state. Spot-check `readlink` on override files points
|
||||
into `dotfiles/zz-<host>/`, not base packages.
|
||||
|
||||
## Related
|
||||
|
||||
- [[SSH and Remote Access]] — the transport mechanics this deploy rides
|
||||
- [[Noctalia and Niri]] — the desktop packages this repo ships
|
||||
- [[Deploy State Triage]] — the serverAppConfig deploy flow (different repo)
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
type: reference
|
||||
title: OMP Harness
|
||||
description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
|
||||
tags: [omp, harness, compaction, models-yml, context, samplers]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# OMP Harness
|
||||
|
||||
The harness reaches the fleet through `~/.omp/agent/models.yml` (model
|
||||
registry; single source of truth — there is no copy on the servers) and
|
||||
`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
|
||||
this file is repo-managed; edits are harness-side only.
|
||||
|
||||
## Compaction chain arithmetic (verify before ANY window change)
|
||||
|
||||
When a fleet endpoint's context grows past its harness-declared window, OMP
|
||||
compacts by sending the conversation to a designated target model. Before
|
||||
changing any `contextWindow`/`maxTokens`, verify the chain arithmetically —
|
||||
a target whose window is smaller than the source's trigger point silently
|
||||
breaks compaction.
|
||||
|
||||
Where the contract lives:
|
||||
|
||||
- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL),
|
||||
`methodOrder`, `thresholdPercent` (70), idle settings
|
||||
- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
|
||||
`compactionModel` references
|
||||
- Implementation: omp plugin `dist/core/compaction/compaction.js`
|
||||
|
||||
The arithmetic (the coupled contract every endpoint's maxTokens comment
|
||||
encodes):
|
||||
|
||||
1. **Source trigger** = `thresholdPercent × source contextWindow`
|
||||
(0.70 × 200,704 = 140,492 for frank-primary on the current shape)
|
||||
2. **Output budget** = capped at the target model's `maxTokens`
|
||||
(32,768 for frank-small; 36,864 frank-primary; 57,344 frank-large — the
|
||||
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
|
||||
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
|
||||
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
|
||||
200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
|
||||
(frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-small).
|
||||
|
||||
Procedure (recompute from the LIVE files every time — windows rotate with
|
||||
every swap; the numbers above are the 2026-10-05 state):
|
||||
|
||||
1. Read the `config.yml` compaction block + `models.yml` entries for source
|
||||
and target. Every endpoint's `contextWindow` comment carries its own
|
||||
arithmetic — read them.
|
||||
2. Read the target checkpoint's `max_position_embeddings` (native window) —
|
||||
the served window may be a concurrency contract, not the native limit
|
||||
(`--parallel N` DIVIDES the pool; the per-slot number is the contract).
|
||||
3. If native ≥ trigger + budget, reslice the pinned pool:
|
||||
`MAX_RUNNING × CONTEXT_LENGTH ≤ MAX_TOTAL_TOKENS` (the pin exists because
|
||||
the runtime's autofit leaves too little for deferred loads + prefill
|
||||
workspace; check `nvidia-smi` headroom — expect ~0.8 GB workspace margin).
|
||||
4. Changes: host env (CONTEXT_LENGTH-style env tunables) via git →
|
||||
`./tests/run.sh` → deploy; harness `models.yml` directly (no repo copy).
|
||||
5. Verify: `/health` 200, boot log's pool line, then a long-prompt probe at
|
||||
≥ trigger height (the compaction-fit guarantee), then concurrent requests.
|
||||
|
||||
Known reconciliation trap: comments in models.yml have described compaction
|
||||
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
|
||||
dankclaw — the comment/config disagreed. The comments now name the real
|
||||
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
|
||||
truth; fix comments in the same change.** `maxTokens` never changes for a
|
||||
reslice — it caps output; `contextWindow` caps input. Boot-shape OOM
|
||||
laddering is a different problem ([[Llama Cpp Serving Sizing]]).
|
||||
|
||||
## Per-model sampler config (`compat.extraBody`)
|
||||
|
||||
`models.yml` has **no per-model sampler keys**. Sampling is global
|
||||
(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
|
||||
`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is
|
||||
`compat.extraBody`:
|
||||
|
||||
**Why it works:** `applyOpenAIExtraBody` runs **last** in the
|
||||
openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
|
||||
`Object.assign(params, extraBody)`. So extraBody overrides global settings
|
||||
and compat-policy output.
|
||||
|
||||
```yaml
|
||||
compat:
|
||||
reasoningEffortMap:
|
||||
max: xhigh
|
||||
extraBody:
|
||||
temperature: 1.0
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
min_p: 0.0
|
||||
presence_penalty: 0.0
|
||||
repetition_penalty: 1.0
|
||||
preserve_thinking: true
|
||||
```
|
||||
|
||||
Use **wire names**, not OMP camelCase. Two traps:
|
||||
|
||||
1. **`preserve_thinking` is not user-settable as a compat key.**
|
||||
`qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
|
||||
*auto-detected* `thinkingFormat` + local baseUrl, **before**
|
||||
`applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
|
||||
models.yml does **not** enable it. Custom-named models (`frank-large`)
|
||||
match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
|
||||
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
|
||||
also sets it — `Object.assign` is shallow and replaces the whole object,
|
||||
silently dropping the `reasoning_effort` kwarg the compat policy set.
|
||||
Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
|
||||
FreeToken entry currently carries a `chat_template_kwargs` extraBody with
|
||||
a comment asking to verify-and-prune — the exact trap to check at that
|
||||
endpoint's next swap.)
|
||||
|
||||
**Gotcha: silent greedy decode.** halogen has **no server-side sampler
|
||||
defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
|
||||
endpoint decodes greedy while looking configured. This is why frank-large's
|
||||
entry carries the recipe on the wire and a comment saying "do not drop it".
|
||||
|
||||
**Reasoning-effort folding.** Qwen3.8's template accepts only
|
||||
`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
|
||||
else ("Unexpected reasoning effort high"). Every endpoint's
|
||||
`reasoningEffortMap` folds all harness levels onto that set; a role whose
|
||||
level is unmapped fails on its FIRST request, and the failure names Jinja,
|
||||
not the model.
|
||||
|
||||
### Verify the exact wire body (echo-capture)
|
||||
|
||||
Don't trust the YAML — capture what OMP actually emits:
|
||||
|
||||
1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and
|
||||
returns a canned `chat.completion` (include `reasoning_content` so
|
||||
thinking paths exercise).
|
||||
2. Copy the **real** models.yml into a throwaway profile with only the
|
||||
baseUrl swapped:
|
||||
```bash
|
||||
sed 's|http://<host>:7770/v1|http://127.0.0.1:18099/v1|' \
|
||||
~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml
|
||||
```
|
||||
3. Run and inspect:
|
||||
```bash
|
||||
omp --profile wiretest --model "<provider>/<model>" -p "hi" \
|
||||
--no-tools --no-session --no-extensions --no-skills --no-rules
|
||||
jq 'del(.messages, .tools)' last-body.json
|
||||
```
|
||||
Confirm every intended field is present with the right value.
|
||||
|
||||
**Verify a field is *honoured*, not just accepted** — unknown fields often
|
||||
return 200 anyway. Prove behavior with a **prompt-token delta**: replay an
|
||||
assistant turn carrying a long `reasoning_content`, then compare
|
||||
`usage.prompt_tokens` with the flag true vs false. A large delta means the
|
||||
field changed template rendering; zero delta means it was ignored.
|
||||
|
||||
Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
|
||||
never `rm`.
|
||||
|
||||
## Static-preamble audit ("the context feels bloated")
|
||||
|
||||
What loads statically every session (measure each, don't guess):
|
||||
|
||||
1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k
|
||||
tokens — historically the largest block).
|
||||
2. Skill index — only names+descriptions load; bodies are on-demand. Sum
|
||||
description bytes across all SKILL.md frontmatter:
|
||||
```bash
|
||||
python3 - <<'EOF'
|
||||
import glob, re, os
|
||||
for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
|
||||
os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
|
||||
n=tot=0
|
||||
for sk in glob.glob(root+"/*/SKILL.md"):
|
||||
fm=re.search(r'^---.*?---', open(sk, encoding="utf-8", errors="replace").read(), re.S)
|
||||
d=re.search(r'description:\s*(.*(?:\n(?:\s{2,}.*)*)?)', fm.group(0) if fm else "")
|
||||
n+=1; tot+=len(d.group(1)) if d else 0
|
||||
print(root, n, tot)
|
||||
EOF
|
||||
```
|
||||
Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user
|
||||
skills.
|
||||
3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name
|
||||
'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest
|
||||
MEMORY.md.
|
||||
4. Harness/system prompt + tool schemas + CLI persona: fixed cost, ~20 KB;
|
||||
not prunable.
|
||||
|
||||
Known rot patterns:
|
||||
|
||||
- Near-duplicate auto-captures of the same lesson (three "authentik ORM"
|
||||
entries, the freetoken-build pair, the three job-runner skills) — dedupe
|
||||
via `manage_skill delete`/`update`.
|
||||
- Corrupted captures containing stray CJK characters (`一团ly`, `等方法`)
|
||||
mid-sentence — a known defect of the auto-capture path; delete the corrupt
|
||||
copy, keep the clean one.
|
||||
- Stale host/service names in consolidated memory (pre-rename
|
||||
`frank-default-large`) — the repo's AGENTS.md topology table is ground
|
||||
truth.
|
||||
|
||||
Fix strategy: AGENTS.md prose (measured runbook narratives) → move to
|
||||
`serverAppConfigs-docs/LLM/` runbooks, keep laws + tables in AGENTS.md (the
|
||||
repo's own docs-tree rule mandates this). Managed skills live in
|
||||
`~/.omp/agent/managed-skills` (safe to edit); user-authored skills in
|
||||
`~/.omp/agent/skills` are NEVER touched by tooling. Memory store edits:
|
||||
stage removed captures to trash first (`~/.AGENTTRASH/YYYYMMDD-topic`).
|
||||
|
||||
Baseline to beat (2026-10-05): ~35–40k tokens of preamble — AGENTS.md ~18k,
|
||||
skill index ~10k, lessons ~3.5k, harness ~6k. Proposed reductions: −12k
|
||||
(AGENTS.md slimming), −3k (skill dedupe).
|
||||
|
||||
## Managed-skill library hygiene
|
||||
|
||||
Use when auditing, pruning, or consolidating `~/.omp/agent/managed-skills`
|
||||
(the auto-captured skill store; the full index is injected into EVERY
|
||||
session's context, so dead or duplicate entries are a permanent tax).
|
||||
|
||||
The static-context budget (measured 2026-10-05):
|
||||
|
||||
| Block | Size | Notes |
|
||||
|---|---|---|
|
||||
| `AGENTS.md` (repo) | 72,179 B ≈ 18k tok | biggest single item |
|
||||
| Skill index (~173 skills) | ~40 KB ≈ 10k tok | names+descriptions only |
|
||||
| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures |
|
||||
| Harness system prompt + tool schemas | ~23 KB | fixed |
|
||||
|
||||
Audit procedure:
|
||||
|
||||
1. Build an inventory: per skill, name, mtime, body word count, description,
|
||||
first 12 headings (one pass, out to a file; read in pages).
|
||||
2. Group into: retired-runtime (host/service gone; its runbook already lives
|
||||
in `serverAppConfigs-docs/LLM/`), verbatim-duplicate pairs/triples (same
|
||||
lesson captured twice — often the SAME DAY), same-lesson families
|
||||
(N skills = one procedure), keep.
|
||||
3. Read full bodies of every merge candidate before proposing a merge;
|
||||
description alone proves overlap only for exact dupes.
|
||||
4. Present A/B/C/D verdicts for user approval. NEVER execute before the
|
||||
discussion concludes — the user gates execution explicitly.
|
||||
5. On approval: stage everything to trash first
|
||||
(`~/.AGENTTRASH/YYYYMMDD-<topic>/`), never rm. After merging: recreate
|
||||
the surviving skill via `manage_skill` update with body = superset,
|
||||
delete absorbed ones.
|
||||
|
||||
Known rot classes (this library, 2026-10-05):
|
||||
|
||||
- Same-day duplicate captures (freetoken-build-dankclaw vs
|
||||
freetoken-dankclaw-build; the three job-runner skills).
|
||||
- CUDA-skill vs its self-declared HIP twin (pcie-link-capability-probe vs
|
||||
pcie-link-probe-hip-rdna4) → merge.
|
||||
- Era ghosts: skills naming retired runtimes (freetoken, lucebox, r9v/
|
||||
radiance, old halogen-flash-server image, ik_llama, GLM-5.3 experiment).
|
||||
- Skills contradicting user standing orders (live-bench-dashboard vs the
|
||||
2026-09-29 delete-dashboards order) — flag for user, don't assume.
|
||||
- Memory store: corrupted captures with CJK noise from broken ssh commands;
|
||||
stale host/service renames.
|
||||
|
||||
Sizing verdicts that worked: bench family → 2 skills (harness vs
|
||||
discipline), not one mega-skill; keep profiler/kernel-time attribution
|
||||
standalone; fold go/no-go arithmetic skills into their probe skill
|
||||
(heterogeneous-accelerator-split-triage → PCIe probe).
|
||||
|
||||
## Related
|
||||
|
||||
- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
|
||||
- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
|
||||
- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
|
||||
- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
type: index
|
||||
description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
|
||||
tags: [agent, index]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Agent
|
||||
|
||||
Pages about the agent's own tooling and context — how omp is configured, how
|
||||
docs get edited safely, and how the agent's home is deployed.
|
||||
|
||||
- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
|
||||
`compat.extraBody`, preamble and skill-library audits
|
||||
- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
|
||||
hashline edit recovery
|
||||
- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
|
||||
transport, override packages
|
||||
|
||||
## Related
|
||||
|
||||
- [[Testing Policy]] — the seams these pages' checks ride
|
||||
- [LLM](../LLM/index.md) — the fleet these agents operate
|
||||
@@ -0,0 +1,151 @@
|
||||
---
|
||||
type: reference
|
||||
title: Deploy State Triage
|
||||
description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
|
||||
tags: [deployment, deploy-state, systemd, podman, triage]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Deploy State Triage
|
||||
|
||||
The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||||
on each target (rootless, user `dank`, format exactly two lines
|
||||
`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
|
||||
+ `mv`). It is written ONLY by a `deploy.sh` dispatch.
|
||||
|
||||
## 1. Stale marker vs live container
|
||||
|
||||
**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
|
||||
a restart after the checkout ff-pulled to a newer revision — starts the
|
||||
container from the CURRENT `scripts/<svc>.env` but does NOT advance the
|
||||
marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
|
||||
"behavior pending".
|
||||
|
||||
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
||||
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
|
||||
a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
|
||||
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
||||
possibly zero behavioral change.
|
||||
|
||||
**Procedure (read-only first):**
|
||||
|
||||
1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
|
||||
operational delta is usually one `scripts/<svc>.env`.
|
||||
2. Probe the target (ssh as `dank`, passwordless sudo present):
|
||||
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||||
- `git -C ~/serverAppConfigs rev-parse HEAD`
|
||||
- `systemctl is-active/is-enabled <svc>`;
|
||||
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
|
||||
- the on-host env file already carries the new keys (it landed via the
|
||||
ff-pull).
|
||||
3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
|
||||
`dank` (rootless store; bare `docker` is NOT the shim → "No such object";
|
||||
`sudo podman` inspects an empty store):
|
||||
- `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
|
||||
the pull ⇒ it read the new env
|
||||
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
|
||||
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
|
||||
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
|
||||
- confirm the launcher maps each tunable to its flag
|
||||
(`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
|
||||
--enable-prefix-caching`.
|
||||
|
||||
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
|
||||
live — **do NOT restart**; reconcile the marker (below). Only if the live
|
||||
container lacks the flag/env (or predates the pull) is a real
|
||||
`deploy.sh <target> --service <svc>` warranted.
|
||||
|
||||
**Reconcile (no restart)** — mirror the deployer's atomic idiom:
|
||||
|
||||
```bash
|
||||
S=$HOME/.local/state/serverAppConfigs/deploy-state
|
||||
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
|
||||
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
|
||||
```
|
||||
|
||||
Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
|
||||
`tests/e2e/.env.local`; expect the documented `11-admin` red if the
|
||||
FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
|
||||
a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
|
||||
Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
|
||||
no-op).
|
||||
|
||||
**Anti-patterns:** trusting a commit message ("X deployed") or the marker
|
||||
alone — both are bookkeeping; the container's argv/env is ground truth.
|
||||
Reading container state with `sudo podman` / bare `docker`. Restarting a
|
||||
healthy heavy endpoint to satisfy a stale marker.
|
||||
|
||||
## 2. Pending lockout
|
||||
|
||||
Symptom: `deploy.sh <target>` fails or silently stops with
|
||||
`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
|
||||
appears.
|
||||
|
||||
1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
|
||||
reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
|
||||
`echo $?`.
|
||||
2. Inspect the state store (no sudo):
|
||||
`ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
|
||||
A `status=pending` entry for a service absent from `service_is_allowed`
|
||||
in `scripts/deploy-remote.sh` = retired-service lockout.
|
||||
3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
|
||||
not-allowed pending entries to
|
||||
`~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
|
||||
continues. Verify the WARN line names the service and the state file is
|
||||
gone.
|
||||
4. On a host without the fix: move the stale entry to
|
||||
`~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
|
||||
5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
|
||||
`deploy-state/` no longer holds the retired name.
|
||||
|
||||
Related contracts: the superseded seam
|
||||
(`deployment_scoping.bats` "remote runner recognizes a pending service at
|
||||
the same revision") now asserts prune-and-continue (exit 0), not exit 2;
|
||||
`deployment_flow.bats` extracts `add_pending_services` via sed and
|
||||
HOME-overrides the trash path. **When retiring a service going forward:**
|
||||
delete or reconcile its deploy-state entry as part of the retirement commit —
|
||||
the prune is a safety net, not the intended cleanup path.
|
||||
|
||||
## 3. Interrupted or raced deploys
|
||||
|
||||
The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
|
||||
(runs on the target over ssh). **Killing the dev-side job does NOT kill the
|
||||
host-side one** — two `deploy-remote.sh` processes can end up managing the
|
||||
same unit, one deploying a revision you already superseded. Check:
|
||||
`pgrep -af "deploy-remote"` on the target, read the revision argument on
|
||||
each, kill stale ones by explicit PID.
|
||||
|
||||
A killed deploy leaves behind:
|
||||
|
||||
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
|
||||
explicit stop — start it yourself);
|
||||
- the state marker **pending** (pre-deploy or in-flight revision);
|
||||
- the e2e gate un-run (the deployer runs it last).
|
||||
|
||||
Reconcile the marker ONLY after the service itself is verified healthy — a
|
||||
marker that runs ahead of reality is worse than one that lags (the planner
|
||||
uses it to decide what is pending). Then close the loop:
|
||||
|
||||
1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
|
||||
slots, argv flags, `NRestarts`), not just health 200.
|
||||
2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
|
||||
that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
|
||||
does NOT have it; copy it in first).
|
||||
3. If the killed revision was restoring a known-good shape, confirm the
|
||||
numbers match the last measured-good boot of that shape — equality is the
|
||||
proof, not health alone.
|
||||
|
||||
Boot failures must be read from the journal, not `podman logs`: serve scripts
|
||||
`podman rm -f` the container on every start (previous attempt lost), and
|
||||
`podman logs` on a container still *Stopping* returns the OLD configuration's
|
||||
numbers — which looks exactly like your change having no effect. Key on
|
||||
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
|
||||
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
|
||||
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
|
||||
procedure.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Deployment Flow]] — the deploy pipeline itself
|
||||
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
|
||||
@@ -0,0 +1,209 @@
|
||||
---
|
||||
type: reference
|
||||
title: Quadlet Lifecycle
|
||||
description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
|
||||
tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Quadlet Lifecycle
|
||||
|
||||
Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
|
||||
working container MUST have a self-healing, boot-persistent Quadlet unit —
|
||||
`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
|
||||
deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
|
||||
A container that runs only from compose dies silently at reboot. This page
|
||||
is the operating manual for that lifecycle.
|
||||
|
||||
## 1. Ownership audit — find the containers that will not survive a reboot
|
||||
|
||||
```bash
|
||||
for c in $(podman ps --format '{{.Names}}'); do
|
||||
printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
|
||||
done
|
||||
```
|
||||
|
||||
- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
|
||||
show-user <user> -p Linger` = yes.
|
||||
- **label absent, compose service exists** → the classic gap: dies at reboot.
|
||||
Check the publish address while there — compose forms often bind the
|
||||
wildcard (an unauthenticated UI/API on every interface).
|
||||
- **label present AND a compose service with the same name** → two owners; a
|
||||
stray `podman compose up` can create a duplicate that fights the unit for
|
||||
the name and the port. Remove the compose service, leave an explanatory
|
||||
comment (repo convention).
|
||||
|
||||
## 2. Cutover: compose service → Quadlet unit
|
||||
|
||||
Checklist (precedents to copy:
|
||||
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
|
||||
`dankclaw/observability/quadlet/`):
|
||||
|
||||
1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
|
||||
line for line: `Image=` (`localhost/` prefix for locally built images +
|
||||
`Pull=never`), `ContainerName=` (Caddy and scrape configs address
|
||||
containers by name), `Network=`, `User=0:0` when the image's own user
|
||||
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
|
||||
`Exec=` for compose `command:`, `PublishPort=`, health keys,
|
||||
`[Service] Restart=always` + `RestartSec=10`, `[Install]
|
||||
WantedBy=default.target`. **Never `systemctl --user enable` the generated
|
||||
unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
|
||||
is healthy; explicit enable fails with "transient or generated").
|
||||
2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
|
||||
it (Caddy reaches containers by name over the shared network; the wildcard
|
||||
publish is a guard-chain bypass).
|
||||
3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
|
||||
`systemctl --user restart <name>.service` → `wait_for_health`; retire any
|
||||
legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
|
||||
`~/.config/systemd/user/<svc>.service` that shells out to
|
||||
`podman compose up` collides BY NAME with the generated unit);
|
||||
`podman rm -f <container>` so the unit's container can take the name.
|
||||
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
|
||||
AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
|
||||
ordered. Add a `test-changed` case above the generic
|
||||
`*/quadlet/*.container` fallback.
|
||||
5. Retire the compose entry in the SAME change — two owners is how a stale
|
||||
duplicate comes back.
|
||||
6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
|
||||
wildcard publish, container name matches Caddy/scrape references, health
|
||||
command, deployer installs+restarts+waits, never `rm`, never `enable`.
|
||||
Migrate any older assertion that read the compose file for devices/flags.
|
||||
7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
|
||||
active/generated, container `healthy`, endpoint answers,
|
||||
`list-dependencies default.target` shows the unit, off-host probes on
|
||||
previously published ports now refuse, and the container no longer carries
|
||||
`io.podman.compose.project`.
|
||||
|
||||
**Traps that cost real time:**
|
||||
|
||||
- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
|
||||
no shell → the check fails forever. `--health-cmd` won't take exec form
|
||||
either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
|
||||
with `--format docker` — the default OCI format has no HEALTHCHECK field
|
||||
and drops it with only a warning.
|
||||
- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
|
||||
gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
|
||||
the unit dies with `stat /root/.config: permission denied`. Use a
|
||||
per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
|
||||
handler with a single-variable filter, validated so a missing secret fails
|
||||
loudly.
|
||||
- `Network=host` exporters scraped by containerized Prometheus must bind the
|
||||
host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
|
||||
for the homepage probe — two `--web.listen-address` flags, never the
|
||||
wildcard.
|
||||
- alloy 1.x refuses the `ui` config block; the unit must pass
|
||||
`--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
|
||||
- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
|
||||
no supplementary groups).
|
||||
- When the last service leaves a compose file, the compose-validity seam
|
||||
fails (requires ≥1 service): keep the file as comments and relax the seam
|
||||
deliberately, or delete the file and fix the planner's compose-diff path —
|
||||
decide with the owner, the deploy tooling reads it.
|
||||
|
||||
## 3. Drift / crash-loop diagnosis (5 minutes, in order)
|
||||
|
||||
Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
|
||||
looks right; or a service quietly down for days because a restart policy kept
|
||||
something running. Cause: the running container was created from an OLDER
|
||||
definition — a compose/unit edit does not touch an existing container, and
|
||||
the restart policy faithfully resurrects it forever.
|
||||
|
||||
1. **Who started it:** `podman inspect <name> --format 'unit={{index
|
||||
.Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
|
||||
created={{.Created}}'`; compare `created` with when the definition changed
|
||||
(`git log -p --follow <config>`). Created-before-the-edit is the signature.
|
||||
2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
|
||||
`{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
|
||||
for containers started with `--device`/`--group-add` — a false negative
|
||||
that sends you hunting the wrong root cause.
|
||||
3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
|
||||
(a capability error like `No HIP GPUs are available` beside a missing
|
||||
device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
|
||||
instance reached 9,146 restart attempts on stale config.
|
||||
|
||||
Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
= the §2 cutover. **Verify the capability, not the status:** `podman exec
|
||||
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
|
||||
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
|
||||
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
|
||||
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
|
||||
|
||||
## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
|
||||
|
||||
- **Hard-fail keys** — the whole unit silently fails to generate
|
||||
(`systemctl --user cat <unit>` shows "could not be found"): any
|
||||
dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
|
||||
in group 'Container'`); `Command=` is unsupported, use `Exec=`.
|
||||
- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
|
||||
spaces into the healthcheck argv; a quoted probe
|
||||
`/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
|
||||
command-not-found → container reads **unhealthy while the DB is up**.
|
||||
HealthCmd must contain only space-free tokens
|
||||
(`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
|
||||
unhealthy but logs look fine, compare the baked
|
||||
`{{.Config.Healthcheck.Test}}` array against the intended command before
|
||||
believing the status.
|
||||
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
|
||||
`daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
|
||||
"unsupported key"`, `systemctl --user cat probe.service` (shows the
|
||||
generated ExecStart verbatim), then remove + reload.
|
||||
|
||||
## 5. Rootless Quadlet log retrieval
|
||||
|
||||
Step 0 — establish ownership: `systemctl --user status <name>.service`
|
||||
(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
|
||||
`--user` knows the unit, EVERY command runs as that user with NO sudo.
|
||||
|
||||
| Need | Command |
|
||||
|---|---|
|
||||
| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
|
||||
| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
|
||||
| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
|
||||
| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |
|
||||
|
||||
Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
|
||||
recreation** — the journal is the only durable record. Traps:
|
||||
`sudo podman logs` → "no such container" (root's store does not contain
|
||||
rootless containers; same for `sudo podman ps`); a system-level
|
||||
`journalctl -u <name>` returns nothing for a user unit; during first-start
|
||||
weight downloads health reports unhealthy BY DESIGN (check the payload file
|
||||
is growing). Verbose vendor switches are a config change — through git and
|
||||
deploy.sh, never a live env poke.
|
||||
|
||||
## 6. Compose bring-up with ground truth (rootless hosts)
|
||||
|
||||
1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
|
||||
check every namespace (`sudo podman ps -a`) and native units
|
||||
(`systemctl list-units`) before declaring a service absent. Compare
|
||||
compose mount sources against the real filesystem — compose files name
|
||||
phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
|
||||
`/mnt/entropy`). Fix compose through git first, deploy second.
|
||||
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
|
||||
or creation fails `statfs …: no such file or directory`.
|
||||
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
|
||||
`{{.State.Health.Status}}`, then the rest. Fresh containers answer
|
||||
000/refused for the first seconds — poll.
|
||||
4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
|
||||
request for nonexistent/private repos ("manifest unknown"/token len 0 =
|
||||
repo or tag does not exist, not auth). Query the registry API
|
||||
(`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
|
||||
`/v2/<ns>/<repo>/tags/list`).
|
||||
5. **Read the interface from inside the image** before guessing flags/envs:
|
||||
`podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
|
||||
means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
|
||||
entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
|
||||
flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
|
||||
configured it serves cwd read/write for everyone (an unauthenticated
|
||||
world-writable share).
|
||||
6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
|
||||
mangled pattern is not absence — re-verify the target exists.
|
||||
7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
|
||||
(host 1000 == container 0): drop the user line, run as container-root.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Deployment Flow]] — the deployer that installs these units
|
||||
- [[Stale Signal Audit]] — the audit that catches drift's victims
|
||||
- [[Container Network Namespaces]] — reachability between containers
|
||||
- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
|
||||
- [[Infrastructure/Observability]] — same class, twice
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
type: reference
|
||||
title: Stale Signal Audit
|
||||
description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
|
||||
tags: [testing, observability, audit, signals, gates]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Stale Signal Audit
|
||||
|
||||
Run after any change and before trusting a green gate. The productive question
|
||||
is **which signal here cannot possibly be correct?** Every pass on this repo
|
||||
(4 of 4) found real defects — including one in the rule file every future
|
||||
agent reads first.
|
||||
|
||||
## The surfaces, and what each vantage point can reach
|
||||
|
||||
| Surface | Producer's vantage point | Can reach |
|
||||
|---|---|---|
|
||||
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
|
||||
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
|
||||
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
|
||||
|
||||
Anything else — another host's loopback, a container-internal port never
|
||||
published — is unreachable no matter how healthy the service is. The failure
|
||||
mode is always the same shape: a signal that **cannot** succeed by
|
||||
construction, which teaches the operator to ignore the surface it lives on.
|
||||
|
||||
## 1. Permanently-red probes (reachability)
|
||||
|
||||
For every monitor/probe/alert target ask: *can this checker physically reach
|
||||
that endpoint?* Seen classes: a card probing another host's container-internal
|
||||
port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
|
||||
`/metrics` that is disabled (Gitea 404).
|
||||
|
||||
- A scrape cannot express "is this HTTP service answering?" — the right probe
|
||||
is the **blackbox exporter** in probe mode (which also gives guarded vhosts
|
||||
a real uptime signal, since Prometheus cannot solve an Anubis challenge).
|
||||
- **Fix in the direction of honesty, never of silence:** unreachable target →
|
||||
mark the homepage card `wip: true` with the reason in `description` (the
|
||||
checker skips `wip`; `wip` is the existing convention for "real service,
|
||||
not probeable"), or repoint the job at something reachable. NEVER open a
|
||||
database port on the LAN to make a badge green, and never delete a scrape
|
||||
job — an absent scrape and a healthy one are indistinguishable.
|
||||
- Alert rules must not depend on always-red signals — scope them to services
|
||||
that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
|
||||
exactly that reason).
|
||||
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
|
||||
`connection refused` to `unsupported Content-Type "text/html"` means the
|
||||
service came up and only the scrape is wrong. Read the error, not just
|
||||
up/down.
|
||||
|
||||
## 2. Vacuous assertions (highest severity — the gate is already dead)
|
||||
|
||||
A seam that greps a file passes *by absence* once the file is retired:
|
||||
`grep` exits 2 on a missing path and `!` inverts it.
|
||||
|
||||
```bash
|
||||
grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
|
||||
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
|
||||
```
|
||||
|
||||
Fix shape — one place asserts the invariant, each local site keeps a live
|
||||
check:
|
||||
|
||||
```bats
|
||||
@test "no <host> compose file competes with the quadlet units" {
|
||||
local cf="$REPO_ROOT/<host>/compose.yaml"
|
||||
[ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
|
||||
}
|
||||
# at the per-service site: still fails if the entry returns
|
||||
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
|
||||
```
|
||||
|
||||
**Prove teeth:** recreate the offending file
|
||||
(`printf 'services:\n <svc>:\n image: x\n' > <path>`), run the seams,
|
||||
confirm RED, then move the file to trash. A re-armed check you have never
|
||||
seen fail is still unverified.
|
||||
|
||||
## 3. Gates that never run
|
||||
|
||||
Playwright specs in `tests/e2e/` are matched by **number prefix** in
|
||||
`playwright.config.ts`; an unregistered spec silently never runs and the
|
||||
suite count stays the same. Count tests before/after;
|
||||
`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
|
||||
|
||||
## 4. Validate the checker before acting on it
|
||||
|
||||
A checker's output is a hypothesis about another tool's semantics. Modelling
|
||||
Obsidian with naive text matching produced 45 "broken links" of which 3 were
|
||||
real — the rest were case-sensitivity (a `qBittorrent` link to
|
||||
`qbittorrent.md`), path-qualified resolution (a path-qualified
|
||||
`folder/Note` link resolves by path SUFFIX, not
|
||||
stem), and `%20` URL-encoding in markdown links. Fix the checker first,
|
||||
re-derive the report, only then edit documents. The result lives in
|
||||
`scripts/check-vault-links.py` with those three rules in its docstring,
|
||||
gated by the `no unresolved links in the docs vault` seam. Same class: a
|
||||
`find … | head -1` selector that reads the wrong file; a truncated listing
|
||||
that makes you "fix" a file that exists.
|
||||
|
||||
## 5. Prose invalidated by your own change
|
||||
|
||||
After retiring a file or adding a feature, grep for EVERY reference, not just
|
||||
code:
|
||||
|
||||
```bash
|
||||
grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
|
||||
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
|
||||
```
|
||||
|
||||
This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
|
||||
after it was restored with GPU access; a unit said "there are no alerting
|
||||
rules" hours after five were added; a docs page documented three scrape jobs
|
||||
as working when they could never report up. **Clean cutover includes prose** —
|
||||
docs, unit docstrings, and the repo's rule file (the most expensive place to
|
||||
be stale).
|
||||
|
||||
## Order and verification
|
||||
|
||||
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
|
||||
gates, docs/rule files, dashboard panels).
|
||||
2. For each, ask what the signal CAN express, not what it says.
|
||||
3. Validate any checker before editing the things it flags.
|
||||
4. Fix at the root; never weaken an assertion to make it pass.
|
||||
5. Gate the class so it cannot return (a seam), then prove the seam fails on
|
||||
the defect.
|
||||
|
||||
Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
|
||||
Prometheus panel expression against the API (0 series on a panel that should
|
||||
show data is a defect; 0 `ALERTS` when nothing fires is not); read the
|
||||
deployed `/srv/homepage/status.json` and triage every `state != up` entry
|
||||
against reality; query `/api/v1/targets?state=active` and print
|
||||
`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
|
||||
Report per finding: the surface, why the signal could not have been correct,
|
||||
the fix, and the red/green evidence — flag any regression you introduced
|
||||
yourself in the same breath (one was found this way: a tightened bind killed
|
||||
a reverse-proxied vhost while every gate stayed green, because the spec
|
||||
covering that hostname did not exist).
|
||||
|
||||
## Related
|
||||
|
||||
- [[Testing Policy]] — the seam inventory this audit walks
|
||||
- [[Observability]] — the scrape/probe surfaces it audits
|
||||
- [[Quadlet Lifecycle]] — the ownership labels the probes read
|
||||
@@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto
|
||||
The npx package cache lives in the container layer, so a recreation (not just a
|
||||
restart) pays one first-call npm download; subsequent calls are cached.
|
||||
|
||||
## Config-home drift ("cannot connect" to models that answer curl)
|
||||
|
||||
`podman exec <container> printenv HERMES_HOME` for each hermes container. If
|
||||
the webui reads the image default (`/opt/data`) while the gateway reads
|
||||
`/data`, the webui runs the image-default template config (`provider: auto` →
|
||||
openrouter.ai with no key = connect failure).
|
||||
|
||||
The recreation trap is the same one the volumes table shows: image `VOLUME`
|
||||
points get a fresh anonymous volume on every container recreation
|
||||
(`podman run --replace --rm`) — edits inside such a mount are lost on
|
||||
restart/redeploy. Compare volume ids before/after a restart to confirm.
|
||||
|
||||
Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
|
||||
root), deploy service-scoped. Verify with the app's own code, not guesses:
|
||||
|
||||
```python
|
||||
from api import config as C; C.resolve_model_provider("<model-id>")
|
||||
# -> (model_id, provider, base_url); bare ids map to the owning provider by
|
||||
# scanning the providers: dict models allow-list; @provider:model forces one.
|
||||
hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
|
||||
# -> ProviderDef with real base_url; accepts api/url/base_url keys.
|
||||
```
|
||||
|
||||
Named providers (`providers:` dict) are a catalog + connection source;
|
||||
`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
|
||||
neither holds the provider name, the request has no endpoint and fails to
|
||||
connect. Probe the endpoint with the **exact served model id** — the retired
|
||||
FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
|
||||
with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
|
||||
checks miss this class entirely.
|
||||
|
||||
## Live config edits
|
||||
|
||||
Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
|
||||
uid 109999; all reads/edits need `sudo`). The repo file
|
||||
`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
|
||||
live — always diff the live file first; never treat the example as truth**.
|
||||
|
||||
1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
|
||||
(in-dir `.AGENTTRASH/` also exists for removals).
|
||||
2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
|
||||
the live file are tool-generated — preserve them.
|
||||
3. **Sweep stale references** across the whole data dir:
|
||||
`sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
|
||||
--include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
|
||||
historical — leave them.
|
||||
4. **Derived caches:** `webui/models_cache.json` is keyed by a config
|
||||
fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
|
||||
is stale. Let it regenerate lazily or trash it — never hand-edit.
|
||||
5. **Restart the model-consuming units** (they parse config at startup):
|
||||
`systemctl --user restart hermes-webui.service hermes.service`.
|
||||
`hermes-dashboard.service` is unaffected by routing changes.
|
||||
6. **Verify behaviorally**: one tiny completion per endpoint with the exact
|
||||
served id — POST `/v1/chat/completions` and read the body, never just the
|
||||
status code (endpoint names rotate; see the LLM knowledge base).
|
||||
7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
|
||||
live shape, redacting secrets (`password_hash`, phone account stay
|
||||
live-only), then `./tests/run.sh` before commit.
|
||||
|
||||
`provider: custom` + `base_url` entries route by base_url and send the model
|
||||
id verbatim; named providers resolve via their `api:` URL. **Renaming a
|
||||
served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
|
||||
config in the same change** or Hermes requests 404 on nonexistent models —
|
||||
this exact gap caused the 2026-09-05 outage.
|
||||
|
||||
## Auxiliary routing against local backends
|
||||
|
||||
When a Hermes feature that calls an auxiliary task (e.g. title generation)
|
||||
fails against a local endpoint, the usual cause is a **request shape the
|
||||
local runtime does not implement**, not a routing mistake.
|
||||
|
||||
Diagnose with the real call path — run the actual function inside the
|
||||
container with the app's own constants (the probe cannot drift from
|
||||
production):
|
||||
|
||||
```bash
|
||||
base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
|
||||
'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
|
||||
podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
|
||||
podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
|
||||
```
|
||||
|
||||
Copy the file AFTER any restart (`--replace --rm` discards the container
|
||||
layer). The CLI at `/opt/hermes/hermes` uses system python and dies
|
||||
`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
|
||||
|
||||
```python
|
||||
import sys; sys.path.insert(0, "/opt/hermes")
|
||||
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
|
||||
from agent.auxiliary_client import call_llm
|
||||
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
|
||||
temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
|
||||
```
|
||||
|
||||
What actually breaks, and the fix:
|
||||
|
||||
- **`response_format` (json_object / json_schema) is a hard 400 on every
|
||||
local backend here** — neither FreeToken nor halogen implements constrained
|
||||
decoding. Config cannot override it: `call_llm` merges the task's
|
||||
`extra_body` first and then `update()`s the caller's, so the hardcoded one
|
||||
wins. Only temperature and max_tokens have reactive retry paths.
|
||||
- **Fix: move that one task to the Responses transport**, which never
|
||||
forwards `response_format`:
|
||||
|
||||
```yaml
|
||||
auxiliary:
|
||||
title_generation:
|
||||
provider: dankserver-small # names track the live config; diff first
|
||||
model: frank-small
|
||||
api_mode: codex_responses # real per-task key
|
||||
extra_body:
|
||||
reasoning:
|
||||
effort: none # mandatory, see below
|
||||
```
|
||||
|
||||
- **Disable thinking for any cap-limited task.** The adapter omits reasoning
|
||||
only when config says `reasoning.enabled: false`. The
|
||||
`reasoning_effort: none` *shorthand* produces exactly that and leaves the
|
||||
server default (thinking ON) — the 64-token title budget gets spent on the
|
||||
think: empty content, `finish_reason: length`. The explicit
|
||||
`extra_body.reasoning.effort: none` instead sends
|
||||
`{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
|
||||
honors.
|
||||
- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
|
||||
proxy needed for the alternate transport.
|
||||
|
||||
Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
|
||||
it must pass from config alone (that is the production path); keep the live
|
||||
file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
|
||||
config change is docs+example only, no service deploy needed.
|
||||
|
||||
Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
|
||||
`apply_instant_title`, so disabling a broken title task also removes the
|
||||
derived title — it does not degrade gracefully the way a failure does. A
|
||||
local call can also fail at the model level: a 200 with an error-text body,
|
||||
or a chatty non-title answer. Always read the raw content, not the status.
|
||||
|
||||
## Webui password rotation ("Hermex")
|
||||
|
||||
Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
|
||||
`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
|
||||
on match, 401 on miss. No username field. `/health` answers 200
|
||||
unauthenticated and `/` answers 302 — **never verify auth via those**; probe
|
||||
login with a deliberately wrong password (expect 401) and the real one
|
||||
(expect 200). Details on the webui surface: [[Hermes-WebUI]].
|
||||
|
||||
1. Run the whole probe as a heredoc'd bash script
|
||||
(`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
|
||||
inline `${…}` even inside single-quoted ssh args; never interpolate curl
|
||||
`-w %{http_code}` inline.
|
||||
2. Update BOTH env files with
|
||||
`sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
|
||||
- `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
|
||||
- `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
|
||||
quadlet's `[Container] EnvironmentFile=` is what the container reads at
|
||||
create; rotating only the root file leaves the running container serving
|
||||
the old secret.
|
||||
3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
|
||||
env; no deploy needed if the unit file itself didn't change).
|
||||
4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
|
||||
client at `http://192.168.0.143:8787` (webui publishes on the LAN
|
||||
interface since commit `4443897`; off-network ingress is SSH tunnel only —
|
||||
user `tunnel`, port **22**, key-only auth, forward remote end
|
||||
localhost:8787).
|
||||
|
||||
Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
|
||||
see rootless containers; run `podman exec hermes-webui printenv
|
||||
HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
|
||||
600). If rotating because a secret leaked into transcripts/logs, rotate in
|
||||
place via a script that fetches the live value from the running container
|
||||
(`printenv`) instead of transcribing it.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes-Dashboard]] — the LAN browser management UI
|
||||
|
||||
@@ -64,6 +64,49 @@ The Ollama connection is disabled (`ollama.enable=false`).
|
||||
> repository. Model swaps, context sizing, and spec tuning happen there. See
|
||||
> the [LLM knowledge base](../LLM/index.md).
|
||||
|
||||
## Container introspection (DB and backend python)
|
||||
|
||||
- Working dir for imports: `/app/backend`; package name `open_webui`
|
||||
(underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
|
||||
BEFORE importing any `open_webui` module (import-time hard requirement);
|
||||
the secret lives at `/app/backend/.webui_secret_key`.
|
||||
- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
|
||||
module (`podman exec open-webui python3 -c "import sqlite3;
|
||||
conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config
|
||||
keys use parameterized LIKE queries (avoids shell quoting).
|
||||
- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` —
|
||||
`Models.get_all_models()` is a coroutine (await it); model objects are
|
||||
pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
|
||||
`status`/`enabled` attribute — use `is_active` from the DB.
|
||||
- **API auth for verification:** `create_token` lives in
|
||||
`open_webui.utils.auth` (NOT `open_webui.utils.utils`);
|
||||
`get_verified_user` requires an `id` claim (not just `sub`);
|
||||
`SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
|
||||
container:
|
||||
|
||||
```
|
||||
podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \
|
||||
python -c "from open_webui.utils.auth import create_token; \
|
||||
print(create_token({\"id\":\"<UID>\",\"sub\":\"<UID>\",\"email\":\"…\",\"role\":\"admin\"}))"'
|
||||
```
|
||||
|
||||
(admin UID from the `user` table).
|
||||
|
||||
## Verification battery
|
||||
|
||||
1. `GET /api/v1/models/list?limit=100` → returns EXACTLY the intended
|
||||
models. Response key is `items`, not `data`.
|
||||
2. `POST /openai/chat/completions` per model with a trivial prompt ("Reply
|
||||
with exactly one word: pong") → each returns content `pong`. Reasoning
|
||||
models need `max_tokens>=100` — the thinking block eats the budget.
|
||||
3. A 500 "Server Connection Error" → `podman logs open-webui` for
|
||||
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
|
||||
`host.containers.internal`), not a model problem.
|
||||
|
||||
**Durability:** the model table + connections are DB-only (not git-managed);
|
||||
a fresh data volume loses them. For reproducibility, add an idempotent
|
||||
first-start seed script or document the setup in the service doc.
|
||||
|
||||
## Web search
|
||||
|
||||
Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
|
||||
@@ -147,6 +190,23 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
|
||||
|
||||
`podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
|
||||
|
||||
## Image generation (ComfyUI)
|
||||
|
||||
Image generation rides the local [[ComfyUI]] integration, not the chat
|
||||
picker — ComfyUI's models never appear in the chat model picker (that is
|
||||
chat LLMs only), so ComfyUI needs its own image-generation connection
|
||||
configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
|
||||
serves `:8188`.
|
||||
|
||||
Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
|
||||
route to `POST /api/v1/images/generations`; the `comfyui` engine is still
|
||||
upstream) with `Authorization: Bearer <token>` and body
|
||||
`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
|
||||
carries base64 image data or a URL.
|
||||
|
||||
Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
|
||||
on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
|
||||
|
||||
## Retired instances
|
||||
|
||||
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
|
||||
|
||||
@@ -117,6 +117,18 @@ frank-large `health=200` throughout.
|
||||
the model is loaded. Trust the engine's `host memory left for everything
|
||||
else` line, not `free`.
|
||||
|
||||
- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
|
||||
directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
|
||||
snapshots in the swapfile path — check `df -T` before planning, or
|
||||
`mkswap` will reject the file.
|
||||
- **Renaming an *active* swapfile works (inode-based) but leaves a stale
|
||||
path in `/proc/swaps`** — swap it off first, then rename, then `swapon`
|
||||
(which is exactly what the step-3 consolidation does). Migration is
|
||||
observable while it runs: in the 2026-09-12 run `swapoff` of 4 GiB took
|
||||
65 s and used-RAM rose only ~3 GiB (pages touched during migration), the
|
||||
rest relocating on disk. If used-RAM climbs toward the whole swapped
|
||||
amount, let it finish rather than aborting.
|
||||
|
||||
## Trash
|
||||
|
||||
The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
---
|
||||
type: reference
|
||||
title: Noctalia and Niri
|
||||
description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
|
||||
tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Noctalia and Niri
|
||||
|
||||
Noctalia (`/usr/bin/noctalia`) is the desktop shell on niri, running on both
|
||||
the workstation and blankpad. Since the 2026-09-10 unification both run the
|
||||
**bare `noctalia` v5 binary** (niri `spawn-at-startup "noctalia"`); the old
|
||||
Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is
|
||||
retired on both hosts.
|
||||
|
||||
## Config layers
|
||||
|
||||
- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g.
|
||||
`config.toml`). Main config: `~/.config/noctalia/config.toml`; the niri
|
||||
config includes `noctalia.kdl` for display settings.
|
||||
- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and
|
||||
**overrides declarative values for the same keys — never hand-edit it**;
|
||||
UI/IPC writes land there.
|
||||
- Main reference: `noctalia-reference` facts — IPC `noctalia msg <command>`
|
||||
replaced the `qs -c noctalia-shell ipc call` wrapper; app templates live at
|
||||
`/usr/share/noctalia/assets/templates/` (gtk, ghostty, alacritty, hyprland,
|
||||
sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango,
|
||||
umbriel, foot, emacs, kitty); palette generation `noctalia theme [image]`.
|
||||
|
||||
## Remote IPC over ssh (the D-Bus trap)
|
||||
|
||||
`noctalia msg <command>` resolves the running instance via the D-Bus session
|
||||
bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain
|
||||
ssh exec prints `error: noctalia is not running` even when the shell IS
|
||||
running — do not conclude "shell dead" from this error. Working remote
|
||||
procedure (runs in the systemd user-manager context where the bus address
|
||||
exists):
|
||||
|
||||
```
|
||||
systemd-run --user --pipe --wait noctalia msg <command> [args]
|
||||
```
|
||||
|
||||
Verified: `msg status` returns live JSON state; unknown panel ids return an
|
||||
error listing all valid panel ids (clipboard, control-center, launcher,
|
||||
polkit, session, setup-wizard, test, tray-drawer, wallpaper).
|
||||
|
||||
Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
|
||||
`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
|
||||
`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
|
||||
`media next|previous|playPause` (graceful no-op, rc 0, with no active
|
||||
player), `brightness-up [target] [step]`, `brightness-down`,
|
||||
`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
|
||||
`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
|
||||
`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
|
||||
`screenshot-fullscreen`.
|
||||
|
||||
**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
|
||||
resolves to XDG Pictures via user-dirs.dirs; an explicit directory key
|
||||
overrides. Fleet default pattern: point `directory` at the repo-tracked
|
||||
wallpapers dir so git-tracked wallpapers serve the shell without copying
|
||||
files. After writing config.toml over ssh the file watcher hot-reloads
|
||||
automatically; `config-reload` via systemd-run forces it immediately. Note:
|
||||
the *current* wallpaper stays the built-in default until the next rotation
|
||||
or an explicit `wallpaper-next`; `wallpaper-get` then reporting a path
|
||||
inside the configured directory is the confirmation the config took effect.
|
||||
|
||||
**Lock chain (unified):** swayidle-lock.service ExecStart on both hosts:
|
||||
`/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock'
|
||||
before-sleep '…' after-resume '…'`. `msg` resolves via the session bus, so
|
||||
no WAYLAND_DISPLAY pinning is needed in unit context.
|
||||
|
||||
## Custom-palette recolor (without restarting the session)
|
||||
|
||||
Why custom, never the community cache:
|
||||
`~/.local/state/noctalia/community-palettes/*.json` is a **sync cache** —
|
||||
`.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches
|
||||
from `https://api.noctalia.dev/palettes`, so edits there get reverted.
|
||||
Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
|
||||
(`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never
|
||||
synced. Missing/invalid file → silent fallback to the builtin palette.
|
||||
|
||||
Palette JSON shape = the community file's shape: `dark`/`light` maps of
|
||||
`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
|
||||
`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
|
||||
(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
|
||||
`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
|
||||
-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
|
||||
*generator* format, not this one. Copy an existing community JSON and mutate
|
||||
with `jq`.
|
||||
|
||||
Procedure:
|
||||
|
||||
1. Read the live scheme first — never assume: `noctalia msg color-scheme-get`
|
||||
(e.g. `community Gruber Darker`); `noctalia msg theme-mode-get`.
|
||||
2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml
|
||||
/tmp/noctalia-settings-backup-$(date +%s).toml`.
|
||||
3. Build the palette from the current one (jq keeps structure):
|
||||
|
||||
```sh
|
||||
mkdir -p ~/.config/noctalia/palettes
|
||||
jq '.dark.mPrimary = "#f43841"
|
||||
| .dark.terminal.normal.yellow = "#f43841"
|
||||
| .dark.terminal.bright.yellow = "#ff4f58"
|
||||
| .dark.terminal.foreground = "#f43841"' \
|
||||
"$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \
|
||||
> ~/.config/noctalia/palettes/"Gruber Red.json"
|
||||
```
|
||||
|
||||
Prefer a hex already present in the palette (its own `mError` red) so the
|
||||
theme stays internally consistent. Set BOTH `normal` and `bright`
|
||||
variants, and both `dark` and `light` if the user toggles modes.
|
||||
4. Switch (persists `theme.source = custom` + `theme.custom_palette`):
|
||||
`noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
|
||||
`color-scheme-get`.
|
||||
5. Re-render every configured template: `noctalia msg templates-apply`.
|
||||
Regenerates `~/.config/ghostty/themes/noctalia`,
|
||||
`~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
|
||||
starship, etc. Symlinked-into-dotfiles paths show up as modified files
|
||||
worth committing.
|
||||
6. Reload surfaces: compositor — `niri msg action load-config-file`
|
||||
(`reload-config` is NOT a valid niri subcommand); ghostty — see the trap
|
||||
below.
|
||||
|
||||
Verification: `ghostty +show-config | grep -E '^(foreground|palette)'`
|
||||
(proves the theme file resolves — note it prints only the delta from
|
||||
defaults; cross-check with `ghostty +show-config --default` before
|
||||
concluding a key is unset); `noctalia config validate` → `✓ Config is
|
||||
valid`; visual via `noctalia msg screenshot-fullscreen` (saves to
|
||||
`~/Pictures/screenshot_<ts>.png` and ALSO overwrites the clipboard with the
|
||||
PNG — warn the user); `grep -c <old-hex> ~/.config/noctalia/palettes/*.json`
|
||||
→ 0 for no leftovers.
|
||||
|
||||
Track it in the dotfiles repo (`~/.omp`, the command-center —
|
||||
`scripts/deploy.sh` is a stow-equivalent over `dotfiles/<pkg>/` trees):
|
||||
mirror the shape for the new palette so `git pull` + `deploy.sh` is the
|
||||
whole propagation story — move the JSON into
|
||||
`~/.omp/dotfiles/noctalia/.config/noctalia/palettes/` and symlink it back
|
||||
into `~/.config/noctalia/palettes/`; re-set the custom palette (re-reads
|
||||
through the symlink). Do the link by hand for one file instead of running
|
||||
`deploy.sh` — sync mode quarantines and replaces every divergent live file.
|
||||
|
||||
Traps:
|
||||
|
||||
- **NEVER `kill -USR1` ghostty to reload.** It does not reload config on
|
||||
this fleet — it kills/respawns the terminal (took out Frank's window
|
||||
2026-09-13). Correct: the default keybind `ctrl+shift+,` = `reload_config`
|
||||
(per `ghostty +show-config --default`). Existing windows keep old colors
|
||||
until reloaded; new windows inherit immediately.
|
||||
- Palette hex values are NOT a proxy for what the user sees: in
|
||||
`Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change
|
||||
the yellow" means the accent too, not just terminal index 3.
|
||||
- Saturating terminal text makes ANSI 1/9 indistinguishable from default
|
||||
text (grep highlights flatten) — flag it and offer a lighter red.
|
||||
- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats
|
||||
white-on-color here, ~3.9:1 — keep `mOnPrimary` dark).
|
||||
- Revert = `noctalia msg color-scheme-set community "<original name>"`; the
|
||||
community cache is untouched by the whole flow.
|
||||
- Generated theme files symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0`
|
||||
`noctalia.css`, ghostty theme) show up as tracked modifications after
|
||||
`templates-apply`; commit them together with the new palette JSON so every
|
||||
machine gets the same recolor.
|
||||
|
||||
## Offline template render (when templates-apply never ran)
|
||||
|
||||
Failure signature: niri `failed to read included config from
|
||||
~/.config/niri/noctalia.kdl: No such file or directory` — the desktop runs
|
||||
but with DEFAULT config (niri 26.04 falls back, does not exit): no custom
|
||||
keybinds/rules, no noctalia autostart, default output placement. ghostty:
|
||||
`theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`.
|
||||
alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing.
|
||||
|
||||
Root cause pattern: upstream dotfiles commits theme switches (niri include,
|
||||
ghostty `theme = noctalia`, alacritty import) that depend on noctalia
|
||||
TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from
|
||||
the local palette) and are intentionally NOT in the repo; on any host where
|
||||
`templates-apply` never ran, they don't exist.
|
||||
|
||||
Fix (offline, works while a session runs) — built-in templates live at
|
||||
`/usr/share/noctalia/assets/templates/` (declared in `builtin.toml`); render
|
||||
each needed one with the live palette:
|
||||
|
||||
```bash
|
||||
# niri include (plain host-local file; do NOT symlink into repo — regenerated per host)
|
||||
noctalia theme --theme-json ~/.config/noctalia/colors.json \
|
||||
--render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl"
|
||||
mkdir -p ~/.config/ghostty/themes && \
|
||||
noctalia theme --theme-json ~/.config/noctalia/colors.json \
|
||||
--render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia"
|
||||
mkdir -p ~/.config/alacritty/themes && \
|
||||
noctalia theme --theme-json ~/.config/noctalia/colors.json \
|
||||
--render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml"
|
||||
```
|
||||
|
||||
All template tokens (primary, surface, error, primary_container, …) resolve
|
||||
from the flat 16-key colors.json; container tokens are derived, never blank.
|
||||
|
||||
Verify: `niri validate` → rc 0; `ghostty +validate-config` → rc 0; next
|
||||
login journal shows `loaded config from config.kdl` with NO parse WARN,
|
||||
custom output placement (`niri msg outputs` vs display.kdl), noctalia
|
||||
autostarted (`pgrep noctalia`).
|
||||
|
||||
Traps: **`noctalia msg templates-apply` on 5.0.1 returns "ok" but applies
|
||||
NOTHING** (this build's opt-in config schema does not act on it) — do not
|
||||
trust its exit code, use the explicit `--render` calls above. Niri config
|
||||
errors at boot are NOT fatal — the compositor keeps running on defaults,
|
||||
which reads as "some settings silently stopped working", not a crash.
|
||||
Regenerate after any palette change; the rendered files are static
|
||||
snapshots.
|
||||
|
||||
## Niri output rotation and pinning
|
||||
|
||||
Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
|
||||
|
||||
**Ground truth first:** `niri msg outputs` (connector names, current mode,
|
||||
scale, transform, logical pos/size) and `niri --version`. Output **names are
|
||||
connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
|
||||
the EDID strings printed before them are decoration. On blankpad the live
|
||||
config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
|
||||
and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
|
||||
repo path (that is what gets committed); check with `ls -ld`.
|
||||
|
||||
**Transform direction is counter-clockwise** (`niri msg output <NAME>
|
||||
transform --help`): `90` = content rotated 90° CCW ⇒ the panel's **top
|
||||
bezel sits on the LEFT**; `270` = top bezel RIGHT; `180` = upside down;
|
||||
`flipped-90`/`flipped-270` = mirrored variants. `90` vs `270` is a
|
||||
physical-stand fact no tool can read (portable monitors expose no
|
||||
orientation in EDID/MCCS) — pick one, apply, and hand the user the one-liner
|
||||
to flip; don't stall on the question.
|
||||
|
||||
Two traps when editing the config:
|
||||
|
||||
1. **An `output "X" { … }` block resets every property you don't restate to
|
||||
its default.** Pinning position on `eDP-1` without repeating `scale 1.25`
|
||||
silently drops the laptop panel to scale 1.0. Restate `scale` (and
|
||||
`mode`) in every block you touch.
|
||||
2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning
|
||||
only the externals shoved `eDP-1` from `0,0` to `3936,0`. Pin every
|
||||
output's position to keep the left-to-right order. Positions are logical
|
||||
pixels at each output's own scale; x-offsets chain (eDP-1 logical width
|
||||
1536 → DP-2 at 1536 → +1200 portrait width → HDMI-A-1 at 2736):
|
||||
|
||||
```kdl
|
||||
output "DP-2" { scale 1 transform "90" position x=1536 y=0 }
|
||||
output "HDMI-A-1" { scale 1 transform "90" position x=2736 y=0 }
|
||||
output "eDP-1" { scale 1.25 position x=0 y=0 } // MUST restate scale
|
||||
```
|
||||
|
||||
Apply and verify:
|
||||
|
||||
```bash
|
||||
niri validate -c ~/.config/niri/config.kdl # exit 0
|
||||
niri msg action load-config-file # live reload; NOT reload-config
|
||||
sleep 1; niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
|
||||
```
|
||||
|
||||
Success looks like `Logical size: 1200x1920` (swapped) plus
|
||||
`Transform: 90° counter-clockwise` on each rotated panel, positions matching
|
||||
the intended chain. One-off rotation without touching config (temporary):
|
||||
`niri msg output DP-2 transform 90`. Commit via the dotfiles repo
|
||||
(`~/.omp`, `main`; push to the local Gitea permitted): `git pull --ff-only`
|
||||
→ add the display.kdl → commit → push. Pitfalls: `reload-config` is not a
|
||||
niri subcommand; never SIGUSR1 a terminal to "reload" (niri config reloads
|
||||
do not touch ghostty themes); no man page for niri on this host — `niri
|
||||
<sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
|
||||
|
||||
## GPU RGB over I2C (EVGA iCX3)
|
||||
|
||||
Turn RGB on/off (or read it back) on an EVGA FTW3 / iCX3-controller GPU
|
||||
(the 3090 on dankclaw) using the GPU's own nvidia i2c bus. No display
|
||||
engine, no X, no daemon — the mode is EEPROM-saved and survives reboot.
|
||||
|
||||
- **Find the controller:** the nvidia i2c driver exposes the card's lighting
|
||||
ports as `/dev/i2c-N`; on the 3090 FTW3 that is ports 1,5,6,7,8 →
|
||||
i2c-1..i2c-5, and only i2c-1 carries a live slave (NVIDIA port 1). Probe
|
||||
each bus. The iCX3 MCU answers at address **0x2D**. `/dev/i2c-*` are
|
||||
`root:i2c 660` — run with `sudo` (dank has passwordless sudo).
|
||||
- **Protocol (raw I2C_RDWR, NOT the SMBus ioctls)** — write the register
|
||||
byte first, then the payload:
|
||||
- enable `0xB2` write `04 C6 EB EA 15` (required once before mode writes)
|
||||
- mode `0xC0` write `09 m0 m1 m2 m3 .. .. .. .. SYNC`; m0..m3 are the four
|
||||
zone modes, `MODE_OFF = 0x00`; OFF = `09 00 00 00 00 FF FF FF FF 00`
|
||||
- save `0x90` write `04 9E EB 00 90` (commit to EEPROM — the
|
||||
reboot-persistence step)
|
||||
- fw `0xB1` block read (firmware version)
|
||||
- Read the mode back: write reg `0xC0`, then read 16 bytes; bytes 1..4 are
|
||||
the zone modes (0x00 = OFF), bytes 6..9 the per-zone LED counts.
|
||||
- **Verify persistence:** after a reboot, read the mode — all four zone
|
||||
modes `0x00` means the EEPROM save held (confirmed 2026-09-19 post-reboot).
|
||||
- **Tool:** a ~90-line I2C_RDWR reader/writer lived at `/tmp/evga_raw.c`
|
||||
(`evga_raw <bus> mode|off|save|enable|fw`) — `/tmp` is wiped by reboot, so
|
||||
recreate it if needed; OpenRGB's `EVGAGPUv3Controller.cpp` carries the
|
||||
same protocol plus the full colour API.
|
||||
- Gotchas: raw I2C_RDWR only (the iCX3 answers the raw form, not the SMBus
|
||||
ioctls); the `enable` write must precede the first mode write in a
|
||||
session; without the `save` write the mode is RAM-only and resets on power
|
||||
cycle.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Command Center Dotfiles]] — the stow-equivalent deploy this page links into
|
||||
- [[Hermes]] — the agent that drove the CDP daemon alongside this desktop
|
||||
- [[GGUF Compatibility]] — unrelated but shares the "read the file, not the name" lesson
|
||||
@@ -5,3 +5,115 @@ description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Reada
|
||||
tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
|
||||
timestamp: 2026-09-04T00:00:00Z
|
||||
---
|
||||
|
||||
# *arr Stack
|
||||
|
||||
The *arr automation stack runs on **FrankBook-Pro** (the Mac mini) as quadlets
|
||||
inside its podman machine; all egress rides the gluetun WireGuard netns. The
|
||||
per-service pages ([[Prowlarr]], [[Sonarr]], [[Radarr]], [[Lidarr]],
|
||||
[[Readarr]], [[qBittorrent]]) carry deployment facts; this page holds the
|
||||
cross-cutting API and auth-posture procedures.
|
||||
|
||||
## API version split (memorize before any *arr HTTP call)
|
||||
|
||||
- Sonarr/Radarr: `/api/v3/…`; Lidarr/Readarr/Prowlarr: `/api/v1/…`.
|
||||
- Prowlarr is port **9696** — 8090 is Jackett.
|
||||
- **Empty body = endpoint absent**, not a failure: probe endpoints by status
|
||||
code first; v1 has no `/api/v1/profile` endpoint at all.
|
||||
- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object
|
||||
(GET it, patch `settings`, PUT the whole object back). An array body fails
|
||||
with `Path: $ | BytePositionInLine: 1`.
|
||||
- API keys live in each container's config:
|
||||
`podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
|
||||
They are local-only secrets — they stop being secret the moment a vhost
|
||||
serves the API unauthenticated.
|
||||
|
||||
## Schema-first probing (validators are free documentation)
|
||||
|
||||
Send the FULL schema template object with fields filled; v1 validators reject
|
||||
partials, and each error names the next required field — peel validators one
|
||||
live call at a time instead of guessing payload shapes:
|
||||
|
||||
- `'Fields' must not be empty` → send a full schema copy, not `fields:[]`.
|
||||
- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
|
||||
required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
|
||||
- `DefaultMetadataProfileId must be > 0` → fetch real ids from
|
||||
`GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
|
||||
`GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
|
||||
profiles.
|
||||
- `targetType null` → configContract not registered in that build (this
|
||||
Prowlarr build has no Prowlarr-plugin indexer; integration is
|
||||
Torznab-family only).
|
||||
- `testingPackageId` in a PUT body downgrades validation failures to
|
||||
warnings.
|
||||
- `syncLevel` is enum `ApplicationSyncLevel` — omit it or use the int; a
|
||||
string is rejected.
|
||||
|
||||
**Verification rules:** `POST /downloadclient` returns the COLLECTION, not
|
||||
the created object — verify via GET. Test endpoints return verdicts in the
|
||||
BODY at HTTP 200 (`{}` = pass, a list = fail) — judge from bodies, not
|
||||
status codes. Command POSTs (`MoviesSearch`, `ApplicationSync`) run
|
||||
server-side and respond only when the work finishes: curl `-m` timeouts
|
||||
(HTTP 000) during a search are NOT failures — poll queue/history afterward
|
||||
instead of re-firing; warm small commands (`RefreshMonitoredDownloads`)
|
||||
answer in ~20 ms, so a hang means live work in progress, not a dead API.
|
||||
|
||||
**DB as arbiter:** when endpoint shapes are opaque, stop the container and
|
||||
edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly,
|
||||
then start. The indexers table columns are
|
||||
Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the
|
||||
enable flag column is `Enable` (not `EnableRss`). Prowlarr's real application
|
||||
settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
|
||||
`/api/v1/applications` GET responses carry `fields: []`).
|
||||
|
||||
## Auth posture probe (ground truth is behavior, not config flags)
|
||||
|
||||
`AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in
|
||||
`config.xml` is ambiguous on paper; on the wire it means a login screen with
|
||||
**zero users** — first-account-wins.
|
||||
|
||||
1. **Confirm which host runs the service before probing** (`podman ps` on
|
||||
the target, not the box you ssh'd into first — the arrs live on
|
||||
FrankBook-Pro).
|
||||
2. Probe an API endpoint that bypasses the SPA shell, locally and through
|
||||
any vhost: `curl -s -o /dev/null -w "%{http_code}"
|
||||
http://<host>:<port>/api/v3/system/status`. **SPA roots (`/`) always
|
||||
serve a 200 shell — never conclude "open" or "gated" from `/`.**
|
||||
401 = gated; 200 + JSON = API fully open (anyone on the net can drive it
|
||||
with curl).
|
||||
3. For config truth, dump the whole file
|
||||
(`podman exec <svc> cat /config/config.xml`), not a piped
|
||||
`grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the
|
||||
pipeline hides a miss and you conclude "empty field" when the grep itself
|
||||
failed.
|
||||
4. Decision framing: when posture = gated-with-zero-users, present it as a
|
||||
decision menu (None behind the front-door wall vs External/authentikarr),
|
||||
not a silent change — first-account-wins means whoever logs in first owns
|
||||
the instance.
|
||||
|
||||
## The `None` + `Enabled` auth-config trap (v1 .NET apps)
|
||||
|
||||
`<AuthenticationMethod>None</AuthenticationMethod>` +
|
||||
`<AuthenticationRequired>Enabled</AuthenticationRequired>` is
|
||||
self-contradictory: the UI Save is permanently disabled, and any save throws
|
||||
`DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every
|
||||
page after restart. The valid no-auth representation is **absence of the
|
||||
`<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix =
|
||||
delete the key from `/config/config.xml`
|
||||
(`sed -i "/<AuthenticationRequired>/d"`), restart the container. Tell-tale:
|
||||
one v1 app boots clean while siblings 500-sprawl → diff their config.xml
|
||||
auth blocks; the healthy one shows the correct shape. (Prowlarr runs
|
||||
`Forms` + `Enabled` healthily — the crash combo is `None`+`Enabled` only.)
|
||||
|
||||
General rules that saved the session these came from: container
|
||||
`State=running` ≠ app serving — read `podman logs --tail 12 <c>` before
|
||||
theorizing; when a UI gate blocks saving, go under it (these apps keep
|
||||
plaintext XML config on disk — read/edit via `podman exec`).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Prowlarr]] — indexer feeds and the v2 sync model
|
||||
- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] — the managers
|
||||
- [[qBittorrent]] — download client, 4.6.7 pin, E2E verification
|
||||
- [[FrankBook-Pro]] — the host and its podman machine
|
||||
- [[Homepage Status Probes]] — how the e2e gate treats this sleeping host
|
||||
@@ -111,6 +111,90 @@ Restart=always
|
||||
probe one ssh-context out returns the residential IP — a false leak alarm.
|
||||
- Ensure umask is set to `022` so downloaded files are readable by arr apps.
|
||||
|
||||
## E2E verification after any pipeline change
|
||||
|
||||
Run after ANY change to the torrent pipeline — WireGuard credential rotation,
|
||||
qB version change, gluetun reconfig, category re-seeding. Proves the full
|
||||
grab path (tunnel egress → qB download → category save path → clean
|
||||
removal). All commands from a dev box via `ssh frankbook-pro`; inside the
|
||||
Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
|
||||
`podman machine ssh`.
|
||||
|
||||
**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
|
||||
(NOT core — `core` is the podman-machine VM user, reached only via
|
||||
`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
|
||||
2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
|
||||
low MaxAuthTries: when password-bootstrapping use `-o
|
||||
PreferredAuthentications=password -o PubkeyAuthentication=no
|
||||
-o IdentitiesOnly=yes` or the key offers exhaust the attempts before the
|
||||
password is tried.
|
||||
|
||||
**1. Egress gate — MUST run inside gluetun:**
|
||||
|
||||
```bash
|
||||
podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
|
||||
```
|
||||
|
||||
Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config); judge by org,
|
||||
never country alone. NEVER run this one ssh-context out (bare `wget` on the
|
||||
Mac host): it silently returns the residential Comcast IP and reads as a
|
||||
hard-gate leak that does not exist. A residential reading ⇒ re-run inside
|
||||
the container before diagnosing.
|
||||
|
||||
**2. Login — generic cookie parse.** The QPW fetch is flaky (~1 in N returns
|
||||
empty); assert length before use — an empty-password run once produced a
|
||||
Forbidden loop that cost hours:
|
||||
|
||||
```bash
|
||||
QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
|
||||
[ ${#QPW} -eq 20 ] || exit 1
|
||||
SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
|
||||
http://127.0.0.1:8080/api/v2/auth/login \
|
||||
| grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
|
||||
curl -b "$SID" … # pass the WHOLE match
|
||||
```
|
||||
|
||||
WebUI from the LAN: `http://192.168.0.112:8080` (published on the gluetun
|
||||
unit). Username `admin`; password = the machine-local `QBIT_WEBUI_PASSWORD`
|
||||
(20 chars, generated once at provisioning by the deploy handler — never in
|
||||
the repo). Parse-bug 403s never reach qB as failed logins — no ban accrues;
|
||||
server-side truth is `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log`
|
||||
(`WebAPI login success/failure` + source IP). Repeated REAL failures accrue
|
||||
toward qB's 5-failures/3600 s IP ban.
|
||||
|
||||
**3. Add a legal test torrent.** Blender open-movie torrents (~300 MB, e.g.
|
||||
Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file
|
||||
(`-F "file=@…"`) and `category=Finished Torrents`. Never leave a real-looking
|
||||
grab behind.
|
||||
|
||||
**4. Poll and verify.** Poll `GET /torrents/info` (~20 s): state
|
||||
`downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading`
|
||||
at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents`
|
||||
(qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT
|
||||
inherit category paths — one more reason for the 4.6.7 pin). Confirm on
|
||||
disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is
|
||||
normal for the tunnel.
|
||||
|
||||
**5. Cleanup — delete only your artifact:**
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
|
||||
--data-urlencode "hashes=<literal-hash>" \
|
||||
--data-urlencode "deleteFiles=true" \
|
||||
http://127.0.0.1:8080/api/v2/torrents/delete # expect 200
|
||||
```
|
||||
|
||||
Target the known hash literally — the user's real grabs share the directory
|
||||
and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir
|
||||
gone, all other entries intact.
|
||||
|
||||
**Version-coupling constraints:** qB is pinned to 4.6.7 (Readarr's EOL
|
||||
client rejects qB ≥ 5's `204` login at runtime — see the pin rationale
|
||||
above). Do not unpin while Readarr exists. After ANY qB version flip,
|
||||
regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k
|
||||
iterations) — a 5.x-written hash line makes 4.6.7 reject every login.
|
||||
Rewrite while qB is stopped.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ARR-Stack]] — full *arr stack documentation
|
||||
|
||||
@@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
|
||||
distinguish challenge from deny at this release; grep the body for
|
||||
`Access Denied` / `Making sure you` instead.
|
||||
|
||||
## Guard-chain layer diagnosis (a user reports a guarded page "broken")
|
||||
|
||||
Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
|
||||
"broken / links gone / not loading" and the fault must be localized to
|
||||
server-side or client-side/session. Chain topology:
|
||||
`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
|
||||
→ Caddy :8017 (@home/@data handlers) → app/static`.
|
||||
|
||||
Layer-by-layer checks (run from the host via ssh):
|
||||
|
||||
1. **Caddy unit + guard containers**: `systemctl --user is-active
|
||||
caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
|
||||
'anubis|authentik|caddy'` — expect `Up … (healthy)`.
|
||||
2. **Anubis challenge page validity**: `curl -sk
|
||||
https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
|
||||
— a healthy challenge contains `"anubis_challenge"` JSON with
|
||||
`difficulty` and the main JS.
|
||||
3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
|
||||
-iE 'error|exception|traceback|500|provider'` — routine dramatiq task
|
||||
logs are normal.
|
||||
4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
|
||||
its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
|
||||
connection refused, which is EXPECTED, not a fault):
|
||||
`podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
|
||||
www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
|
||||
`/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
|
||||
this container has NO `-H` flag — use `--header "Host: …"`.)
|
||||
5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
|
||||
parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
|
||||
|
||||
Tool/behavior notes: the anubis container is distroless — `podman exec anubis
|
||||
sh` fails (`sh: executable not found`); inspect via its logs or the served
|
||||
challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
|
||||
podman IP) for ALL clients — rootless-podman published-port SNAT;
|
||||
clients are indistinguishable by IP and clearance is cookie-based. Do NOT
|
||||
mistake that for the bug. Homepage render: `app.js` does
|
||||
`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
|
||||
failure the catch sets `#summary` to `status unavailable: <err>` and the
|
||||
grid stays EMPTY — which a user reads as "links gone / not loading." A
|
||||
30-service status.json with a 9-service services.json renders fine
|
||||
(stateByName lookup).
|
||||
|
||||
**Decision:** if every layer is healthy (challenge valid, outpost clean,
|
||||
handlers 200, files render), the fault is client-side/session. STOP
|
||||
thrashing the live system — repeated failed browser challenges share the
|
||||
masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
|
||||
symptom (challenge spinning? login loop? empty grid? "status unavailable"
|
||||
text?) or the authentik login to drive the page in a real browser.
|
||||
|
||||
## Verify live
|
||||
|
||||
```
|
||||
|
||||
@@ -342,6 +342,168 @@ The silent re-auth path was proven end-to-end: after deleting
|
||||
`qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
|
||||
state and passed with no login prompt (2 tests, 2.9 s).
|
||||
|
||||
## Guard-chain flip recipe (adding a guarded vhost)
|
||||
|
||||
Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
|
||||
client → TLS edge → Anubis :8923 → outpost decision → internal
|
||||
`http://:8017` listener → Host-matched handler → real upstream.
|
||||
|
||||
**Order (never reorder):**
|
||||
|
||||
1. **Provider first.** Create the outpost ProxyProvider before touching
|
||||
routes, or authed traffic hits the outpost error page instead of the app.
|
||||
Fleet convention for `external_host` is the exact browser-facing origin
|
||||
(the inventory table carries both bare origins and trailing slashes;
|
||||
routing matches on hostname, verified 2026-09-02).
|
||||
`internal_host = http://caddy:8017`, all three flows set, and an
|
||||
Application attached — the two provisioning rules above.
|
||||
2. **Route flip in ONE commit/deploy**: the vhost becomes
|
||||
`reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
|
||||
inside the `http://:8017` listener add `@<svc> host <vhost>` +
|
||||
`handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
|
||||
edge, `:8017` routing table, and provider list change together.
|
||||
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
|
||||
|
||||
**Verification (all three):**
|
||||
|
||||
- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
|
||||
challenge page; grep the body for `challenge|not a bot|anubis` (~8
|
||||
markers on a live challenge). A 502 here instead of the challenge = chain
|
||||
mis-wired.
|
||||
- Post-auth routing proof: grep the LIVE config inside the container —
|
||||
`podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
|
||||
after the deploy prints its reload line; confirm the reload actually ran,
|
||||
don't trust exit 0.
|
||||
- Outpost reachability: `podman exec authentik-server getent hosts caddy`
|
||||
resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
|
||||
|
||||
**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
|
||||
000 (connection refused) — the post-auth listener lives in the caddy
|
||||
container's netns; probe inside it or grep the config. Non-browser clients
|
||||
cannot solve the JS challenge — before flipping, audit service-to-service
|
||||
traffic: ingestion/push paths must target internal addresses (`loki:3100`),
|
||||
never a public hostname of a guarded service. Exempt-by-construction
|
||||
categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
|
||||
endpoints, services with their own complete auth — document the exemption
|
||||
in the vhost comment.
|
||||
|
||||
## Driving the ORM (`ak shell`)
|
||||
|
||||
All authentik object work happens through the server container's ORM shell —
|
||||
no repo change, no deploy (these providers are NOT repo-managed; capture new
|
||||
provider+application pairs as a blueprint or setup script to avoid drift).
|
||||
|
||||
```bash
|
||||
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
|
||||
from authentik.core.models import User, Group
|
||||
...
|
||||
EOF
|
||||
```
|
||||
|
||||
- The heredoc IS the program: `ak shell` reads all of stdin and execs it
|
||||
(heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
|
||||
one).
|
||||
- Bootstrap prints JSON log lines, then
|
||||
`### authentik shell (2026.8.3)` and `229 objects imported automatically…`
|
||||
— pipe through `sed -n '/229 objects/,$p'` to drop the noise.
|
||||
- 229 models are pre-imported. Import paths (each wrong guess costs a failed
|
||||
exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
|
||||
`PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
|
||||
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
|
||||
`LDAPProvider` → `authentik.providers.ldap.models`;
|
||||
`authentik.core.models.application` does NOT exist as an import path; there
|
||||
is no `authentik.policies.group` module in this build.
|
||||
- For live DB truth prefer the postgres container
|
||||
(`podman exec authentik-postgres psql -U authentik -d authentik -At -c
|
||||
"select * from …"`) — **query with `select *` and stop guessing column
|
||||
names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
|
||||
FieldError — `type` moved to subclass tables; the outpost `enabled` column
|
||||
may not exist under that name; provider pk is `oauth2provider_ptr_id`).
|
||||
|
||||
## User provisioning playbook (user-level account)
|
||||
|
||||
There is no `User.create_user` classmethod, and `is_superuser` is NOT a
|
||||
constructor kwarg (it is a property derived from group membership) — passing
|
||||
it raises `TypeError: User() got unexpected keyword arguments`:
|
||||
|
||||
```python
|
||||
from authentik.core.models import User, Group
|
||||
u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
|
||||
u.set_password("...") # password arrives on stdin only, never argv
|
||||
u.save()
|
||||
for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
|
||||
u.groups.add(Group.objects.get(name=g))
|
||||
```
|
||||
|
||||
`path` defaults to `users` (regular). The access map (verified live
|
||||
2026-10-04): there are NO per-app policy bindings on the proxy providers
|
||||
(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
|
||||
access is decided entirely by GROUP MEMBERSHIP consumed downstream —
|
||||
|
||||
- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
|
||||
regex gate) and the homepage admin panel; regular users get only
|
||||
`category: user` panels from `services.json`.
|
||||
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
|
||||
"Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
|
||||
"Open WebUI roles" property mapping → `roles` claim. Verify live with
|
||||
`u.app_entitlements(app)`.
|
||||
- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
|
||||
(filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
|
||||
- `Non-Admin Users` = the plain-user marker group (precedents: alex,
|
||||
qa-e2e).
|
||||
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
|
||||
`openwebui_admins`, `authentik Admins`, `authentik Read-only`.
|
||||
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
|
||||
Plex, Seerr, Marreta) need no per-user group: any authenticated user
|
||||
passes the outpost.
|
||||
|
||||
Verify login behaviorally — `authenticate()` is the exact PasswordStage code
|
||||
path:
|
||||
|
||||
```python
|
||||
from django.contrib.auth import authenticate
|
||||
assert authenticate(request=None, username="ridley", password="...") # -> User
|
||||
assert authenticate(request=None, username="ridley", password="nope") is None
|
||||
```
|
||||
|
||||
Look flows up by slug, not name:
|
||||
`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
|
||||
"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
|
||||
Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
|
||||
enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
|
||||
|
||||
More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
|
||||
can be null (guard with `getattr`); `LDAPProvider` has
|
||||
`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
|
||||
`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
|
||||
`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
|
||||
(use `type(p).__name__`); `User` has no `.status`.
|
||||
|
||||
Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
|
||||
every outpost session fleet-wide; purge the stale `proxysession` rows
|
||||
instead (section above).
|
||||
|
||||
## Probing the outpost from inside the netns
|
||||
|
||||
Rootless podman gives each container its own netns; **host `ss -ltn` cannot
|
||||
see listeners inside other containers' namespaces** — a missing listener on
|
||||
the host is an observation artifact, not a fault. Distroless images have no
|
||||
shell (`podman exec authentik-server sh` fails), so probe by joining the
|
||||
namespace:
|
||||
|
||||
```
|
||||
podman run --rm --network container:<container> --entrypoint /bin/busybox \
|
||||
docker.io/library/busybox:latest \
|
||||
wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
|
||||
```
|
||||
|
||||
- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
|
||||
triggers globbing (`no matches found`).
|
||||
- Boot transience: immediately after a restart the same probe can return 503
|
||||
(configure task pending); poll until the answer changes before concluding.
|
||||
- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
|
||||
version — the field is `.HostConfig.NetworkMode`.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
---
|
||||
type: reference
|
||||
title: Homepage Status Probes
|
||||
description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
|
||||
tags: [homepage, status, probes, e2e, homepage-status]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Homepage Status Probes
|
||||
|
||||
The homepage is a static site —
|
||||
`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
|
||||
— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
|
||||
`scripts/deploy-remote.sh` refuses other targets). A checker
|
||||
(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
|
||||
each card's target and writes `/srv/homepage/status.json` (mode **0644** so
|
||||
Caddy can serve it — losing the mode 403s the page).
|
||||
|
||||
## The checker's vantage and its three false-down shapes
|
||||
|
||||
The checker probes TCP/HTTP **from the host netns**. It only has three states
|
||||
— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
|
||||
never render honestly and must be dropped or re-pointed, not "fixed".
|
||||
|
||||
1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
|
||||
only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
|
||||
card says down while PMS answers `200` over HTTPS. Fix:
|
||||
`"method":"tcp"` (connect is the honest liveness signal; the public URL
|
||||
stays HTTP because Caddy re-keys).
|
||||
2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
|
||||
compose→quadlet move moved the dashboard to `9119` and the webui to
|
||||
`8787`; nothing listened on 8090, so a healthy stack read `down`.
|
||||
3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
|
||||
`systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
|
||||
and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
|
||||
netns-scoped, so from the host these are unreachable *by construction* —
|
||||
no probe value makes them honest. Drop those cards.
|
||||
|
||||
A host-vantage probe can only reach a listener **published to the host**
|
||||
(rootlessport/pasta); unpublished ports are invisible from the host netns.
|
||||
See [[Container Host Reachability]] for the netns rules.
|
||||
|
||||
## Triage procedure
|
||||
|
||||
1. Read the card's probe target in `shared/homepage/services.json`; note
|
||||
method (`tcp` vs `http`).
|
||||
2. Does that port have a **host-visible listener**?
|
||||
`podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
|
||||
`podman inspect <c> --format '{{json .NetworkSettings.Networks}}
|
||||
{{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
|
||||
published nothing → structurally unprobeable from the host).
|
||||
3. Probe from the **correct vantage**: from a bridge-net container the host
|
||||
is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
|
||||
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
|
||||
which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
|
||||
bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
|
||||
they expand **inside** `podman exec bash -c`.
|
||||
4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
|
||||
SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
|
||||
killer) vs false down (running + unreachable target).
|
||||
5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
|
||||
only exposure is an internal netns; never invent a probe target.
|
||||
6. Verify by re-running the checker path, not by eyeballing: after deploy,
|
||||
re-read `/srv/homepage/status.json` and diff the `down` list
|
||||
before/after.
|
||||
|
||||
Removing a card's `probe` key without a code change crashes the checker
|
||||
(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
|
||||
fix. The `description` field is consumed only by `app.js` `descHTML()`, so
|
||||
adding a `details` object (role/spec/inputs/use) is schema-safe:
|
||||
`tests/homepage_config.bats` checks only name/category/section/probe.
|
||||
|
||||
## Local render check (before every deploy)
|
||||
|
||||
The page fetches `services.json` + `status.json` at load; `file://` is
|
||||
CORS-blocked, so serve over HTTP:
|
||||
|
||||
```bash
|
||||
cd shared/homepage
|
||||
python3 -c "import json; s=json.load(open('services.json')); \
|
||||
json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
|
||||
python3 -m http.server 8899 --bind 127.0.0.1
|
||||
```
|
||||
|
||||
Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
|
||||
(panels render after the async fetch; checking immediately returns 0 panels),
|
||||
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
|
||||
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
|
||||
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
|
||||
visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
|
||||
time) — harmless.
|
||||
|
||||
The scratch `status.json` is untracked and NOT git-ignored — move it to
|
||||
`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
|
||||
|
||||
CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
|
||||
`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
|
||||
`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
|
||||
reset so `?bare=1` strips it to black.
|
||||
|
||||
## Deploy
|
||||
|
||||
```
|
||||
./scripts/deploy.sh dankclaw --service homepage --skip-e2e
|
||||
```
|
||||
|
||||
Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
|
||||
datetime), restarts the `homepage-status` timer. Verify live:
|
||||
`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
|
||||
|
||||
**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
|
||||
asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
|
||||
Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
|
||||
no remote wake) times out that test — a pre-existing environmental failure,
|
||||
not a regression. All non-Mac-mini e2e tests pass.
|
||||
|
||||
## e2e raw-IP *arr failures — probe from the browser's box, not a peer
|
||||
|
||||
When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
|
||||
Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
|
||||
so a healthy probe from dankclaw proves nothing about the **browser's** path:
|
||||
|
||||
```bash
|
||||
# 1. From the box that RUNS the browser:
|
||||
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
|
||||
# 2. From a LAN peer:
|
||||
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
|
||||
--max-time 6 http://192.168.0.112:$p/; done'
|
||||
# 3. Liveness (ICMP, not TCP):
|
||||
ping -c 2 -W 3 192.168.0.112
|
||||
```
|
||||
|
||||
| peer probe | dev-box probe | ping | meaning |
|
||||
|---|---|---|---|
|
||||
| 200 | 200 | ok | transient — re-run the spec |
|
||||
| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
|
||||
| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
|
||||
|
||||
Third row: confirm the Mac is innocent through the dankclaw jump
|
||||
(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
|
||||
`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
|
||||
--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
|
||||
between the dev box and the Mac.
|
||||
|
||||
Don't hold a deploy hostage to a known-environmental red gate: verify the
|
||||
deployed service **directly** (its own ledger/health/real completion), state
|
||||
the gate was skipped or partially red and why (`--skip-e2e` prints the
|
||||
warning, so the skip is recorded), and report the residual gap plainly
|
||||
("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
|
||||
probe returns 200; deploy verified by other means"). Do not weaken or skip
|
||||
the spec — it guards a real SSO-plus-RBAC contract.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Container Host Reachability]] — the netns rules behind the vantage
|
||||
- [[Observability Provisioning]] — the monitoring stack reading the same fleet
|
||||
- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
|
||||
- [[Testing Policy]] — the seam/e2e contract this page's checks live in
|
||||
@@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the
|
||||
| HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
|
||||
| Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
|
||||
|
||||
## Per-client fix: resolver drop-in + browser DoH bypass
|
||||
|
||||
Symptom: a LAN machine gets `connection timeout` (curl rc=28) on
|
||||
`https://*.franksdank.cloud` while the server is healthy. Root cause: the
|
||||
public DNS hands out the WAN IP `71.227.227.172`, which has **no 443
|
||||
port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN
|
||||
answer → SYN to a filtered port → timeout. There are TWO resolver layers to
|
||||
check, in order:
|
||||
|
||||
**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in
|
||||
sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route
|
||||
servers:
|
||||
|
||||
```bash
|
||||
dig +short www.franksdank.cloud @192.168.0.143 # expect 192.168.0.143
|
||||
dig +short www.franksdank.cloud @1.1.1.1 # expect 71.227.227.172 (the trap)
|
||||
getent hosts www.franksdank.cloud # what the OS actually resolves
|
||||
curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \
|
||||
https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n' # 200 = server fine
|
||||
|
||||
sudo mkdir -p /etc/systemd/resolved.conf.d
|
||||
sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
|
||||
[Resolve]
|
||||
DNS=192.168.0.143
|
||||
Domains=~franksdank.cloud ~franksdank.casa
|
||||
EOF
|
||||
sudo systemctl restart systemd-resolved && resolvectl flush-caches
|
||||
```
|
||||
|
||||
The `~` (route-only) prefix is load-bearing: it routes only those domains to
|
||||
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
|
||||
`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
|
||||
|
||||
**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
|
||||
Layer 1 is verified but the BROWSER still times out, the browser is doing
|
||||
its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR
|
||||
provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it:
|
||||
`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead
|
||||
end). Diagnosis details:
|
||||
|
||||
- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT
|
||||
`~/.mozilla/firefox/` — find it via the running process if the default
|
||||
path is empty.
|
||||
- `grep network.trr. <profile>/prefs.js` — **no entries = shipped default,
|
||||
and since Firefox 113 the shipped default is DoH ON.**
|
||||
|
||||
Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy
|
||||
& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** →
|
||||
`franksdank.cloud` (and `franksdank.casa`). The exception list = domains
|
||||
that do NOT use DoH; they fall back to the system resolver, which has the
|
||||
route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome:
|
||||
Settings → Privacy and security → Security → "Use secure DNS" → off, or
|
||||
"Use the same settings as your system". Do NOT hand-edit `prefs.js` while
|
||||
the browser is running — it holds prefs in memory and rewrites the file on
|
||||
exit, clobbering the edit.
|
||||
|
||||
This is a per-client fix: every LAN client that must reach the homelab by
|
||||
name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not
|
||||
bypass it with DoH. The server side needs no change. Verified on blankpad
|
||||
2026-09-21: the drop-in was present since 09-12, curl returned 200, and
|
||||
Firefox 155 with default (ON) DoH was the remaining hole.
|
||||
|
||||
## Porkbun zone ops (franksdank.cloud)
|
||||
|
||||
Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so
|
||||
the API key/secret sourced from `/home/dank/serverAppConfigs/.env`
|
||||
(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe
|
||||
scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' <
|
||||
/tmp/script.sh`.
|
||||
|
||||
Endpoints (all POST, JSON body):
|
||||
|
||||
- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud`
|
||||
— auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong
|
||||
here). Response shape is **top-level `.records[]`**, NOT
|
||||
`.content.records[]` as some docs claim — parse with
|
||||
`jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
|
||||
- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
|
||||
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
|
||||
`type`, `content`, `ttl`, `prio`).
|
||||
|
||||
Quirks that bite (verified live 2026-08-31):
|
||||
|
||||
1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud`
|
||||
yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare
|
||||
label (`ai`) and the API appends the zone itself. Verify after every
|
||||
rename with a retrieve before declaring done.
|
||||
2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of
|
||||
rapid mutations makes some calls return `<!DOCTYPE html>...` where jq
|
||||
then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do
|
||||
not assume the call succeeded — sleep ~8 s and retry the failed ids;
|
||||
verify by re-listing the zone at the end.
|
||||
3. Deletes/renames are per-record-id: always retrieve first, map name→id,
|
||||
then mutate by id.
|
||||
|
||||
Rename procedure (vhost rename = DNS + Caddy together):
|
||||
|
||||
1. `retrieve` → collect ids for the renamed names.
|
||||
2. `edit` each id with the new **relative** label, preserving
|
||||
type/content/ttl.
|
||||
3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
|
||||
4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the
|
||||
Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test
|
||||
fails the suite if any Caddyfile vhost lacks an addn-hosts A record on
|
||||
192.168.0.143.
|
||||
5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed
|
||||
paths). Capture the suite's true exit with
|
||||
`out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit
|
||||
code behind `tail`'s own status and has shipped untested commits here.
|
||||
6. Verify live: `dig +short <name> @192.168.0.143` then
|
||||
`curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302;
|
||||
first hit triggers dns-01 issuance, which works without inbound port 80).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
type: reference
|
||||
title: Observability Provisioning
|
||||
description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
|
||||
tags: [observability, prometheus, grafana, provisioning, deployment]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Observability Provisioning
|
||||
|
||||
The monitoring stack is **provisioned as code**: dashboards, datasources,
|
||||
scrape jobs and alert rules live in the repo and ride the normal deploy flow.
|
||||
No hand-edits inside containers — they are wiped on recreate and never
|
||||
committed.
|
||||
|
||||
## The config triad
|
||||
|
||||
| Repo path | What it holds |
|
||||
|---|---|
|
||||
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
|
||||
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
||||
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
||||
|
||||
Targets and datasources reference **container names** (`loki:3100`,
|
||||
`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
|
||||
by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
|
||||
are host-netns services, not containers. Prometheus 3 refuses HTML responses
|
||||
as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
|
||||
exporter (see [[Stale Signal Audit]]).
|
||||
|
||||
## The deploy path
|
||||
|
||||
`scripts/deploy-remote.sh` carries two copier functions:
|
||||
|
||||
- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
|
||||
|
||||
`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
|
||||
`shared/grafana/**` to the `prometheus`/`grafana` services, so
|
||||
`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
|
||||
touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
|
||||
exist, the copier functions exist, and the provisioned paths match the units'
|
||||
mounts.
|
||||
|
||||
## Entry points
|
||||
|
||||
**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
|
||||
vantage first: container name if the exporter is a container on the same
|
||||
netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
|
||||
if it is an HTTP probe, not a metrics endpoint.
|
||||
|
||||
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
|
||||
Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
|
||||
Rules reload with the prometheus service.
|
||||
|
||||
**Add a datasource** — drop a YAML file into
|
||||
`shared/grafana/provisioning/datasources/`. Point it at the container name of
|
||||
the backend (`http://prometheus:9090`), not loopback.
|
||||
|
||||
**Add a dashboard** — drop the JSON into
|
||||
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
|
||||
unified storage, not (only) the SQLite table — grep the API, not the DB, when
|
||||
verifying (see the verify section of [[Fleet Audit and Posture]]).
|
||||
|
||||
## Verify through the APIs, not the filesystem
|
||||
|
||||
```bash
|
||||
# prometheus: rules and targets actually loaded
|
||||
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
|
||||
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
|
||||
# grafana: datasource + dashboard provisioned
|
||||
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
|
||||
```
|
||||
|
||||
Health answers from a stale process and provisioned resources can hide in
|
||||
unified storage — a file existing on disk is not proof the running server
|
||||
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
||||
table (refused = outage, 404/HTML = config defect).
|
||||
|
||||
## Current delivered state (2026-09-17)
|
||||
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
|
||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
||||
via `host.containers.internal`) → signal-cli. Some targets read down for
|
||||
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
||||
the blackbox exporter. See [[Observability]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — stack topology and the alert delivery chain
|
||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
|
||||
- [[Quadlet Lifecycle]] — the units that host these services
|
||||
- [[Deploy State Triage]] — how the planner selects these services
|
||||
@@ -271,6 +271,83 @@ graph TD
|
||||
6. Step up in size: 9B, then gpt-oss-20b.
|
||||
7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
|
||||
|
||||
## Registering packs absent from the catalog (NPU2)
|
||||
|
||||
Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
|
||||
to the FLM registry. Registry facts: user registry
|
||||
`/home/dank/.config/flm/model_list.json`, shape
|
||||
`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
|
||||
file_url, size, flm_min_version, files[], vlm, default_context_length,
|
||||
max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
|
||||
resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
|
||||
env var set lists custom tags (`⏬` = not downloaded).
|
||||
|
||||
1. Back up the registry JSON first (`cp -a`).
|
||||
2. Insert the entry via a python json edit + atomic `os.replace`. Set
|
||||
`files` to the FULL HF file list — **including `vision_weight.q4nx` even
|
||||
for text-only packs**: the qwen3.5-family loader opens it unconditionally
|
||||
and a missing file makes the first request HANG silently (no error log).
|
||||
3. **Do not use `flm pull <tag>` for custom entries** — it throws
|
||||
`json.exception out_of_range key '<tag>:<size>' not found` while building
|
||||
the download list. Place files manually into
|
||||
`/home/dank/.config/flm/models/<PackName>/` with
|
||||
`curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
|
||||
— plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
|
||||
weights; verify byte counts against the HF tree API before trusting a
|
||||
download.
|
||||
4. Kernel packs: copy from the same-shape family pack (shape-match
|
||||
`config.json` on hidden_size/num_hidden_layers/head_dim first) into
|
||||
`/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
|
||||
(root-owned dir).
|
||||
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
|
||||
the new process loads with all files present.
|
||||
6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
|
||||
`/v1/models` alone proves registration, not loadability.
|
||||
|
||||
Hard constraints: only ONE `flm serve` process can hold the NPU hardware
|
||||
context — a second concurrent serve process fails every load with
|
||||
`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
|
||||
0x2000003); never run scratch/verify instances while production holds the
|
||||
NPU. Tag switching happens INSIDE one serve process; the first request
|
||||
after a model switch pays ~6–11 s cold-load cost (visible in
|
||||
`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
|
||||
pattern present in your own command line — use bracket-guarded patterns
|
||||
(`[q]wen3.5:4b`) or exact PIDs.
|
||||
|
||||
## Tag load-failure diagnosis (200 with an error body)
|
||||
|
||||
`POST /v1/chat/completions` answers **HTTP 200** with a flat
|
||||
`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
|
||||
returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
|
||||
200. A status-code-only client reads this as a successful empty completion —
|
||||
**always check for the `error` key**. Two distinct causes, distinguished via
|
||||
`journalctl -u <flm-service> --full`:
|
||||
|
||||
1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
|
||||
Correct behavior, not a bug.
|
||||
2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
|
||||
downloads missing weight files on the fly (first request hangs ~90–110 s),
|
||||
then fails `Failed to load model: No such file
|
||||
'/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
|
||||
attempts fail fast (~2 s). Root cause: the serve script exports
|
||||
`FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
|
||||
kernel lookup resolves under the user dir, which only contains symlinks
|
||||
someone linked. **A tag listed in `/v1/models` is NOT proof it is
|
||||
loadable on this host.** Fix: one symlink per missing pack
|
||||
(`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
|
||||
/home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
|
||||
symlink the first request loaded and answered in 7.4 s wall.
|
||||
|
||||
Switch-cost behavior: the `model` field routes per-request (no restart
|
||||
needed); a request whose model differs from the resident one triggers a full
|
||||
reload (7–11 s wall for an 8-token completion on ~4B-class packs);
|
||||
alternating tags pays the swap each time. Single-lane NPU: concurrent
|
||||
requests queue FIFO behind the current lock holder (`NPU Locked!` →
|
||||
`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
|
||||
the journal for `Failed to load model` (the quoted xclbin path names the
|
||||
missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
|
||||
installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
|
||||
|
||||
## Incident: qwen3.6-moe unreachable (2026-08-16)
|
||||
|
||||
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
type: reference
|
||||
title: FreeToken Builds
|
||||
description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
|
||||
tags: [freetoken, build, dankclaw, vllm, mtp]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# FreeToken Builds
|
||||
|
||||
FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
|
||||
vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
|
||||
different dependency traps. Both were verified end-to-end 2026-09-11/12.
|
||||
|
||||
## Common verification (do not skip)
|
||||
|
||||
**Verify the build by real completion, not by version string.** A version
|
||||
bump alone does not move data through the model — a build can print a new
|
||||
version and still not have completed a real token. Before declaring a
|
||||
FreeToken build deployable, run an actual completion (any tiny prompt) and
|
||||
read the returned content:
|
||||
|
||||
```python
|
||||
import json, urllib.request
|
||||
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
|
||||
json.dumps({"model": "frank-small", "messages": [{"role": "user",
|
||||
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
|
||||
print(json.load(urllib.request.urlopen(req, timeout=600)))
|
||||
```
|
||||
|
||||
A 200 with a real `choices[0].message.content` is the only build green.
|
||||
|
||||
## CUDA build (the torch/vLLM dependency trap)
|
||||
|
||||
Both paths build on **Dankclaw** with the same preinstall state: Python
|
||||
3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
|
||||
download.pytorch.org/whl/nightly/cu130, installed with
|
||||
`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
|
||||
--index-url https://download.pytorch.org/whl/nightly/cu130` — use
|
||||
`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
|
||||
version). Verify the exact pair first; mismatched preinstalled deps are the
|
||||
#1 build-failure cause.
|
||||
|
||||
Then install the **matching vllm dev wheel**:
|
||||
|
||||
```bash
|
||||
uv pip install --index-strategy unsafe-best-match \
|
||||
--index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu130
|
||||
```
|
||||
|
||||
The wheel URL pattern is fixed:
|
||||
`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
|
||||
— read it back from vLLM's CI-run page (`run details →
|
||||
build_and_install_package` job, "install the vllm wheel" step). **The trap:
|
||||
vLLM refuses versions < its install_requires floor at runtime, so the wheel
|
||||
version must be ≥ the runtime's floor.** A wheel built at a lower base
|
||||
(0.5.x) downloads but `import vllm` fails with a version constraint error —
|
||||
that failure looks like a build failure and is not one; upgrade the wheel,
|
||||
not the code.
|
||||
|
||||
If `uv pip install -e . --no-deps` refuses (some repos pin the
|
||||
`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
|
||||
after `python -m venv --copies .venv && source .venv/bin/activate` and
|
||||
`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
|
||||
re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
|
||||
|
||||
## ROCm build (gfx1151 via TheRock nightly)
|
||||
|
||||
Same venv discipline, ROCm wheels from TheRock's nightly index:
|
||||
|
||||
```bash
|
||||
uv pip install --index-strategy unsafe-best-match \
|
||||
--index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
|
||||
"torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
|
||||
```
|
||||
|
||||
Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
|
||||
repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
|
||||
prebuilt extension wheels into the tree (the repo's `setup.py` skips
|
||||
compiling them when found). The ROCm build also needs
|
||||
`install-fastrak-gfx1151.sh` from the repo's scripts and
|
||||
`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
|
||||
x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
|
||||
`franksdank/freetoken-ops-gfx1151` — fetch it once and point
|
||||
`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
|
||||
the extension API changed.
|
||||
|
||||
`--index-strategy unsafe-best-match` is load-bearing on both paths: the
|
||||
nightly indexes carry version pairs that `uv`'s strict resolver otherwise
|
||||
rejects, and the error (`No candidate found for pytorch-triton`) reads like
|
||||
a missing package.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
|
||||
- [[Llama Cpp Serving Sizing]] — the harness measurement discipline
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
type: reference
|
||||
title: Heterogeneous Split Triage
|
||||
description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
|
||||
tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Heterogeneous Split Triage
|
||||
|
||||
Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
|
||||
the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
|
||||
across device A and device B" with arithmetic instead of enthusiasm.
|
||||
|
||||
## 1. Characterise the link (bytes vs latency)
|
||||
|
||||
```bash
|
||||
lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:" # capability vs actual
|
||||
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
|
||||
```
|
||||
|
||||
Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
|
||||
upshifts under load. Measure while the device is busy, or you size the design
|
||||
against the wrong number. Per-hop cost against the hidden state
|
||||
(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
|
||||
Gen4 x4.
|
||||
|
||||
- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
|
||||
for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
|
||||
never the constraint.**
|
||||
- The real budget line is **hop COUNT**: every device boundary is a
|
||||
host-side sync. A runtime that splits per *op* hits ~100 splits/token and
|
||||
pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
|
||||
assume.
|
||||
- **Design rule: partition by where the weights live, never by where the
|
||||
activations flow.** Resident weights, activations crossing.
|
||||
|
||||
## 2. Get the workload's real bottleneck (KV is usually it)
|
||||
|
||||
```
|
||||
KV/token = kv_bytes_per_position × context
|
||||
```
|
||||
|
||||
Attention reads the whole cached context every token, so KV/token scales
|
||||
with context — compare it against expert/weight traffic per token and total
|
||||
token time. At long context on a hybrid arch (few attention layers, many
|
||||
positions), **KV dominates**. Consequences:
|
||||
|
||||
- The highest-leverage placement is **KV cache on the fastest-memory device,
|
||||
together with its attention compute** — never KV alone: separating them
|
||||
puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
|
||||
hundreds of ms/token).
|
||||
- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
|
||||
attention_trunk ≤ VRAM`.
|
||||
|
||||
## 3. What placement never buys
|
||||
|
||||
Prefill is compute-bound and expert weights are read once per batch, so
|
||||
moving them to faster memory buys nothing while adding a serial stage.
|
||||
Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
|
||||
(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
|
||||
GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
|
||||
`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
|
||||
decode gain to within ~2 points — usable for decode projections, and it
|
||||
assumes sequential stages (true at batch 1; no pipelining).
|
||||
|
||||
## 4. Is the second device even a separate pool?
|
||||
|
||||
On a UMA APU, check before assuming additivity:
|
||||
|
||||
```bash
|
||||
cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small)
|
||||
cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM
|
||||
rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees
|
||||
```
|
||||
|
||||
A large GTT window means the "GPU memory" is the *same* LPDDR under one
|
||||
memory controller: adding a stage there adds a compute engine, not capacity
|
||||
or bandwidth. Only the discrete card's VRAM is additive. Fencing more
|
||||
(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
|
||||
|
||||
## 5. Speculative decoding as the "cheap" split
|
||||
|
||||
A drafter on the *other* device is bandwidth-free (token ids per round, tens
|
||||
of bytes), so it is tempting. Before proposing it:
|
||||
|
||||
- the drafter needs the target's **exact tokenizer** for a plain draft model
|
||||
(compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
|
||||
must be a head **trained for that target** (EAGLE/DFlash/MTP are
|
||||
conditioned on the target's hidden states and are not portable between
|
||||
models);
|
||||
- a plain drafter costs **K sequential forwards per round**, so it must be
|
||||
10–50× faster than the target. Break-even with `r = t_draft/t_target`:
|
||||
`rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
|
||||
(a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
|
||||
- compare against the **incumbent**, not the no-draft baseline: a target
|
||||
already running its own MTP head is a high bar;
|
||||
- check the runtime accepts an external drafter at all — closed engines may
|
||||
enumerate a fixed drafter set and 400 anything else (capability gates
|
||||
keyed on the loaded checkpoint). That is a design boundary, not a config
|
||||
gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
|
||||
(draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
|
||||
|
||||
## 6. Expressing a split (llama.cpp, the usual vehicle)
|
||||
|
||||
```bash
|
||||
-dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest)
|
||||
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (KV co-location)
|
||||
```
|
||||
|
||||
- buffer types match by name at load (`ggml_backend_buft_name`), so device
|
||||
names work as `-ot` values;
|
||||
- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
|
||||
- several backends in one binary is fine (CUDA + Vulkan is the practical
|
||||
pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
|
||||
build's CMakeCache tells you what needs adding;
|
||||
- contiguous ranges are the WRONG split when KV bandwidth dominates: they
|
||||
drag half the attention layers (and their KV) back to the slow pool.
|
||||
|
||||
## 7. Hazards checklist
|
||||
|
||||
- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
|
||||
are read from; moving them is pure loss.
|
||||
- Keep `mmap` on for models with huge embedding tables (anonymous memory
|
||||
gets the process cgroup-killed).
|
||||
- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
|
||||
— check before assuming optimisations apply.
|
||||
- A brand-new backend on the second device may not implement the arch's ops
|
||||
at all: make the first measurement a **baseline of the runtime on that
|
||||
device alone** — a split's arithmetic is worthless if the baseline is
|
||||
worse than the incumbent engine.
|
||||
- A split consumes the fast device entirely — check what else it was
|
||||
serving before trading it away.
|
||||
- Benchmark discipline on production hosts: stop and mask first
|
||||
([[Fleet Ops Playbook]]).
|
||||
|
||||
## 8. Verdict shape
|
||||
|
||||
Rank and report; prefer the boring answer when it wins:
|
||||
|
||||
1. **Two independent endpoints** (zero shared traffic) — often the real
|
||||
answer to "what's a useful combination", especially over a narrow link.
|
||||
2. **One process, weights-resident, KV co-located with attention on the
|
||||
fast pool** — promising and quantifiable (+% decode at long context);
|
||||
needs a dual-backend build and a baseline measurement.
|
||||
3. **Hot-expert cache / per-token weight streaming** — affordable only if
|
||||
the miss set per token is small (a few experts; check
|
||||
`misses × bytes × tok/s` against link bandwidth) and overlapped; injects
|
||||
PCIe latency into the critical path; no mainstream runtime implements a
|
||||
dynamic expert cache.
|
||||
4. List explicitly what does NOT help: TP/collectives across unequal
|
||||
devices, KV on the far device, model loading over the link,
|
||||
prefill-motivated placements.
|
||||
|
||||
## Related
|
||||
|
||||
- [[PCIe Link and Atomics]] — the link and collective walls underneath
|
||||
- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
|
||||
- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic
|
||||
@@ -0,0 +1,271 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Bench Windows and Measurement
|
||||
description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
|
||||
tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# LLM Bench Windows and Measurement
|
||||
|
||||
Fleet law first: **STOP the production service before any GPU experiment**
|
||||
(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
|
||||
GPU and corrupts results; an unisolated GPU experiment has precedent for
|
||||
wedging a host until power cycle. Reference harnesses:
|
||||
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
|
||||
`scripts/autoresearch/frank-primary-arms.sh`.
|
||||
|
||||
## 1. The window structure (one config per arm, a driver owns the window)
|
||||
|
||||
1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
|
||||
no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
|
||||
EXITED containers and can never clear — `rm -f` the exited one first) AND
|
||||
port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
|
||||
`ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
|
||||
then polls `/health` gets **200 from the still-running PRODUCTION engine**
|
||||
— the arm "passes" with production's boot log, then `podman exec` dies
|
||||
with `container state improper`.
|
||||
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
|
||||
LLM hosts — one card, one server). The serve script takes its config via
|
||||
env-file override (`FRANK_SMALL_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
|
||||
pointing at a staged copy; the checkout is never mutated.
|
||||
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
|
||||
the loop — a refused boot dies in one poll instead of 20 min. Capture the
|
||||
serve script's FULL stdout on failure (a `tail -8` loses engine-init root
|
||||
causes; raise happens above the tail).
|
||||
4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
|
||||
SpecDecoding lines) — they survive measure crashes.
|
||||
5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
|
||||
`podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
|
||||
before the next boot; restore production from the EXIT trap and verify the
|
||||
**conjunction oracle**: health 200 + `is-active` + `is-enabled` +
|
||||
mask-symlink clean. A bare `systemctl start` is not a restore oracle.
|
||||
(`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
|
||||
enough: systemd never auto-restarts an explicitly stopped unit.)
|
||||
6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
|
||||
mixed speeds.
|
||||
|
||||
**Destructive tails (2026-10-01):** a scratch script whose tail does
|
||||
`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
|
||||
probe loop survived the driver kill — it can destroy a just-deployed
|
||||
production engine (signature: deploy exits green, then seconds-to-minutes
|
||||
later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
|
||||
crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
|
||||
and kill strays by explicit PID; killing the dev-side job does NOT kill the
|
||||
host-side `deploy-remote.sh` (two of them once managed the same unit).
|
||||
|
||||
Two failure shapes inside the window that read as "the bench died":
|
||||
|
||||
- **Stale reads from a stopping container.** `podman stop` answers while the
|
||||
engine inside is still flushing; a `podman logs` taken in that window
|
||||
returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
|
||||
`podman wait`) before reading logs, or you will write "no results" into
|
||||
the table for a run that has them.
|
||||
- **A D-state worker wedges the stop.** A bench child that wedged in the
|
||||
kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
|
||||
then the container is `Stopping` forever. Kill by container first
|
||||
(`podman kill <ctr>` → SIGKILL path), verify the process is gone from
|
||||
`podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
|
||||
window script proceed to a redeploy with a zombie bench worker holding
|
||||
VRAM.
|
||||
|
||||
## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
|
||||
|
||||
**What llama-bench cannot express — decide BEFORE designing:** no
|
||||
`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
|
||||
activation buffers fit beside the DEPLOYED KV — only the real unit answers
|
||||
that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
|
||||
(`tg` numbers are the drafterless baseline, useful as a control only), and a
|
||||
warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
|
||||
cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
|
||||
~zero depth and meaningless for long conversations.
|
||||
|
||||
**The separator trap:** `/` separates devices WITHIN one configuration, `,`
|
||||
separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
|
||||
single value 36 → everything on one card. `--tensor-split 36,29` (server) →
|
||||
`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
|
||||
|
||||
**Harness shape:** one invocation per cell (an infeasible cell calls
|
||||
`exit(1)` and kills every later cell of a cross-product invocation — and
|
||||
infeasibility is usually the question); TSV row per cell as it completes +
|
||||
`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
|
||||
--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
|
||||
intended looks fast); resume by skipping non-empty cell outputs; stop the
|
||||
service in the harness, restore from `trap … EXIT`, verify afterwards. Match
|
||||
the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
|
||||
|
||||
**Run under a transient unit, not a shell:**
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/<harness>.sh
|
||||
```
|
||||
|
||||
Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
|
||||
8 MiB and fails the load for an unrelated reason. This also survives ssh
|
||||
disconnects. **Boot rungs under the production environment, not a bench
|
||||
shell** — a rootless shell probe booted 3×262144 bf16 that production refused
|
||||
(systemd `User=` + memlock change what the pinned-expert path holds):
|
||||
`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
|
||||
Restart=no env VARFILE=… /path/serve.sh`.
|
||||
|
||||
**Verify before burning hours:** dry-run against a STUB binary with
|
||||
`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
|
||||
including one forced-`exit 3` cell — prove the TSV row, failure capture,
|
||||
resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
|
||||
nonexistent file. **Reading:** bracket the sweep (reference cell first AND
|
||||
last); check per-cell `stddev_ts` (a difference smaller than spread is not a
|
||||
result); single-sequence decode is usually flat across batch/ubatch (prefill
|
||||
knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
|
||||
smaller-batch cells); report what the grid cannot prove (drafter, slots, real
|
||||
context).
|
||||
|
||||
## 3. HTTP benches of server-side parameters
|
||||
|
||||
`llama-bench` benches the bench binary, not the running server — server-side
|
||||
params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
|
||||
HTTP:
|
||||
|
||||
- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
|
||||
`prompt_tokens / wall_time`.
|
||||
- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
|
||||
spec acceptance all stabilize over thousands of tokens); measure
|
||||
`predicted_per_second` from `timings` or `completion_tokens / wall_time`;
|
||||
temperature 0.
|
||||
|
||||
Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
|
||||
`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
|
||||
sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
|
||||
reasoning parser streams zero deltas — use `/v1/completions`.
|
||||
|
||||
## 4. The measurement law (under speculative decoding this is absolute)
|
||||
|
||||
**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
|
||||
mapping; chunk counting reported 8.3 tok/s for a config whose server counters
|
||||
said 23–26 — a 3x false regression that nearly abandoned the change that
|
||||
worked. Always:
|
||||
|
||||
```python
|
||||
decode_tok_s = usage.completion_tokens / (wall - ttft)
|
||||
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
|
||||
```
|
||||
|
||||
vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
|
||||
measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
|
||||
them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
|
||||
acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
|
||||
temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
|
||||
(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
|
||||
repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
|
||||
before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
|
||||
nearly shipped the wrong setting.
|
||||
|
||||
## 5. Honest methodology
|
||||
|
||||
- **Match the corpus to the workload.** Token soup under-reports speculative
|
||||
decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
|
||||
settings; acceptance, not step time, moves throughput. Build the corpus
|
||||
from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
|
||||
`--custom-output-len`.
|
||||
- **Salt every run** with a unique per-arm token in the first bytes — it is
|
||||
the prefix-cache key (and a proxy routing key). Without it arm 2 measures
|
||||
arm 1's warm cache.
|
||||
- **Two workload shapes need two clients:** distinct prompts (batch
|
||||
behaviour, prefill parallelism) and sequential turns with growing history
|
||||
(TTFT of the first streamed chunk per turn — the only measurement that
|
||||
exposes prefix-cache behaviour). These pull in opposite directions: a cold
|
||||
batch spreads prefill across replicas, a conversation's next turn only pays
|
||||
if it lands where its prefix is cached.
|
||||
- **Read the engine's ledger, not the card:** `Available KV cache memory` /
|
||||
`GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
|
||||
request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
|
||||
is a LIFETIME average across every prompt the replica ever saw — send the
|
||||
same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
|
||||
→ 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
|
||||
full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
|
||||
chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
|
||||
three points before treating a knob as a lever.
|
||||
- **Soak before believing:** 15–30 min of concurrent families, count failures
|
||||
AND latency; report hit/miss split and medians with min/max — the median of
|
||||
a bimodal distribution describes neither mode.
|
||||
- **Verification that is not the benchmark:** run the client the fleet
|
||||
actually uses; and a deployed unit is not proven until an unattended reboot
|
||||
brings it back.
|
||||
- **Record the "no" answers** with numbers and mechanism — placement balance,
|
||||
concurrency admission and chunk-size tweaks all measured flat or worse
|
||||
here; the wall was arithmetic (cached prefixes + live sequences share one
|
||||
pool).
|
||||
|
||||
## 6. Sweep hygiene (the arms that produced wrong verdicts)
|
||||
|
||||
- **Prefetch uniformity:** every arm in a sweep carries the same
|
||||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
|
||||
prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
|
||||
12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
|
||||
verdict.
|
||||
- **Knob-landed preflight:** for every `-e` knob, confirm
|
||||
`podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
|
||||
produces a "parity" arm that is really the default (v1 of the gather sweep
|
||||
lost all 3 arms this way; serve-script hooks are additive and can drop
|
||||
vars).
|
||||
- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
|
||||
deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
|
||||
compare against default-off, not "incumbent + one flag".
|
||||
- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
|
||||
autofit (remainder) so pool numbers are reproducible; compute need =
|
||||
tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
|
||||
states — measure, don't assume 2x).
|
||||
- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
|
||||
deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
|
||||
one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
|
||||
- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
|
||||
`mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
|
||||
|
||||
## 7. Bash laws that ate windows (each cost an arm)
|
||||
|
||||
1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
|
||||
`SPEC=3: command not found`, nothing runs. Pass literal assignments:
|
||||
`SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
|
||||
2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
|
||||
an if/else it aborts ONLY that compound statement — the script CONTINUES,
|
||||
every arm tears down clean with zero measurements. Detection law: `ls`
|
||||
the per-arm `.measure.txt` files before trusting any summary; arm echoes
|
||||
are not evidence.
|
||||
3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
|
||||
it aborts AFTER the harness stopped production. Two statements.
|
||||
4. `sed -i` renames the file; already-running bash keeps the OLD inode —
|
||||
mid-flight patches miss arms launched in the preceding minutes. Patch
|
||||
early or run a make-up window.
|
||||
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
|
||||
(`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
|
||||
fails silently. Same trap in `git commit -m`: backticks become command
|
||||
substitution.
|
||||
6. No comments INSIDE a backslash-continued command (the continuation joins
|
||||
the comment); verify multi-line command construction with `bash -n` AND a
|
||||
render test.
|
||||
|
||||
## 8. Results location and recovery (frank-primary arms)
|
||||
|
||||
| copy | path |
|
||||
|---|---|
|
||||
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
|
||||
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
|
||||
| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
|
||||
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
|
||||
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
|
||||
|
||||
`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
|
||||
constants 99.9/88.8 identical across arms by construction; **identical
|
||||
numbers across arms == stub or bug, never a measurement.** After compaction,
|
||||
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
|
||||
logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
|
||||
window.
|
||||
|
||||
## Related
|
||||
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
|
||||
- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
|
||||
- [[LLM Performance Diagnosis]] — profiling and attribution methods
|
||||
- [[Remote Job Runner]] — detached execution of long sweeps
|
||||
- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Performance Diagnosis
|
||||
description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
|
||||
tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# LLM Performance Diagnosis
|
||||
|
||||
When a served endpoint is slower than wanted and someone names a suspect
|
||||
("PCIe is saturated", "the gather is the bound"), throughput arithmetic cannot
|
||||
settle it — it is usually circular (it assumes the suspect consumes the step).
|
||||
Get data in this order; the profiler window alone has retired multi-day
|
||||
optimization plans.
|
||||
|
||||
## 1. Is decode context-bound at all? (the flat-depth probe)
|
||||
|
||||
Sweep decode across context depth. **Flat** ⇒ attention over history is not
|
||||
the cost; the problem is per-token expert movement and KV/attention tuning is
|
||||
ruled out entirely. A genuinely context-bound server shows a clear downward
|
||||
slope. Record cold vs warm separately — a small gap means the cache is
|
||||
working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304
|
||||
to 16,095 tokens.
|
||||
|
||||
## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
|
||||
|
||||
Symptom: smooth ~25–35 % utilization with 100 % peaks on an expert-offload
|
||||
MoE. This is the EXPECTED signature, not a fault:
|
||||
|
||||
```
|
||||
per decode step: compute burst ~100 ms (100 %) → expert-gather wait ~400 ms (~0 %)
|
||||
resident fraction = arena slots / total experts (e.g. 6574/24576 ≈ 27 %)
|
||||
PCIe bytes/token = miss_rate × lookups × slab_bytes (0.73 × 480 × 2.344 MiB ≈ 0.84 GB)
|
||||
predicted duty ≈ compute_ms / (compute_ms + wait_ms)
|
||||
```
|
||||
|
||||
Validate the terms against the deployed engine's boot plan
|
||||
([[RDNA4 MoE Expert Offload VRAM Budget]]). If predicted ≈ observed, the low
|
||||
average is structural. Note small-transfer reality: bulk bandwidth (91 % of
|
||||
ceiling at 1 GiB) overstates what 2–3 MiB slabs get (~72 % plus ~20 %
|
||||
per-transfer latency). Before blaming the link, measure it
|
||||
([[PCIe Link and Atomics]]) and compare the engine's effective gather rate to
|
||||
the measured slab-size bandwidth — a 40–50 % gap is the software read path,
|
||||
not the wire. `rocm-smi --showuse` reads ~0 % during the whole multi-minute
|
||||
weight load: show VRAM (`--showmeminfo vram`) as the load-progress signal.
|
||||
|
||||
## 3. Kernel-time attribution (the profiler window)
|
||||
|
||||
Run BEFORE building any replacement kernel.
|
||||
|
||||
1. Stop + mask production; boot the shape under test with ONLY the profiler
|
||||
config added. vLLM argv:
|
||||
`--profiler-config '{"profiler":"torch","torch_profiler_dir":"/prof","torch_profiler_with_stack":false,"torch_profiler_record_shapes":false,"torch_profiler_with_memory":false,"torch_profiler_dump_cuda_time_total":false}'`
|
||||
— `with_stack=false` is load-bearing (with stacks the stop hangs and the
|
||||
trace buffer grows ~40 GB → kernel OOM-kill of the engine core, anon-rss
|
||||
~95 GB; the worker-side stop can hang 5 min on ROCm/kineto). Bind-mount
|
||||
`/prof`. If a shell substitutes values into that JSON, verify the escapes
|
||||
actually landed.
|
||||
2. Window it: warm request → `POST /start_profile` → N generations of ≥128
|
||||
tokens (~15 s of traffic) → `POST /stop_profile` (client may time out at
|
||||
60 s; the trace still lands). `telemetry_report` has ONE caller: the
|
||||
gpu_worker profiler-stop path. Overhead is small enough to keep decode
|
||||
comparable (9.6 vs 9.9–10.0 tok/s measured).
|
||||
3. **Parse ON THE SERVER** (traces are 100–150 MB gzipped, GB expanded).
|
||||
Rank `ph=="X"` events with `cat in {kernel, gpu_memcpy, gpu_memset}` by
|
||||
summed `dur` (µs): report % of device time, launch count AND µs/call
|
||||
(22 µs × 708k launches is a different problem than 4 ms × 4k), plus the
|
||||
first-to-last event span (fraction of wall the device was busy). Aggregate
|
||||
by FULL mangled name (template args are the tuning signal) and by stripped
|
||||
symbol. **Trap: `gpu_user_annotation` events masquerade as GPU time** (a
|
||||
naive sum reported 63 % in one `execute_context_0(...)` entry; spans
|
||||
overlap and can exceed the wall clock).
|
||||
4. Divide by the window's token count → ms/token; if kernel time ≈ step
|
||||
time, the table IS the optimization plan.
|
||||
|
||||
Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
|
||||
|
||||
```
|
||||
34.8 % 708,500 launches vectorized_elementwise_kernel <- the actual bound
|
||||
10.0 % 42,000 MoE W4A16 GEMM
|
||||
8.9 % 83,160 router/top-k (sort-based)
|
||||
6.5 % 133,920 hyper-connection pre/post kernels
|
||||
1.05 % 20,664 <the long-suspected expert gather>
|
||||
```
|
||||
|
||||
No amount of link arithmetic would have produced those two conclusions: the
|
||||
suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
|
||||
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
|
||||
predicted a negative result in advance (spec decode cannot help a shape whose
|
||||
per-token cost scales with the token count) which matched an unexplained flat
|
||||
MTP measurement.
|
||||
|
||||
## 4. Telemetry without benchmarking (live hosts)
|
||||
|
||||
When benching is forbidden (production serving), mine the runtime's own
|
||||
per-request telemetry — it measures the real workload and costs nothing.
|
||||
|
||||
- llama.cpp (root unit, `-lv 3`): `sudo journalctl -u <unit> --since "-3h"
|
||||
--no-pager` → `slot print_timing` lines (prompt eval, eval, total, `graphs
|
||||
reused`, `draft acceptance … mean len`, `n_gen = N, tg = X t/s, tg_3s = …`).
|
||||
- halogen (rootless quadlet): `journalctl --user -u <unit>` (**never sudo** —
|
||||
per-user storage) → `serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16
|
||||
rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok`.
|
||||
- Read the envelope, never one sample: short-prompt `prompt eval` is
|
||||
overhead-dominated (57 tok @ 168 t/s) — quote the longest available (1106
|
||||
tok @ 1013 t/s); decode and acceptance are content-dependent (0.46–0.76
|
||||
measured on one unchanged build); `graphs reused > 0` / `selected slot by
|
||||
LCP similarity` prove prefix-cache reuse.
|
||||
- **Attribute PCIe without touching the model:** background `nvidia-smi dmon
|
||||
-s t -c 16 -o T`, ONE ordinary `max_tokens=200` request, `wait` at the end
|
||||
(without it ssh closes and the client dies — the journal shows `W srv stop:
|
||||
cancel task` and NO print_timing while curl still reports plausible
|
||||
status). Measured: peak ~230 MB/s against a ~7,900 MB/s link = ~3 %, 0 at
|
||||
idle — the sampled-logits stream, not the transport, is decode traffic on
|
||||
that shape.
|
||||
- One `max_tokens=200` request is production traffic and fine; anything that
|
||||
pins the card is a bench → stop-and-mask first. Check WHICH instance you
|
||||
are reading (`systemctl status | head -12`; task ids jump across
|
||||
restarts). The engine's own ledger (`KV self size`, `n_slots`,
|
||||
`n_ctx_slot`) outranks your model of it.
|
||||
|
||||
## 5. The fork's data path (before blaming the link)
|
||||
|
||||
Offload engines pick between DMA copy-in and read-through UVA dereference
|
||||
inside the GEMM. Grep the installed package for the LRU/offload module and
|
||||
read the BRANCH CONDITION, not just the env names. A token-count threshold
|
||||
decides the path, and it interacts with spec decoding: an MTP step is k+1
|
||||
tokens, so the step can silently cross the threshold onto the slow path as
|
||||
draft depth grows. Check the threshold's premise: a default tuned at
|
||||
12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
|
||||
several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
|
||||
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
|
||||
check for the skip line before concluding autotuning ran. Gather-geometry
|
||||
knobs read at import time → each arm is a full reload.
|
||||
|
||||
## 6. RAM-bandwidth-bound MoE ceiling (before recommending a quant)
|
||||
|
||||
```
|
||||
routed experts/token = top_k × n_layers
|
||||
bytes/token = routed_params × bytes_per_param + shared_expert_bytes
|
||||
ceiling (tok/s) = measured_bandwidth / bytes_per_token
|
||||
```
|
||||
|
||||
Measure the HOST's real bandwidth, never assume DIMM ratings: 4-DIMM AM5
|
||||
boards downclock to DDR5-3600 regardless of 5600 sticks (two large DIMMs beat
|
||||
four small). Single-threaded reads under-report dual-channel ~4x — use the
|
||||
multi-threaded plateau of a checksumming reader (`scripts` pattern:
|
||||
`/tmp/bw.c`, 1/2/4/8/16 threads). Worked example (DeepSeek-V4.1-Flash):
|
||||
6×40 experts × 35.4M params = 8.5B params/token; at 3 bpw + 0.53 GB shared =
|
||||
**3.7 GB/token** → 42 GB/s ⇒ **11.3 tok/s ceiling**. Sanity anchors: H100
|
||||
NVL + CPU offload 226 GB ⇒ 5.8–9.0 tok/s; 2×3090 + 768 GB DDR4 ⇒ 12.2.
|
||||
CPU compute rarely binds below 4 bpw (~5.3 flops/byte ⇒ ~223 GFLOPS at 42
|
||||
GB/s); PCIe expert STREAMING is worse than CPU compute (Gen5 x8 ~22.7 GB/s
|
||||
effective < half of RAM); prefill reads the union of a batch's experts
|
||||
(prefill ≈ bandwidth / total_expert_bytes × ubatch). A quant that fits only
|
||||
by ignoring the Engram table does not fit (Engram = sparse n-gram lookup,
|
||||
NVMe-tolerant; the routed experts are not). An iGPU GTT aperture is an ALIAS
|
||||
of the same RAM, not extra memory.
|
||||
|
||||
## 7. IBS model-access attribution (AMD CPUs)
|
||||
|
||||
Real CPU access patterns for mmap'd weights, live, no stop (established on
|
||||
frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
|
||||
`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
|
||||
analyze; `--analyze-only`).
|
||||
|
||||
1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
|
||||
from CONSECUTIVE offsets, never a type-size table** — a wrong per-element
|
||||
size silently under-covers every tensor and dumps tail samples into a junk
|
||||
bucket (first run lost 59 % of expert samples; a uniform gap/exps ratio
|
||||
~1.4 across parts is the tell). Data start = `(tell + 31) & ~31`.
|
||||
2. `/proc/<pid>/maps` VMAs (7-group regex; pgoff = group 4, path = group 7)
|
||||
matched against part files; multi-VMA per part is normal.
|
||||
3. `sudo perf record -e ibs_op// -c 50000 --data -o out.perf -p <pid> --
|
||||
sleep <dur>` WHILE a mixed workload runs (idle perf numbers mean nothing).
|
||||
`sudo perf script -i out.perf -F addr` for addresses.
|
||||
4. Attribute: bisect VMAs, `foff = pgoff + (addr − vma_start)`, bisect tensor
|
||||
starts; per-expert index = `int(lo / (span / N_EXPERTS))` (expert-major
|
||||
rows).
|
||||
5. Verify before believing histograms: gap ≈ 0, every tensor-bearing part
|
||||
nonzero, and MoE blocks should be FLAT by construction (top-k fires every
|
||||
block every token) — a non-flat block histogram means mis-attribution
|
||||
first. Established law: expert routing is per-block flat, per-EXPERT
|
||||
skewed.
|
||||
|
||||
## 8. Prefix-cache-aware routing (multi-replica serving)
|
||||
|
||||
Measure with a **sequential-turn client**, not only a batch benchmark — a
|
||||
batch of distinct prompts parallelises prefill across replicas and hides the
|
||||
problem. The two failure modes: round-robin loses every other cache hit
|
||||
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
|
||||
pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
|
||||
pinned). **The rule that gets both: route by request shape** — a request
|
||||
carrying more than one message is a continuation (agent loop re-sending its
|
||||
context) → pin to the replica that served that prefix; anything else → spread
|
||||
with a COUNTER, not a prefix hash. Add failover: demote a replica on
|
||||
connection failure and retry on a live one BEFORE writing anything to the
|
||||
client (safe pre-stream; never retry after streaming started).
|
||||
|
||||
The wall no policy fixes: **a running sequence and its cached prefix draw on
|
||||
the SAME pool.** Three concurrent 50–80k families against 122,636 tokens/card
|
||||
thrash under every placement and admission policy tried (counter placement,
|
||||
load-balanced placement, one-large-request-per-card admission, both chunk
|
||||
directions — the last COSTS KV in both directions). When the working set
|
||||
exceeds aggregate KV, say so: the levers are capacity, fewer concurrent
|
||||
large contexts, or a different platform. Traps: a lifetime hit-rate counter
|
||||
is not a per-request verdict (probe directly: same prompt twice to an idle
|
||||
replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
|
||||
`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
|
||||
against the live endpoint before dashboards. Test shape: drive the routing
|
||||
function directly with synthetic bodies (stdlib-only policy module);
|
||||
conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
|
||||
hashes differently per turn and pinning silently never happens.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
|
||||
- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
|
||||
- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
|
||||
@@ -0,0 +1,281 @@
|
||||
---
|
||||
type: reference
|
||||
title: Model Swap and Rollback
|
||||
description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
|
||||
tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Model Swap and Rollback
|
||||
|
||||
Applies to every endpoint swap on the fleet (frank-small / frank-large /
|
||||
frank-primary). Repo law: all config flows through git; the live models are
|
||||
host state under `/home/dank/models/<name>/`.
|
||||
|
||||
## 1. Fit preflight — never deploy a swap without confirming it FITS
|
||||
|
||||
1. Get the checkpoint's on-disk/weight size AND precision from the HF card
|
||||
(`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params:
|
||||
BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
|
||||
2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that
|
||||
stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload,
|
||||
layer splits). Budget weights + KV pool + OS + page cache; **if weights
|
||||
alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink
|
||||
weights.
|
||||
3. **The failure mode is a self-sustaining thrash, not a crash:** a large
|
||||
swap makes the kernel PAGE instead of OOM-killing → sshd/user space
|
||||
starved indefinitely, no recovery window opens by itself; systemd
|
||||
`Restart=always` turns it into a crash-reload-rethrash loop; `enabled` +
|
||||
`WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle
|
||||
does NOT fix it.
|
||||
|
||||
Recovery (may need console / SysRq-to-GRUB):
|
||||
|
||||
```bash
|
||||
sudo systemctl mask --now <svc> # break the reboot trap FIRST
|
||||
cd <checkout> && git fetch origin && git reset --hard origin/main
|
||||
grep MODEL_DIR <env> # confirm the fixed target
|
||||
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
|
||||
curl -s <health> # then a REAL completion
|
||||
```
|
||||
|
||||
Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
|
||||
confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
|
||||
FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
|
||||
quant by size fit, then accept the compute path the card supports. Before
|
||||
choosing a smaller quant, state WHICH of the two you are buying — headroom,
|
||||
or a design change (a smaller quant does not change what the runtime can
|
||||
express; per-tensor cross-device placement stays refused under
|
||||
`--override-tensor`, and a layer split moves only contiguous layer ranges).
|
||||
|
||||
## 2. Ground truth first
|
||||
|
||||
- Read the HF model card (runtime pin, hard constraints, recommended args)
|
||||
+ the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
|
||||
`scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
|
||||
- Check the host before downloading anything: `ls /home/dank/models/`,
|
||||
`df -h /home`, `du -sh` — bench-era or prior-deployment files often already
|
||||
exist.
|
||||
|
||||
## 3. Fetch and verify (HF payloads)
|
||||
|
||||
**Inventory by QUANT NAME, never by extension** — large quants ship as
|
||||
`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
|
||||
`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
|
||||
mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
|
||||
repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
|
||||
return 403 even with a valid token until the account accepts the gate —
|
||||
prefer the ungated copy.
|
||||
|
||||
```bash
|
||||
# token out of band: mode-600 curl config; a token passed as -H is world-readable
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
|
||||
chmod 600 /tmp/hf.curlrc
|
||||
curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"
|
||||
```
|
||||
|
||||
`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes
|
||||
the error body as the artifact (identical 153-byte shards, cheerful DONE,
|
||||
zero bytes of model). Never `pgrep -a` a running fetch (token is in
|
||||
`/proc/<pid>/cmdline`); watch progress by size:
|
||||
`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`.
|
||||
|
||||
**Verify, then assemble:** per part, byte count == Hub's declared size AND
|
||||
`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total ==
|
||||
sum of parts (compute the sum with python, not by hand) and `head -c 4` ==
|
||||
`GGUF`. The assembled file has no published hash — the per-part oids + exact
|
||||
total are the chain. Keep the fragments when disk allows (only copies
|
||||
re-verifiable against the Hub); failure artifacts and the replaced payload
|
||||
go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants
|
||||
and the local file may be the last copy.
|
||||
|
||||
Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already
|
||||
do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to
|
||||
skip all three. HF cache snapshots SYMLINK every file into `blobs/` and
|
||||
those links dangle inside containers — serve scripts and the deployer refuse
|
||||
symlinks and missing tokenizer files for exactly this reason.
|
||||
|
||||
## 4. Runtime build (when the card pins a fork/build)
|
||||
|
||||
- Clone + build with the card's exact cmake line, plus
|
||||
`-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui
|
||||
bucket) aborts llama.cpp link at 98 % on these hosts; the API server is
|
||||
unaffected.
|
||||
- Verify every flag the serve script passes exists in the new binary BEFORE
|
||||
touching the service: `llama-server --help | grep -q -- "<flag>"` for each;
|
||||
list MISSes. Read the GGUF's tensor histogram first if the payload uses a
|
||||
custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]).
|
||||
|
||||
## 5. Config via git only
|
||||
|
||||
Edit `scripts/<svc>.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve
|
||||
script header/defaults locally; `bash -n`; `./tests/run.sh` must pass;
|
||||
commit; push; `git pull --ff-only` on the target. Never edit configs on the
|
||||
remote. Renaming a model dir (`mv`) is host state and allowed. The env-file
|
||||
hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged
|
||||
arms possible — keep it intact.
|
||||
|
||||
## 6. Live validation
|
||||
|
||||
- No scratch ports — one GPU cannot host two big models. Ensure the old unit
|
||||
is truly gone first: `systemctl is-active` can hang in `stop-sigterm`
|
||||
(llama.cpp never preempts an in-flight generation); poll until the PID
|
||||
exits.
|
||||
- Manual serve for smoke: `setsid nohup bash scripts/serve-<svc>.sh >
|
||||
/tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session.
|
||||
- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows
|
||||
(`reasoning_content` non-empty); if the endpoint takes images, a
|
||||
solid-color probe through an `image_url` content part (legacy top-level
|
||||
`images` is silently ignored; probe with ≥256 px).
|
||||
- vLLM specifics: first request after boot can take ~18 s (deferred Triton
|
||||
loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
|
||||
the contract ([[vLLM Boot Forensics and Config Interrogation]]).
|
||||
|
||||
## 7. systemd handoff — THE TRAP
|
||||
|
||||
`./scripts/deploy.sh <host> --service <svc>` gates health, but a leftover
|
||||
manual server holding the port makes the health check pass while the UNIT
|
||||
crash-loops on bind. Always confirm ownership:
|
||||
`systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server`
|
||||
(or the container runtime PID); only then trust "healthy". If a real
|
||||
long-context task is queued, do the handoff anyway — client aborts are
|
||||
recoverable, drift is not.
|
||||
|
||||
## 8. Close-out (mandatory)
|
||||
|
||||
1. Docs: active runbook updated (build recipe, profile, validation numbers
|
||||
incl. real-workload prefill/decode); superseded runbook gets a RETIRED
|
||||
banner and stays as the rollback record; `LLM/index.md` mermaid + active
|
||||
list; AGENTS.md fleet table + runtime section + history chain; work-log
|
||||
entry in `serverAppConfigs-docs/log.md`.
|
||||
2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
|
||||
context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
|
||||
the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
|
||||
`max-model-len` and its concurrency line. Overselling causes silent
|
||||
client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
|
||||
engine's cap, `input` capabilities (text/image), and
|
||||
`reasoningContentField` per runtime (llama.cpp/SGLang emit
|
||||
`reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry
|
||||
that named the old endpoint.
|
||||
3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary +
|
||||
payload kept on disk, rollback numbers recorded in the runbook.
|
||||
|
||||
## 9. Revert-pin an ad hoc experiment (the three pins)
|
||||
|
||||
A campaign is "done" only when all three exist — without them a
|
||||
`podman pull`, an image update, or a lost scratch script destroys the way
|
||||
back:
|
||||
|
||||
1. **Git tag (annotated, pushed):**
|
||||
`git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline +
|
||||
runbook path + revert command>"` — a future agent must reconstruct the
|
||||
campaign from the tag message alone.
|
||||
2. **Image digest in the serve script:** floating tags move. Capture
|
||||
`podman inspect <ctr> --format '{{.ImageName}} {{.Image}}'` + `podman
|
||||
images --digests`, patch `IMAGE=` to `repo@sha256:<repodigest>` with a
|
||||
comment (image ID, date, old ref), verify `podman image exists
|
||||
repo@sha256:<digest>` and `bash -n`. Digest refs survive upstream pushes;
|
||||
`podman pull repo@sha256:<digest>` re-fetches exact bytes. One host can
|
||||
hold MULTIPLE images of one repo — pin by the digest of the RUNNING
|
||||
container, never by tag name. Never edit the pinned line back to a
|
||||
floating tag; new experiments branch from the digest.
|
||||
3. **Local alias:** `podman tag <id> <exp>-runtime:tuned-YYYYMMDD`.
|
||||
|
||||
Record scratch-script paths + full flag surface in the runbook (hook env
|
||||
vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`.
|
||||
Pure measurement sessions that changed no state need only pin 1.
|
||||
|
||||
## 10. Research verification discipline (for "can model X run here?")
|
||||
|
||||
- Scouted research summaries have shipped fabricated citations on this fleet
|
||||
(invented HF repos, plausible-but-wrong PR numbers, stale engine versions).
|
||||
NEVER copy a scout's citations into a deliverable — re-verify each
|
||||
load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it
|
||||
enters anything.
|
||||
- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim
|
||||
about our own hosts.
|
||||
- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active,
|
||||
CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early
|
||||
GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path
|
||||
(heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K).
|
||||
GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the
|
||||
default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY,
|
||||
flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in
|
||||
unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant
|
||||
ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
|
||||
MMLU-Pro vs Q8_0).
|
||||
- GGUF parsing gotcha: expert tensors are
|
||||
`blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
|
||||
a `len(parts) == 3` filter matches zero layers with no parse error. Filter
|
||||
on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
|
||||
|
||||
## 11. Adding a second LLM endpoint to a host
|
||||
|
||||
When a host must serve two models at once (an iGPU runtime plus a second
|
||||
card), or one endpoint is moving and a new one takes the freed hardware.
|
||||
|
||||
**0. Decide the shape before touching hardware.** One provider per
|
||||
HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
|
||||
— a second endpoint is a new provider block, not a second model in the
|
||||
existing one. Read the budget as an inequality: for unified memory the
|
||||
engine's own fit check is the arbiter (`budget = host − pinned weights −
|
||||
reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV
|
||||
by counting only full-attention layers — and remember `--parallel N`
|
||||
DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free
|
||||
RAM by changing the POOL or the ARENA (the reserve is a fit-check term,
|
||||
never an allocation) and drop optional charges (a vision tower is ~2 GiB)
|
||||
rather than hoping.
|
||||
|
||||
**1. Repo first, and retire in the same commit.** New service files
|
||||
(`scripts/<svc>.env`, `.service`, `serve-<svc>.sh` with hard guards for
|
||||
missing artifacts); planner entries (`SERVICE_ORDER` + the target's service
|
||||
set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the
|
||||
dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a
|
||||
suffix — and check what pointed at them: bench harnesses, a chat template
|
||||
only the old runtime read, and `scripts/test-changed.sh` mappings are the
|
||||
usual victims (a mapping to a retired test file fails the pre-commit
|
||||
wrapper silently for weeks). One bats file per service asserting
|
||||
INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
|
||||
not constants; a negative grep must strip comments first
|
||||
(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
|
||||
the very thing you removed.
|
||||
|
||||
**2. The port-bind race (rootless podman).** pasta binds the published port
|
||||
at container CREATE; a just-stopped predecessor still holds it. Order the
|
||||
handler: retire the old manager's unit → stop THIS service's own unit
|
||||
(no-op on first cutover) → wait for every socket on the port to clear
|
||||
([[Container Host Reachability]]) → install → start. Without the second
|
||||
step a redeploy deadlocks against its own gate.
|
||||
|
||||
**3. Wire the surfaces, refuse impossible signals.** Point the harness
|
||||
small/fast + image roles at the new endpoint; move the vision role WITH the
|
||||
vision tower (one owner per capability). `shared/homepage/services.json`: a
|
||||
card per endpoint, probed from the monitor's own host. Prometheus: only add
|
||||
a job if the runtime actually exposes Prometheus text (halogen publishes
|
||||
`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
|
||||
reason written down rather than left as a permanent red).
|
||||
|
||||
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
|
||||
<large> --service <small>` (order matters); per endpoint: `/health`,
|
||||
`/v1/models` reports the alias, a chat completion returns content AND
|
||||
`reasoning_content`, a `tools` request returns `tool_calls`, an image
|
||||
request is answered (or refused with a named reason), the runtime's own
|
||||
speculation counters move, and the unit is `enabled`/`generated` with linger
|
||||
on. Read the engine's boot ledger for a silent clamp. Finally spend one
|
||||
real request through the harness per changed role — routing labels lie; the
|
||||
servers' own journals do not.
|
||||
|
||||
Traps that look like model faults: `500 tools param requires --jinja flag`
|
||||
(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected
|
||||
reasoning effort <level>` (fold via `compat.reasoningEffortMap`); the same
|
||||
request works by hand but a subagent dies (subagents run at a different
|
||||
thinking level — check both); the container dies instantly although the
|
||||
predecessor "stopped" (the port race above).
|
||||
|
||||
## Related
|
||||
|
||||
- [[GGUF Compatibility]] — which runtime can load a payload at all
|
||||
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
|
||||
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint
|
||||
@@ -0,0 +1,264 @@
|
||||
---
|
||||
type: reference
|
||||
title: Tcclaviger Fork Ops
|
||||
description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
|
||||
tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Tcclaviger Fork Ops
|
||||
|
||||
The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
|
||||
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
|
||||
first-class expert-offload planner) served `frank-primary` at shape G until
|
||||
the 2026-10-05 FreeToken cutover; it remains the rollback target
|
||||
(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM
|
||||
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
|
||||
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
|
||||
[[LLM Bench Windows and Measurement]].
|
||||
|
||||
## The ad-hoc sweep harness (dankserver)
|
||||
|
||||
Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
|
||||
depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
|
||||
`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
|
||||
(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
|
||||
JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
|
||||
into every fresh container — `podman cp` files die on container recreate.
|
||||
|
||||
- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
|
||||
`mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
|
||||
ttft)`; chunk counting is INVALID under spec decode.
|
||||
- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
|
||||
`out NAME` ([[Remote Job Runner]]).
|
||||
- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
|
||||
(else repetition-halt or early EOS), finish_reason, and inspect the text —
|
||||
`bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
|
||||
loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
|
||||
Acceptance swings 48–61% run-to-run — noise; decode is the metric.
|
||||
- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
|
||||
- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
|
||||
`Application startup complete` — it is on STDERR; an unmerged grep can
|
||||
never match and burns the whole wait timeout.
|
||||
- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
|
||||
are CPU-side); VRAM climbing is the load indicator.
|
||||
- Config edit discipline: the serve script is patched by scp'd python
|
||||
scripts with asserts (anchor must exist, addition must not); `bash -n`
|
||||
after every patch.
|
||||
|
||||
## Measured knob verdicts (2026-09-29 campaign)
|
||||
|
||||
Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
|
||||
TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
|
||||
|
||||
| Knob | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
|
||||
| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
|
||||
| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
|
||||
| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
|
||||
|
||||
Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
|
||||
batch-shaped compute knobs move ~nothing.
|
||||
|
||||
The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
|
||||
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
|
||||
the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
|
||||
depth.
|
||||
|
||||
Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
|
||||
flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
|
||||
profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
|
||||
kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
|
||||
(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
|
||||
fires ONLY from the worker's profiler-stop path; worker-side stop can hang
|
||||
>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
|
||||
measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
|
||||
temporal reuse dominates, so never infer miss rates from resident-slot
|
||||
fractions.
|
||||
|
||||
## Sweep hygiene (fork-specific preflight)
|
||||
|
||||
1. **Knob landed:** for each `-e` env knob, confirm in-container
|
||||
`podman exec … env` (or the engine log's confirmed-values line). A knob
|
||||
the fork silently ignores produces a "parity" arm that is really the
|
||||
default — the v1 gather sweep lost all three arms to exactly this
|
||||
(GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
|
||||
2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
|
||||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms
|
||||
with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
|
||||
(no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
|
||||
verdict.
|
||||
3. **Spec arms must prove engagement:** engine log
|
||||
`SpecDecoding metrics: Mean acceptance length …` and
|
||||
`curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
|
||||
4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
|
||||
refused boot dies in one poll instead of 20 min of health waits. Capture
|
||||
>8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
|
||||
refusal) lives higher up.
|
||||
5. Report per-depth tables, never one number: g32×64's win grows with depth
|
||||
(54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
|
||||
rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
|
||||
rows across arms only.
|
||||
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
|
||||
noise).
|
||||
|
||||
Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
|
||||
arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
|
||||
(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
|
||||
measure-stage bug costs decode tables but not boot-log findings.
|
||||
|
||||
**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
|
||||
is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
|
||||
4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
|
||||
0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
|
||||
4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
|
||||
OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
|
||||
tokens). Acceptance is real: mean length 2.4–2.8, per-position
|
||||
0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
|
||||
exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
|
||||
|
||||
## Serving a model larger than VRAM (dankserver recipe)
|
||||
|
||||
Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB
|
||||
RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives
|
||||
a bogus 186.7 GiB (the Raphael iGPU is in the list — see
|
||||
[[Llama Cpp Serving Sizing]] for the reading discipline).
|
||||
|
||||
**Kernel gate — this decides which checkpoint can work at all.**
|
||||
`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*`
|
||||
but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with
|
||||
BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup
|
||||
otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`,
|
||||
`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no
|
||||
kernel (the class exists, the r4d.so entry does not); RFA has neither. Check
|
||||
the checkpoint's quant against this list BEFORE downloading anything.
|
||||
|
||||
Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve
|
||||
"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`.
|
||||
Known-good invocation:
|
||||
|
||||
```bash
|
||||
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
|
||||
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
|
||||
--cap-add SYS_PTRACE \
|
||||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||||
-v /home/dank/models/<model>:/models:ro \
|
||||
docker.io/tcclaviger/vllm:latest /models \
|
||||
--served-model-name <name> --port 7770 \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 32768 \
|
||||
--kv-cache-memory-bytes 2147483648 \
|
||||
--max-num-seqs 4 \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--enable-prefix-caching --trust-remote-code
|
||||
```
|
||||
|
||||
`-p 7770:7770` is not included, so the API is only reachable via
|
||||
`podman exec` — add `-p` if host-side curl is needed. Load ~5 min;
|
||||
`/health` answers connection-refused the whole time; watch for
|
||||
`Application startup complete` in `podman logs <ctr>`. Verified working:
|
||||
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131
|
||||
shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10,
|
||||
multimodal, native 262,144).
|
||||
|
||||
Traps that each cost a boot:
|
||||
|
||||
1. **Mount the whole `/dev/dri`, never individual render nodes.** Render
|
||||
nodes without the primary `card*` nodes → HIP enumerates two agents and
|
||||
refuses a context on either (`RuntimeError: No CUDA GPUs are available`)
|
||||
while `device_count()` cheerfully returns 2. Confirmed by direct A/B.
|
||||
2. **memlock must be raised** (done on Dankserver via
|
||||
`/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any
|
||||
large pinned allocation fail as a misleading
|
||||
`torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`;
|
||||
it cannot be raised from inside a container or root's own shell — only
|
||||
PAM applies it at login. Matters because `--expert-host-mlock` defaults
|
||||
True.
|
||||
3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** —
|
||||
the planner sizes the arena against the budget and leaves KV the
|
||||
remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed
|
||||
at 0.64 GiB.
|
||||
4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the
|
||||
budget subtracts the KV reserve first, so a smaller context shrinks the
|
||||
reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV;
|
||||
32768 → 0.34 GiB).
|
||||
5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token
|
||||
planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost).
|
||||
Fix by pinning the pool: `--kv-cache-memory-bytes`.
|
||||
6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
|
||||
structurally orthogonal to `--expert-precise-memory` (verified
|
||||
in-container 2026-10-05: precise-memory only swaps the GPU overhead term,
|
||||
expert_offload_mem only sets the host bound; they rejoin only inside
|
||||
`place_experts`, and the plan rewrites `expert_offload_mem` back to the
|
||||
cap the plan met, so plan and allocation cannot drift).
|
||||
7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one.
|
||||
8. `podman run -d` without `-a` sends no stdout to a redirect — the log file
|
||||
only receives the container ID; use `podman logs`.
|
||||
|
||||
Status reads: use `scripts/dankstatus.sh`, never hand-rolled
|
||||
`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a
|
||||
TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and
|
||||
awk prints `0.0` — a false "host out of memory" that invites an unnecessary
|
||||
abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv
|
||||
(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count
|
||||
to the Hub's declared size; `-C -` resumes truncated shards.
|
||||
|
||||
**Check whether the stack supports the model natively before porting an
|
||||
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
|
||||
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
|
||||
through two image tags to bring an overlay up was entirely wasted work.
|
||||
|
||||
## Compile-cache persistence (survives `podman rm -f`)
|
||||
|
||||
Serve scripts that recreate the container discard the writable layer, so
|
||||
every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s`
|
||||
cold, of which 173 s was one graph).
|
||||
|
||||
1. **Prove it and locate it:** fresh container vs live
|
||||
(`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
|
||||
`podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
|
||||
f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
|
||||
`/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
|
||||
`/root/.cache/comgr`.
|
||||
2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
|
||||
both trees, safe whenever the image ships no such directory (step 1
|
||||
proves it; otherwise you would be hiding baked state). Wire as an opt-in
|
||||
tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
|
||||
`${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
|
||||
checkout.
|
||||
3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
|
||||
N s`, warm `Directly load the compiled graph(s) for compile range (…)
|
||||
from the cache` with `torch.compile took` in single-digit seconds.
|
||||
4. Traps: the cache key includes the compile RANGE — an MNBT change costs
|
||||
ONE recompile, not a permanent tax; do not claim a proportional boot cut
|
||||
(time to `/health` is dominated by weight loading: 490 s warm vs 188 s
|
||||
cold compile); stale entries are misses, not wrong answers (version +
|
||||
config hashed); if the serve script is seam-gated, assert the mkdir and
|
||||
the exact mount target.
|
||||
|
||||
## Plan-line verification checklist (after any fork deploy)
|
||||
|
||||
```
|
||||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
|
||||
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
|
||||
```
|
||||
|
||||
`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate
|
||||
these columns to `17.`. Verify: new budget/graphs/arena in the plan;
|
||||
`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
|
||||
memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
|
||||
exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
|
||||
real margin; `systemctl show frank-primary -p NRestarts --value` = 0;
|
||||
`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
|
||||
completion from a PEER — `curl localhost:7770/health` inside an ssh session
|
||||
on the host can return `000` and looks like an outage that isn't.
|
||||
|
||||
## Related
|
||||
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
|
||||
- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
|
||||
- [[Model Swap and Rollback]] — where the fork sits in the swap history
|
||||
- [[Remote Job Runner]] — the job.sh harness referenced above
|
||||
@@ -123,3 +123,27 @@ Stop and disable the root unit, trash it, restore the halogen quadlet from
|
||||
`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
|
||||
`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
|
||||
the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
|
||||
|
||||
## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
|
||||
|
||||
Why this deployment could not be tuned into a production endpoint: decode
|
||||
3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
|
||||
1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
|
||||
The decode root cause is structural: ~100 CPU↔GPU backend splits per token
|
||||
with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
|
||||
dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
|
||||
`pause` (ggml_barrier spin): threads wait on the serial critical path.
|
||||
Memory placement (mmap/anon/THP/deferred) does not matter — all variants
|
||||
landed 3.8–4.6.
|
||||
|
||||
- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
|
||||
per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
|
||||
gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
|
||||
stay dead.
|
||||
- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
|
||||
4× slower than ik's iqk) — do not switch engines for speed.
|
||||
- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
|
||||
NextN head is stripped (no `blk.48` tensors).
|
||||
- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
|
||||
decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
|
||||
an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.
|
||||
@@ -134,6 +134,59 @@ slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement
|
||||
0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on
|
||||
both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.
|
||||
|
||||
## The split: re-measure it at the deployed shape
|
||||
|
||||
The batch and the tensor split are **coupled**: compute buffers are not
|
||||
symmetric across the two cards, so any batch change moves the balance point.
|
||||
Order of work — sweep batch with llama-bench (above), THEN re-measure the
|
||||
split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading
|
||||
the real service with a per-config env override and reading `rocm-smi`. A
|
||||
llama-bench number cannot decide the split: it has no `--parallel`, sizes
|
||||
its own context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
|
||||
|
||||
The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND
|
||||
the port to be free (`ss -ltn | grep :7770` until gone) before the next
|
||||
load, or the transient server dies with `couldn't bind HTTP server socket`.
|
||||
Free bytes follow `card = weights_share + KV_share + constant + ubatch_term`,
|
||||
where the ubatch term costs the drafter-carrying card roughly **1.85×** the
|
||||
other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by
|
||||
measurement only: one `-ts` step is one layer, worth ~0.43 GiB on a DeltaNet
|
||||
boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is
|
||||
about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91;
|
||||
36,29 @512 → 6.01/7.32; **35,30 @512 → 6.44/6.90 (deployed)**.
|
||||
|
||||
Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` at the new
|
||||
deployed point and re-anchor the per-card constants; check neighbouring
|
||||
splits still FAIL the balance gate or the seam has stopped discriminating
|
||||
(keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy,
|
||||
then verify live: flags in `/proc/<pid>/cmdline`, `/slots`, VRAM matching
|
||||
the pre-deploy measurement, a tool call, two concurrent requests,
|
||||
`draft acceptance` lines on BOTH slots, 0 ROCm errors.
|
||||
|
||||
**Per-card utilization (who is hot, and the split lever)** — a different
|
||||
question from batch/prefill. Measure decode tok/s AND per-card
|
||||
`rocm-smi --showuse` at 2–3 neighbouring splits in ONE session. Pitfalls
|
||||
(each cost a re-run): run the transient unit as ROOT (a `pkill -9
|
||||
llama-server` as dank auto-restarts the service and `systemctl` as dank
|
||||
fails "Access denied"); warm up 3 throwaway requests before the timed one
|
||||
(first 384-token decode after load measured 1.77 tok/s against a true ~32);
|
||||
`rocm-smi --showmeminfo vram --csv` columns are `$2`=Total, `$3`=**Used** —
|
||||
parsing the wrong column spins the settle gate the full 600 s; the util% is
|
||||
sampled post-hoc (which card finished last), so treat it as "which card is
|
||||
the pipeline tail", not a duty cycle.
|
||||
|
||||
**What the split does and does not buy (measured 2026-09-17, 393216 /
|
||||
MTP n=3):** the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29
|
||||
all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at
|
||||
35:30 because the MTP drafter + output layers + heavier compute buffers are
|
||||
pinned to the card holding the LAST layers — not because PCIe is the
|
||||
bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch).
|
||||
36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput
|
||||
cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing
|
||||
preference is that the card with the FASTEST PCIe lane should be the more
|
||||
utilized one; deploying it requires re-fitting the seam's slack/floor and
|
||||
re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **`llama-bench` cannot run the drafter, the slots or the deployed context.**
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
|
||||
|
||||
**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
|
||||
|
||||
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
|
||||
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
|
||||
on the host into the local image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201` by
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served
|
||||
alias (`frank-primary`), same 131-shard payload
|
||||
(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
|
||||
git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore),
|
||||
runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
|
||||
## Why FreeToken
|
||||
|
||||
The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
|
||||
0.982 was a hard law (1.00 was **unbootable** — worker init raises
|
||||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
|
||||
total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
|
||||
still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
|
||||
fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
|
||||
GPU cache, sizes **KV as an explicit token pool**, and boots in one
|
||||
deterministic pass — no per-boot estimator to fight.
|
||||
|
||||
## The build (on dankserver, rootless podman as dank)
|
||||
|
||||
```bash
|
||||
cd /home/dank/serverAppConfigs
|
||||
podman build -t frank-primary-freetoken:pr587-77a7201 \
|
||||
-f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
|
||||
```
|
||||
|
||||
Laws baked into the Dockerfile (each one cost a failure):
|
||||
|
||||
- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
|
||||
the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
|
||||
rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
|
||||
rocm10.1.0 — an untested pair for this PR).
|
||||
- **`--no-build-isolation --no-deps -e .`** — a plain
|
||||
`pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
|
||||
build (the dependency set is authored for NVIDIA). Runtime deps are
|
||||
curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
|
||||
nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
|
||||
**flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
|
||||
FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
|
||||
are exactly those backends).
|
||||
- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
|
||||
versioned `libhiprtc.so.7` in
|
||||
`/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
|
||||
extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
|
||||
the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
|
||||
- FreeToken source is checked out at the **pinned PR head**, not the branch
|
||||
(a future push to `rocm714_torch214` cannot change the deployed engine).
|
||||
- The stale Docker Hub login (`docker login` creds at
|
||||
`~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
|
||||
with "invalid username/password" — `docker logout` fixes the pull. Hit
|
||||
2026-10-05 while pulling the base image.
|
||||
|
||||
## Measured at cutover (2026-10-05/06)
|
||||
|
||||
|
||||
- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
|
||||
pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
|
||||
**1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
|
||||
pinned expert/PLE slabs)**, 123 GB still available.
|
||||
- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
|
||||
1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
|
||||
FreeToken single-card is the deployed baseline.
|
||||
- **The single-card LRU ceiling**: the KV contract (harness worst case
|
||||
177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
|
||||
the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
|
||||
`ft ctl cache --moe` cannot raise it far on one card. The paths to a
|
||||
better decode rate are TP=2 (below) or a smaller pool (contract
|
||||
violation — the 177,356 floor is real traffic, not padding).
|
||||
- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
|
||||
zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
|
||||
allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
|
||||
resolve `cuda_runtime.h` from the hipified include path
|
||||
(`1 error generated when compiling for gfx1201`, ninja stopped). The PR
|
||||
author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
|
||||
edge, not a board or driver fault (radiance TP=2 allreduces fine on this
|
||||
pair since the 2026-09-20 swap). A future fix = patch the JIT include
|
||||
path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
|
||||
then.
|
||||
- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
|
||||
(finish stop); thinking split — `reasoning_content` non-empty AND
|
||||
`content` carries the answer (17×24 → "408"); native `tool_calls` with
|
||||
correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
|
||||
an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
|
||||
native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
|
||||
[text,image], sampling defaults from generation_config.json
|
||||
(temp 1.0 / top_p 0.95 / top_k 20).
|
||||
- **Thinking counts against max_tokens** (same as the vLLM era): a
|
||||
`max_tokens: 10` probe returns EMPTY content with the budget spent in the
|
||||
think block. Clients must budget for thinking or disable it.
|
||||
- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
|
||||
from the llm-fleet job; FreeToken has no /metrics endpoint). The target
|
||||
will read down until the scrape config is repointed or a metrics shim
|
||||
lands — same class as the retired HTML-probe targets
|
||||
(see [[Observability]]).
|
||||
|
||||
|
||||
## The shape
|
||||
|
||||
| Knob | Value | Why |
|
||||
|---|---|---|
|
||||
| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
|
||||
| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
|
||||
| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
|
||||
| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
|
||||
| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
|
||||
| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
|
||||
|
||||
KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
|
||||
2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
|
||||
not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
|
||||
case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
|
||||
native 262,144-token conversation + full budget. `models.yml`
|
||||
`contextWindow: 200704` stays (compacts early, never 400s).
|
||||
|
||||
## RAM math (why the root unit + memlock)
|
||||
|
||||
246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
|
||||
PLE ngram table in host RAM → the unit carries
|
||||
`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
|
||||
a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
|
||||
container's ~170 GiB MUST be down first; `systemctl stop frank-primary`
|
||||
frees it (measured: 207 GB free right after the stop).
|
||||
|
||||
## Resize live
|
||||
|
||||
```bash
|
||||
ft ctl cache --kv <tokens> # grows the KV pool LIVE (shrink needs restart)
|
||||
ft ctl cache --moe <rate> # same for the expert LRU
|
||||
ft ctl stats # pool occupancy, VRAM, throughput
|
||||
```
|
||||
|
||||
## Verification (the fleet's real-completion law)
|
||||
|
||||
1. `/health` 200 — necessary, not sufficient.
|
||||
2. Plain completion returns the asked-for string.
|
||||
3. Thinking request: `reasoning_content` non-empty AND `content` carries the
|
||||
answer (the models.yml `reasoningContentField: reasoning_content` flip
|
||||
rides on this).
|
||||
4. Vision probe: a solid-color PNG through an `image_url` part answers the
|
||||
exact color (the legacy top-level `images` field trap does not exist
|
||||
here — FreeToken is OpenAI-native).
|
||||
5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
|
||||
2026-09-26 stray-port false green).
|
||||
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
git checkout frank-primary-vllm-shapeG-20261003 -- \
|
||||
scripts/frank-primary.env scripts/serve-frank-primary.sh \
|
||||
scripts/frank-primary.service
|
||||
./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \
|
||||
&& git push && ./scripts/deploy.sh dankserver --service frank-primary
|
||||
```
|
||||
|
||||
The tcclaviger image and the FreeToken image coexist in dank's rootless
|
||||
storage; the deploy re-pulls the pinned digest if absent.
|
||||
|
||||
## Index
|
||||
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
|
||||
(rollback target; its 362-line env lore lives in the tag)
|
||||
- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
|
||||
- [[LLM/index]] — fleet index
|
||||
@@ -45,6 +45,98 @@ timestamp: 2026-09-30
|
||||
> because the fork compiles its own Qwen trunk and not its GLM one. See
|
||||
> [The measured profile](#the-measured-profile-and-the-real-wall).
|
||||
|
||||
## DFlash2 target-side port — the four laws and the KV ladder
|
||||
|
||||
Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder`
|
||||
/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot.
|
||||
|
||||
**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`):
|
||||
|
||||
1. **Patch the instance the runner calls.** `model_runner.py:425` calls
|
||||
`set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model —
|
||||
GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper;
|
||||
`self.language_model = Glm5NextForCausalLM`). Members placed only on the
|
||||
CausalLM fail the check on the conditional — same error text, wrong file.
|
||||
Patch BOTH wrappers.
|
||||
2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is
|
||||
`@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__`
|
||||
(including base-protocol members) with `getattr_static`, breaking on the
|
||||
first miss. Both wrappers need, as class attributes:
|
||||
`supports_eagle3: ClassVar[Literal[True]] = True`,
|
||||
`has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`.
|
||||
Verify in-image with stock typing — no boot needed.
|
||||
3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five
|
||||
Protocol-derived bases; adding `SupportsEagle3` anywhere dies with
|
||||
"Cannot create a consistent method resolution order" (sim-proven: the
|
||||
ordering rejects it in every base position). The attribute route avoids
|
||||
the MRO, so the two methods must be written by hand: the CausalLM
|
||||
unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional
|
||||
delegates to `self.language_model` (pixtral.py:498 pattern);
|
||||
`get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`,
|
||||
with the drafter config's `target_layer_ids` winning when present.
|
||||
4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call
|
||||
— GLM's loop carries the single-stream state entering the layer.
|
||||
|
||||
**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`):
|
||||
|
||||
1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
|
||||
2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads
|
||||
whole token rows) but CAN grow block_size when the max page is an exact
|
||||
multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to
|
||||
the `lcm` of all MLA-family pages first (patch 02).
|
||||
3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's
|
||||
divisible branch grew KpoolTailSpec's block; its kernel pins block ==
|
||||
index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded`
|
||||
instead of growing (patch 03).
|
||||
4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why
|
||||
GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs
|
||||
exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend
|
||||
`_get_kv_cache_groups_glm5_next` to partition
|
||||
HiddenStateCacheSpec + drafter attention specs; the drafter forms its
|
||||
own uniform group (per-group pages, NO unification → the MLA block
|
||||
stays at the engine-auto-set 2,432 from interface.py:918, kpool guard
|
||||
`block_size % 512 == 0` unstressed); layout returns a 10-tuple
|
||||
(+draft_group, +hidden_groups); the builder sizes ONE shared backing
|
||||
allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) =
|
||||
aliased pool + appended draft/hidden tensors (patch 04). After 04 the
|
||||
capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and
|
||||
the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL
|
||||
graphs on both TP ranks).
|
||||
5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter
|
||||
KV) + 5×hidden×2 (aux hidden cache) keyed on
|
||||
`speculative_config.method in {dflash, dspark, extract_hidden_states}`.
|
||||
NEVER key on `use_aux_hidden_state_outputs` — it lives on the
|
||||
gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan
|
||||
time, so the check silently fails and the reserve stays 13,600
|
||||
(patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 +
|
||||
5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.
|
||||
|
||||
**Verify cheaply before booting:** smoke-test the pure KV logic on CPU
|
||||
torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`,
|
||||
register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
|
||||
`from __future__ import annotations`, and run synthetic GLM+draft+hidden
|
||||
spec sets through the patched grouping/layout/builder functions. A 7-minute
|
||||
boot per hypothesis is the alternative.
|
||||
|
||||
**Window discipline:** stop + mask `frank-primary` FIRST and gate on
|
||||
MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
|
||||
reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
|
||||
host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
|
||||
to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once
|
||||
under pipefail via SIGPIPE before log-save, losing the capture. The abort
|
||||
path must restore prod (unmask + daemon-reload + reset-failed + start),
|
||||
never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts
|
||||
the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method":
|
||||
"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's
|
||||
echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH.
|
||||
Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter
|
||||
torch.compile is the frontier (~380 s).
|
||||
|
||||
**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s
|
||||
with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s
|
||||
prefill. Reconcile with a clean single-request measurement before trusting
|
||||
either.
|
||||
|
||||
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
|
||||
hardware — its conclusion was drawn from throughput arithmetic and is superseded
|
||||
by this page's kernel-time measurement),
|
||||
|
||||
@@ -110,6 +110,32 @@ retires the lucebox user quadlet to dated home trash
|
||||
truly free, relinks + enables + restarts the root unit, and waits up to
|
||||
1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).
|
||||
|
||||
### Preflight the GGUF without downloading it
|
||||
|
||||
GGUF metadata lives in the header and HF honours ranged GETs, so a few MB
|
||||
answers every sizing question before a 13 GB transfer
|
||||
(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64
|
||||
n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY
|
||||
= u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers
|
||||
forget). Walk arrays by byte size even for a summary: a naive parser
|
||||
desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read:
|
||||
`general.architecture`, `<arch>.block_count`, `<arch>.context_length`,
|
||||
`<arch>.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp`
|
||||
build carries `blk.<n>.nextn.*`; `--spec-type draft-mtp` then loads the
|
||||
head against the MAIN model's weights, and `--model-draft` would load the
|
||||
file twice), `<arch>.head_count_kv`, `<arch>.key_length`/`value_length`,
|
||||
`<arch>.full_attention_interval`, `<arch>.ssm.state_size`/`group_count`/
|
||||
`time_step_rank`, and for the projector `clip.projector_type` +
|
||||
`clip.has_vision_encoder`.
|
||||
|
||||
KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)):
|
||||
only full-attention layers take KV
|
||||
(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B,
|
||||
q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot
|
||||
buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the
|
||||
MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size`
|
||||
between slots — it does not multiply.
|
||||
|
||||
## Verification (2026-09-26 deploy)
|
||||
|
||||
- `/v1/models` reports `frank-primary`.
|
||||
|
||||
@@ -146,3 +146,70 @@ systemctl --user start frank-medium`) and stop/mask frank-primary. The
|
||||
_Filled in after the cutover: health, real completion, tool call, Open WebUI
|
||||
completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
|
||||
state._
|
||||
|
||||
## Dual-R9700 follow-up (2026-09-28 probe era)
|
||||
|
||||
After this single-card runbook was retired, the DS41 engine got a dual-R9700
|
||||
attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
|
||||
`unsupported model architecture 'deepseek41'`). Findings that outlive the
|
||||
probe:
|
||||
|
||||
**Budget both cards explicitly, or you waste half the box.** The engine
|
||||
sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
|
||||
every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
|
||||
looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
|
||||
cache of things still streaming from NVMe. Pin all four:
|
||||
|
||||
| env | card | meaning |
|
||||
|---|---|---|
|
||||
| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
|
||||
| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
|
||||
| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
|
||||
| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
|
||||
|
||||
Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
|
||||
card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
|
||||
|
||||
**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
|
||||
`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
|
||||
gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
|
||||
tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
|
||||
secondary=hip:1`. It is the only lever that materially raises resident
|
||||
expert count, and resident count is the entire bottleneck. Confirm every run
|
||||
with `expert ownership:` and `hybrid experts ready:` lines.
|
||||
|
||||
**The placement file must live under the mounted models dir.** JSON
|
||||
`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
|
||||
`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
|
||||
keeps what fits its budget and demotes the rest. Anywhere outside
|
||||
`/models/...` gives `cannot open expert placement` and a SILENT single-tier
|
||||
fallback — a silent-wrongness failure, not a crash.
|
||||
|
||||
**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
|
||||
exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
|
||||
is refused there. With ~22% of experts resident each token pulls ~187 cold
|
||||
experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
|
||||
the model is larger than the box's RAM and you are in page-cache thrash at
|
||||
the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
|
||||
with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
|
||||
decode, not a broken run.
|
||||
|
||||
**Harness traps, each of which produced a wrong number:** never `env -i` the
|
||||
podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
|
||||
PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
|
||||
before the container exists — pass container env with `-e`); never put a
|
||||
large prompt on curl's argv (`Argument list too long` at ~128 KB — use
|
||||
`--data-binary @file`); size prompts by growing the file until `wc -c`
|
||||
reaches the byte target AND assert depth against the engine's own
|
||||
`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
|
||||
that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
|
||||
30%); the server runs as the container's PID 1, so tear down with
|
||||
`podman rm -f`, never host `pkill`; drop page cache between arms
|
||||
(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
|
||||
every later one is a lie.
|
||||
|
||||
Read the placement log — it is the whole diagnosis
|
||||
(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
|
||||
hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
|
||||
… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
|
||||
mapped, so they do not evict the expert page cache).
|
||||
@@ -190,6 +190,70 @@ repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
|
||||
on :7770 (`--enable-prefix-caching` in the container argv, health 200,
|
||||
live completion answering).
|
||||
|
||||
## Authoring a new profile (the qualification pipeline)
|
||||
|
||||
Order matters at every step:
|
||||
|
||||
1. **Package manifest** `packages/models/<pkg>/package.json` — target shards
|
||||
+ aux artifacts with sha256. Verify every target's byte count AND sha256
|
||||
after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`).
|
||||
2. **Placement bootstrap** from GGUF headers: parse all shards' tensor
|
||||
tables (a KV-only shard1 is normal for split parts; each part carries its
|
||||
own tensor table). Per-expert full per layer =
|
||||
(`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices
|
||||
rank0 = full×416//640, rank1 = full−rank0. Hot counts =
|
||||
floor(measured_budget / per-expert total). **Anchor budgets to the
|
||||
MEASURED hot_bytes of the qualified sibling profile, never invented.**
|
||||
3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then
|
||||
`expert_budget.validate_manifest(m)` +
|
||||
`expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>)`
|
||||
(runtime/catalog are keyword-only). Declared hot_bytes must match
|
||||
computed or it raises.
|
||||
4. **PLE derive** (containerized `tools/prepare_ple.py` with
|
||||
`--expected-bytes/--expected-type/--expected-shape`); setup skips
|
||||
re-derivation when the file exists with matching size.
|
||||
5. **profile.env**: set values BEFORE `source dual-r9700/profile.env`
|
||||
(set-first-wins; an empty string does NOT survive `:=`).
|
||||
|
||||
The canonical bootstrap.json wrapper schema is exactly
|
||||
`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path,
|
||||
artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the
|
||||
wrapper `id` MUST equal the id in the profile.json placement descriptor
|
||||
(setup rejects mismatches: "does not match profile placement"). Never
|
||||
fabricate qualification metadata — use
|
||||
`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`.
|
||||
|
||||
GGUF parsing laws for the bootstrap: expert tensor names are
|
||||
`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is
|
||||
parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes
|
||||
`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte
|
||||
extents from consecutive OFFSETS, never a type-size table (mis-buckets tails
|
||||
silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536),
|
||||
54,400,261,120 B, an entire shard.
|
||||
|
||||
Setup/start: `./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image
|
||||
sha256:<digest> --local-image --reuse-from <sibling-model-dir>
|
||||
--accept-model-license` (aux artifacts reuse byte-identical files; F16 vs
|
||||
Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed
|
||||
placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises.
|
||||
The host-only doctor FAILs when another instance holds the cards (it wants
|
||||
3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second
|
||||
instance CANNOT coexist, so qualification requires a stop+mask window for
|
||||
the incumbent, then unmask+restart.
|
||||
|
||||
**Qualification gate (unqualified → qualified):** first start runs the
|
||||
workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning
|
||||
proof (`VmLck` ≈ PLE bytes in `/proc/<container-pid>/status` when
|
||||
RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If
|
||||
headroom is violated, lower hot counts via re-trim and re-derive
|
||||
MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook.
|
||||
|
||||
Remote-execution gotchas from the same windows: `nohup X &` inside an ssh
|
||||
command races ssh teardown — the script can die before writing a byte; run
|
||||
long remote scripts synchronously, or setsid + disown, or via a system
|
||||
unit. zsh login shells abort on unquoted `===` separators; grep globs need
|
||||
quoting through ssh+zsh layers.
|
||||
|
||||
## Ops
|
||||
|
||||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||||
|
||||
@@ -1,18 +1,17 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-03):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **shape E since
|
||||
2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
|
||||
conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
|
||||
decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
|
||||
(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
|
||||
(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
|
||||
mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
|
||||
`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
|
||||
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-05):
|
||||
**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
|
||||
image `frank-primary-freetoken:pr587-77a7201` built on the host from
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
|
||||
`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
|
||||
driving `podman run` directly, serving orcarouter
|
||||
/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
|
||||
pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
|
||||
`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
|
||||
decode slots, parsers AUTO (thinking → `reasoning_content` natively;
|
||||
measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
|
||||
runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw
|
||||
runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
@@ -34,12 +33,12 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
B2[FreeToken PR587 ROCm, local image
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8, no spec decode
|
||||
TP=2, expert offload automatic, 3 x 200704 ctx,
|
||||
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
|
||||
expert-precise-memory arena 7,522 slots, /root/.cache mounted]
|
||||
C[Qwen3.8-Flash-Next FP8
|
||||
1x R9700, experts in pinned host RAM + GPU LRU 10%,
|
||||
KV pool 299,968 tokens, 2 slots, parsers AUTO,
|
||||
TP=2 untested on ROCm; second R9700 idle]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -95,8 +94,22 @@ frank-small, port 7771]
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
|
||||
(branch rocm714_torch214) built on the host into the local image
|
||||
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||||
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
|
||||
(~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
|
||||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
|
||||
frank-primary-vllm-shapeG-20261003
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||||
@@ -394,6 +407,42 @@ frank-small, port 7771]
|
||||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||||
|
||||
## Cross-Cutting References (skill distillation, 2026-10-05)
|
||||
|
||||
One polished reference page per skill-cluster, distilled from the managed
|
||||
skill library into the vault (see [log](../log.md) for the wave plan):
|
||||
|
||||
- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
|
||||
of a loader; how the runtime is chosen by the file
|
||||
- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
|
||||
laddering, llama-bench discipline
|
||||
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
|
||||
laws, drafter placement
|
||||
- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
|
||||
heterogeneous-pair verdicts
|
||||
- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
|
||||
lever, and why free VRAM is not arena
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
|
||||
died, and interrogating a running fork's config
|
||||
- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
|
||||
versions/forks; torch.compile/dynamo breakage triage
|
||||
- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
|
||||
the window script laws
|
||||
- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
|
||||
endpoints, IBS attribution, prefix-cache routing
|
||||
- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
|
||||
research verification, second-endpoint bring-up
|
||||
- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
|
||||
verdicts, large-MoE recipe, compile-cache persistence
|
||||
- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
|
||||
proposals (the no-go verdict format)
|
||||
- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
|
||||
dependency traps
|
||||
|
||||
These sit alongside the operational pages under
|
||||
`../Operations/index.md` and `../Agent/index.md`.
|
||||
|
||||
## Related
|
||||
|
||||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
type: reference
|
||||
title: Container Host Reachability
|
||||
description: The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate.
|
||||
tags: [podman, netavark, networking, rootless, reachability, caddy]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Container Host Reachability
|
||||
|
||||
Two failure families on rootless podman + netavark hosts (Dankclaw), both
|
||||
reading as "service down" when the service is up and the *path* is wrong.
|
||||
Fleet source of truth: `serverAppConfigs-docs/Architecture/Container Network
|
||||
Namespaces.md` (commit `2e179b0`).
|
||||
|
||||
## Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly
|
||||
|
||||
A container cannot reach a `Network=host` service at the host's LAN IP.
|
||||
netavark DNATs **published** ports only: from a bridge container,
|
||||
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
|
||||
gateway `10.89.1.1` are both refused. This is what 502'd
|
||||
`hermes.franksdank.cloud` until the Caddyfile upstream became
|
||||
`host.containers.internal` on 2026-09-16 — its vhost comment had claimed the
|
||||
LAN IP was open from that bridge; it never was.
|
||||
|
||||
Probe that proves it in one command:
|
||||
|
||||
```bash
|
||||
ssh <host> "podman run --rm --network <net> --entrypoint sh \
|
||||
docker.io/library/busybox:latest -c \
|
||||
'for p in <target-port> <published-port>; do
|
||||
nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD
|
||||
done'"
|
||||
podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK'
|
||||
```
|
||||
|
||||
Published OK + target BAD ⇒ quirk confirmed. Note busybox `nc -z` reports any
|
||||
failure as failure — use it for bulk sweeps; use `/dev/tcp` (or `nc -v`) when
|
||||
refusal-vs-timeout matters.
|
||||
|
||||
Fix: point the service's host-facing URL at
|
||||
`http://host.containers.internal:<port>` in compose/quadlet env, redeploy via
|
||||
git (`deploy.sh <target> --service <svc>`), verify
|
||||
`podman exec <svc> printenv <URL_VAR>` and the connect log line (~40 s).
|
||||
**Do not publish the daemon port** to make the LAN IP work — publishing is
|
||||
exactly what makes other ports refuse.
|
||||
|
||||
## Quirk 2 — addressability is per netns, not per host
|
||||
|
||||
- `NetworkMode=host` → the container **is** the host netns: dials all of
|
||||
`127.0.0.1`, binds any free port.
|
||||
- bridge/named net → own netns; only published ports are reachable, via the
|
||||
network's **gateway IP**. Unpublished host listeners refuse **instantly
|
||||
(RST)** — no firewall rule, no stale DNAT, `nft`/`iptables -t nat` empty. Do
|
||||
not hunt for REJECT rules; probe.
|
||||
|
||||
Probe sequence (read-only):
|
||||
|
||||
```bash
|
||||
podman inspect <name> --format '{{.HostConfig.NetworkMode}}'
|
||||
sudo ss -ltnp 'sport = :<port>' # host netns only — why 0.0.0.0 can still be unreachable
|
||||
podman exec <name> cat /proc/net/route # default route = the gateway IP to dial
|
||||
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'
|
||||
```
|
||||
|
||||
Interpretation without blurring:
|
||||
|
||||
- **instant refusal** = nothing listening in a reachable namespace (or
|
||||
rootlessport rejected an unpublished port map).
|
||||
- **timeout** = no route to that address at all. `host.containers.internal`
|
||||
(`169.254.1.2`) is an alias on the **default `podman` net only**; containers
|
||||
on a named network time out against it.
|
||||
|
||||
Tool gotchas (podman 5.x, verified live):
|
||||
|
||||
- `podman network inspect <net> --format '{{json .containers}}'` errors
|
||||
(`can't evaluate field containers in type interface {}`) — list membership
|
||||
from the container side: `podman ps -a --format '{{.Names}} {{.Networks}}'`.
|
||||
- An **empty** `Networks` field = host network or shared namespace
|
||||
(`--network=container:<other>`), not "no network".
|
||||
- Quadlet `Network=<name>.network` yields the runtime name **`systemd-<name>`**
|
||||
— probe with `podman network ls`, never the `.network` filename.
|
||||
|
||||
Fix options when a bridge-net container must reach a host-netns listener:
|
||||
(a) move it to `Network=host`, or (b) publish the port — (b) makes the port
|
||||
reachable from every namespace **and the LAN**, an exposure change needing
|
||||
explicit operator approval, never a silent fix.
|
||||
|
||||
## Related trap — the pasta port gate (rootless published ports)
|
||||
|
||||
On hosts publishing ports through pasta, the port is bound in **userspace at
|
||||
container CREATE**, not when the app binds. A predecessor stopped moments
|
||||
earlier can still hold listener/TIME_WAIT state, so the replacement dies
|
||||
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
|
||||
seconds later** under `Restart=on-failure`. The failure looks like it failed,
|
||||
then looks fine, and the missing gate stays invisible because the retry masks
|
||||
it. Observed 2026-09-16 cutting `frank-large` from a root unit to a rootless
|
||||
Quadlet.
|
||||
|
||||
Gate procedure: stop the old unit AND the new one (a redeploy's running
|
||||
replacement is itself the port-holder — without that stop the gate deadlocks
|
||||
against itself), then wait for the port to be empty across **all socket
|
||||
states**:
|
||||
|
||||
```bash
|
||||
wait_port_free() {
|
||||
local port="$1" limit="${2:-90}" i
|
||||
for ((i = 1; i <= limit; i++)); do
|
||||
if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "FAIL: port $port still held after ${limit}s:" >&2
|
||||
ss -tan "sport = :$port" >&2 || true
|
||||
return 1
|
||||
}
|
||||
```
|
||||
|
||||
`sport` catches accepted-connection server sockets and TIME_WAIT residue that
|
||||
`ss -ltn` misses; fail loudly with the socket table attached. Verify the gate
|
||||
by re-running the same deploy against a running service (idempotency) and
|
||||
assert it in seam tests (`wait_port_free` present in the deploy handler).
|
||||
|
||||
## Related
|
||||
|
||||
- [[Quadlet Lifecycle]] — the unit layer these containers run under
|
||||
- [[Container Network Namespaces]] — fleet network namespace source of truth
|
||||
- [[Observability]] — the alerting paths that depend on reachability
|
||||
- [[Deploy State Triage]] — the deploy flow that installs these configs
|
||||
@@ -0,0 +1,331 @@
|
||||
---
|
||||
type: reference
|
||||
title: Fleet Audit and Posture
|
||||
description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work.
|
||||
tags: [audit, security, prometheus, caddy, monitoring, git]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Fleet Audit and Posture
|
||||
|
||||
## Monitoring gap audit
|
||||
|
||||
Use after any reboot, or whenever Prometheus targets look wrong. Two failure
|
||||
modes, only the first announces itself.
|
||||
|
||||
**1. A service that should exist and doesn't is invisible.** A missing
|
||||
exporter shows up as nothing — no failing target if the job was configured
|
||||
for a host that no longer serves, no alert because the series is absent. So
|
||||
check **existence, not health**:
|
||||
|
||||
```bash
|
||||
podman ps -a --format '{{.Names}} {{.Status}}' # includes dead ones
|
||||
ls ~/.config/containers/systemd/*.container # what should be boot-persistent
|
||||
systemctl --user is-enabled <name>.service # expect: generated
|
||||
```
|
||||
|
||||
A container in `ps -a` that is `Exited(…)` **and has no unit** is the classic
|
||||
case: rootless podman does not restart bare containers after the stop that
|
||||
shutdown performs, so a compose-only service dies at the first reboot and
|
||||
stays dead. If the compose file still names it, the repo's own rule is being
|
||||
violated — fix by writing the quadlet (mirror the same service on the other
|
||||
host), deleting the compose entry (one owner per container name), and making
|
||||
the deploy path reach it: planner service-set + path rule, dispatcher
|
||||
allowlist `<target>:<service>`, handler installing from `"$TARGET/..."` never
|
||||
a hardcoded peer's path.
|
||||
|
||||
**2. Classify every down target — do not report a count.**
|
||||
`targets?state=active` gives health **and** last error; two down targets can
|
||||
mean opposite things:
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| `connection refused` | service not listening → real outage (podman ps, logs, exit codes) |
|
||||
| `404` on `/metrics` | exporter path wrong or metrics disabled → config defect, service fine |
|
||||
| `unsupported Content-Type …` | scraping an HTML *page* as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter |
|
||||
| target absent | §1 |
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
|
||||
import json,sys
|
||||
for t in json.load(sys.stdin)['data']['activeTargets']:
|
||||
if t['health'] != 'up':
|
||||
print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
|
||||
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics # hand-probe before believing the scraper
|
||||
```
|
||||
|
||||
"7 of 11 up" without the four reasons is how a real outage hid behind three
|
||||
config defects on this fleet — the crash-looping GPU service was one of four
|
||||
failures reported as a number.
|
||||
|
||||
**3. Exposure posture while you are in there.** An exporter bound to the
|
||||
wildcard publishes an unauthenticated API on every interface. Expect exactly
|
||||
two listeners per host-exporter — loopback for host-local probes, the host's
|
||||
own LAN address for the Prometheus container: `ss -ltn | grep <port>` must
|
||||
show two explicit addresses, never `0.0.0.0`.
|
||||
|
||||
**4. Leave decisions that are not yours.** Fix mechanical drift (missing
|
||||
unit, wrong path, wildcard bind). Do NOT disable a probe to green a
|
||||
dashboard, or "fix" a crash-looping service whose posture depends on owner
|
||||
intent — capture the evidence (restart count, exit code, last error line) and
|
||||
hand it over intact.
|
||||
|
||||
## vhost proxy health probes
|
||||
|
||||
After deploying or auditing a Caddy/Anubis front door, verify **from a
|
||||
client**, not from the backend's localhost — that is where transport
|
||||
mismatches, auth walls, and dead upstreams show up.
|
||||
|
||||
```
|
||||
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/ping
|
||||
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/
|
||||
```
|
||||
|
||||
A status code is a diagnosis, not a failure:
|
||||
|
||||
- `200` on `/ping` from *arrs = alive; **auth state unknown** (`/ping`
|
||||
bypasses auth by design) — probe `/` to decide.
|
||||
- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404
|
||||
there = healthy).
|
||||
- `502` = Caddy could not reach/negotiate with the upstream: dead backend
|
||||
**or transport mismatch**. Probe the backend port directly from the proxy's
|
||||
network (`ssh <proxy-host> curl http://<backend>:<port>/…`). Direct 200 +
|
||||
vhost 502 ⇒ config bug, not an outage.
|
||||
- Login page to anonymous (grep body for `password`) = auth ON; instant
|
||||
redirect to `/transferlist/`-style routes = auth OFF = open door.
|
||||
|
||||
**Transport-mismatch mechanism:** `transport http {versions h2c 1.1}` makes
|
||||
Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it
|
||||
→ 502 per request while the same port answers 200 on plain HTTP/1.1. Fix =
|
||||
drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain
|
||||
HTTP off-loopback and needs a TLS transport. Match the transport to what the
|
||||
backend speaks.
|
||||
|
||||
**Shell trap when gating commits on a test suite:** `grep -c "^not ok"` exits
|
||||
**1 when zero mismatches** — inside an `&&` chain it silently kills
|
||||
everything after it while still printing `0`. Wrap:
|
||||
`(./tests/run.sh | grep -cE '^not ok' || true) && git commit …`.
|
||||
|
||||
## Security posture audit (behavioral)
|
||||
|
||||
Answers "what can the internet touch, and is everything else actually
|
||||
gated?" — **all findings must be behavioral probes**, not config reads.
|
||||
|
||||
**1. Inventory (source of truth).** `dankclaw/Caddyfile`: classify every
|
||||
vhost — guarded (`reverse_proxy anubis:8923`), exempt (direct upstream), or
|
||||
redirect-only. Public DNS: `dig +short +time=3 +tries=1 <name>.franksdank.cloud
|
||||
@1.1.1.1` per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by
|
||||
DNS absence.
|
||||
|
||||
**2. Guard chain (expect the challenge, not the app).** Per guarded vhost:
|
||||
`curl -sk https://<vhost>/` and grep the body for
|
||||
`anubis|challenge|proof-of-work` (~8 markers on a live challenge page). A 200
|
||||
with NO markers means the app is served directly — the guard is bypassed.
|
||||
|
||||
**3. Exempt vhosts — per-app gating probes.** OpenWebUI `/api/v1/users/me` →
|
||||
401. Gitea `/api/v1/user` → 401; repo visibility `/api/v1/repos/dank/<repo>`
|
||||
→ 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex `/web/`
|
||||
→ 302 (login is JS-rendered). Calibre-Web `/` → 302 to `/login`. Jellyfin
|
||||
`/Users` → 401. Prowlarr/Sonarr/Radarr `/api/v3/system/status`; Lidarr/Readarr
|
||||
`/api/v1/system/status` → 401 = gated, 200 = fully open. qBittorrent
|
||||
`/api/v2/torrents/info` → 403 without SID. Hermes/hermesadmin `/` → 302 to
|
||||
`/login`. IdP (`auth.`): `/` → 302 → `/if/flow/default-authentication-flow/`
|
||||
→ 200 (flow HTML is React; grep "password" finds nothing — the 200 on the
|
||||
flow URL is the proof).
|
||||
|
||||
**4. WAN port scan — from an EXTERNAL vantage.** LAN→WAN hairpin is
|
||||
unreliable; scan from gluetun's ProtonVPN egress:
|
||||
|
||||
```bash
|
||||
ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \
|
||||
8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \
|
||||
5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \
|
||||
>/dev/null 2>&1 && echo OPEN: \$p; done"'
|
||||
```
|
||||
|
||||
Confirm the egress is truly external first (`wget -qO- https://api.ipify.org`
|
||||
→ ProtonVPN IP, not Comcast). **Trap: plain-HTTP probes against TLS ports
|
||||
(443, 32400) are false negatives** — port open, protocol mismatch; re-probe
|
||||
with `wget -qO- https://IP[:port]/`. busybox wget has no `--secure` flag; it
|
||||
errors instantly and the `&&` chain silently skips the echo (a sub-second
|
||||
wall time on a "timeout" scan means the flag failed, not that ports closed).
|
||||
|
||||
**5. Residuals.** Grafana basic-auth residual: `curl -u admin:admin
|
||||
http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live
|
||||
(loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds:
|
||||
`sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no
|
||||
auth (OpenAI API, no token). Stale-record check: current WAN IP =
|
||||
`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress);
|
||||
compare against every public A record — a record at a different Comcast IP
|
||||
is stale.
|
||||
|
||||
**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`,
|
||||
`search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all
|
||||
verified) + `auth` (IdP, by design); everything else LAN-only. Open findings:
|
||||
LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback
|
||||
residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP);
|
||||
`plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin
|
||||
192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener
|
||||
(dead config, no exposure).
|
||||
|
||||
## Stash supersession audit before drop
|
||||
|
||||
Use when a repo (especially a live-deployment checkout where the working tree
|
||||
IS the deployment) has an old stash whose content may have been superseded by
|
||||
later commits, and dropping blindly risks losing unique facts.
|
||||
|
||||
1. **Inventory**: `git stash show --stat stash@{0}`.
|
||||
2. **Probe for unique content per file**: `git show "stash@{0}:<path>"` and
|
||||
diff against HEAD; count lines present only in the stash version. A file
|
||||
with zero unique-vs-HEAD lines is fully superseded — safe to drop.
|
||||
3. **Classify each unique-content hit**: *stale fact* (superseded by a later
|
||||
refactor, e.g. old model naming) → drop is correct; *live durable fact*
|
||||
(an operational invariant never re-recorded) → extract it into live
|
||||
docs/memory or flag to the user before dropping.
|
||||
4. **Archive before dropping** — zero loss: tag the stash commit, then drop:
|
||||
|
||||
```bash
|
||||
git tag archive/stash-<YYYY-MM-DD>-<base-branch>-wip "stash@{0}"
|
||||
git stash drop "stash@{0}"
|
||||
```
|
||||
|
||||
The tag pins the full stash object including its index parent; recoverable
|
||||
via `git show archive/stash-…` or `git checkout archive/stash-…^3 -- <path>`
|
||||
for the untracked-files third parent. Stashes are invisible in normal history
|
||||
and expire from reflogs (~90 days); tagging converts "drop" into a
|
||||
reversible archive at zero workflow cost.
|
||||
|
||||
## Whole-fleet audit (git/test/live/docs cross-check)
|
||||
|
||||
Deep-audit the homelab fleet — git/test state, live service health, and
|
||||
docs-vs-live cross-check. Run from the dev checkout
|
||||
(`/home/blank/Projects/serverAppConfig`).
|
||||
|
||||
**1. Probe the live fleet — self-identifying, ONE ssh per host.** Do NOT
|
||||
trust the display order of parallel SSH outputs: they can render crossed and
|
||||
you will attribute host A's output to host B (2026-09-21: a false host↔IP
|
||||
"conflict" dissolved by a single self-contained per-host probe). Put
|
||||
`hostname` + a distinctive marker + the LLM identity in the SAME command:
|
||||
|
||||
```
|
||||
for ip in 192.168.0.143 192.168.0.235; do
|
||||
ssh -o BatchMode=yes dank@$ip \
|
||||
'echo "host=$(hostname)"; \
|
||||
echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \
|
||||
echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo'
|
||||
done
|
||||
```
|
||||
|
||||
Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
|
||||
= AI stack + `frank-large` :7770 + `frank-small` :7771 (RTX 3090 + Strix
|
||||
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary`
|
||||
:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
|
||||
hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
|
||||
it is unaffected by IP drift.
|
||||
|
||||
**2. LLM endpoints — health + identity.** `curl /health` per endpoint, then
|
||||
`/v1/models` (first 120 chars) — a 200 with the right model `id`/`name` is
|
||||
the pass condition.
|
||||
|
||||
**3. Tests — the deploy completeness gate.** `./tests/run.sh` (bats) must
|
||||
end `ALL TESTS PASSED`; `tests/e2e/run.sh` runs Playwright against the LIVE
|
||||
fleet. **A single-test flake is normal**, not a regression: the plex crawl
|
||||
intermittently lands on `app.plex.tv`, the grafana SSO test occasionally
|
||||
times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just
|
||||
that spec (`npx playwright test --project=standard specs/<n>-<name>.spec.ts`)
|
||||
to classify before alarming. A `deploy.sh` e2e gate can fail on a flake
|
||||
while the change itself is fine — verify the actual change live, then re-run
|
||||
the suite.
|
||||
|
||||
**4. Repo state.** `git status --short --branch; git log --oneline -20;
|
||||
git stash list; git worktree list` (prune stale worktrees), and
|
||||
`du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a
|
||||
**shared checkout with parallel instances**: uncommitted peer WIP and
|
||||
stash-list churn are expected. Never commit or drop a peer's files. To
|
||||
deploy with a dirty tree: `git stash push -u -m "..." -- <peer paths>`,
|
||||
deploy, `git stash pop`. `deploy.sh` refuses a dirty tree.
|
||||
|
||||
**5. Prometheus — expected-down targets (not outages).** Query
|
||||
`/api/v1?query=up` from inside the container; the structurally red-by-
|
||||
construction jobs (`comfyui` HTML scrape, `services` up/down-as-scrape,
|
||||
`gitea` exporter off) are documented in [[Observability]] — anything else
|
||||
down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a
|
||||
same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is
|
||||
a *host bind*, not a published port, so the bridge-networked Prometheus
|
||||
container cannot reach it at the host's LAN IP — netavark DNATs published
|
||||
ports only. Scrape same-host services via
|
||||
`host.containers.internal:<port>`; verify the fix from inside the container
|
||||
before editing: `podman exec prometheus wget -qO-
|
||||
http://host.containers.internal:<port>/metrics`.
|
||||
|
||||
**6. Cross-check docs vs live.** Confirm the AGENTS.md host↔IP↔hardware↔
|
||||
stack mapping against step 1. The LLM endpoint table and `models.yml` are
|
||||
probed live and stay current; the prose host labels are the part that goes
|
||||
stale after an IP move. Report discrepancies; do not "fix" prose from a
|
||||
single crossed probe.
|
||||
|
||||
## Bulk-redact an identity from a docs + config tree
|
||||
|
||||
Removing a retired host / machine / project name from a repo that holds both
|
||||
documentation and live config. Evidence-first, surgical, reversible.
|
||||
|
||||
**1. Inventory before touching anything.** Count hits **per file** (the
|
||||
count decides whether this is a 5-minute job or a user-input decision),
|
||||
excluding logs/`*.db`/runtime state, and capture `git status --short` first
|
||||
— you must know which files carry in-flight peer WIP so you never stage
|
||||
them.
|
||||
|
||||
**2. Classify every hit:**
|
||||
|
||||
| Bucket | Test | Action |
|
||||
|---|---|---|
|
||||
| **Narrative** | prose describing the thing | Reword to the class ("the workstation", "a client box") |
|
||||
| **Load-bearing** | the identifier IS a lookup key, hostname match, or config value | **Keep.** Prove it: read the code that consumes it |
|
||||
| **Evidence** | quoted logs, transcripts, reports | Keep timestamps/lines; de-identify only the host field |
|
||||
|
||||
Load-bearing examples that look like prose but are not: `install.sh` doing
|
||||
`PKGS=("common.pkgs" "$HOST.pkgs")` with `HOST="$(hostname)"` (the manifest
|
||||
*filename* is the key); a `--host <name>` case list; a Wi-Fi SSID. Hardware
|
||||
model strings, driver versions, PCI addresses are subject matter, not
|
||||
identity. State the boundary in the commit message.
|
||||
|
||||
**3. Subdirectories: distill, don't rename or keep.** For a per-host doc
|
||||
tree (`docs/system/<host>/`), ask before acting — rename/trash/distill are
|
||||
genuinely different. When distilling: read every page, sort into
|
||||
still-true-and-machine-independent (fold into a general page) vs
|
||||
specific-to-the-retired-box (let go), then `mv` the directory to
|
||||
`.AGENTTRASH/YYYYMMDD-topic/` and repoint every index. Wikilinks resolve by
|
||||
filename anywhere in the tree, so deleting a page can break a link in a
|
||||
file you never grepped — run the link checker.
|
||||
|
||||
**4. Execute with count assertions, never blind sed.** Per-file mapping of
|
||||
`(path, old_exact_string, new_string, expected_count)`; abort the entry if
|
||||
the count differs; exact strings, not regex. **Do not hand this to a
|
||||
subagent** — observed: dropped list markers so bullets stopped rendering, a
|
||||
numbered list with two `3.` items, "corrected" paths to nonexistent
|
||||
directories, a deleted sentence where a reword was required. If a subagent
|
||||
already did it: `git checkout --` its touched paths (excluding the WIP
|
||||
files from step 1) and redo it.
|
||||
|
||||
**5. Audit:** `grep -rniE '<name>|<its-ip>' <scope>` (expect nothing), then
|
||||
`git diff -U0 -- <scope> | grep -E '^\+[^+]'` and **read every added line**
|
||||
— the diff is the step that catches damage; the grep only proves a string
|
||||
is gone. Then the repo's own linter.
|
||||
|
||||
**6. Commit scoped** — only your own paths, never `git add -A`. If the
|
||||
checkout IS the live deployment, committing on the current branch is usually
|
||||
correct; a branch switch can swap live config out from under a running
|
||||
process. Push/PR only when asked.
|
||||
|
||||
**7. Report the residue honestly** — what still carries the name and why:
|
||||
excluded scopes, load-bearing config, git history, and the cost of rewriting
|
||||
history (force-push + re-clones). "Purged from the working tree" is a true
|
||||
claim; "purged" alone is not.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — the monitored stack
|
||||
- [[Stale Signal Audit]] — the down-target decision table
|
||||
- [[Observability Provisioning]] — adding scrape targets and rules
|
||||
- [[Homepage Status Probes]] — the status page that renders probe states
|
||||
- [[Deploy State Triage]] — the deploy plumbing the audits exercise
|
||||
@@ -0,0 +1,272 @@
|
||||
---
|
||||
type: reference
|
||||
title: Fleet Ops Playbook
|
||||
description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host.
|
||||
tags: [operations, incidents, gpu, systemd, wol, forensics]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Fleet Ops Playbook
|
||||
|
||||
## Endpoint down, connection refused — GPU holder reclaim
|
||||
|
||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
||||
`frank-small` (dankclaw, root unit), `frank-large` (dankclaw, rootless
|
||||
Quadlet).
|
||||
|
||||
**1. Classify the stop before touching anything**
|
||||
|
||||
```bash
|
||||
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
|
||||
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
||||
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
|
||||
```
|
||||
|
||||
`inactive` + `enabled` + unmasked + `NRestarts=0` → **intentional stop**,
|
||||
never restored. No crash; do not look for segfaults. `NRestarts>0` /
|
||||
`activating` → different path; read the journal first.
|
||||
|
||||
**2. Are the GPUs actually free?** A check, never an assumption:
|
||||
|
||||
```bash
|
||||
rocm-smi --showmeminfo vram | grep Used # idle baseline ~60 MiB/card
|
||||
ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
|
||||
```
|
||||
|
||||
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
|
||||
prod unit → get the holder, and compare its bytes against the service's
|
||||
requirement before concluding a start would even fit (frank-medium 35:30
|
||||
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
|
||||
restart on allocation, not config).
|
||||
|
||||
**3. Identify the holder and prove it is abandoned**
|
||||
|
||||
```bash
|
||||
ps -o pid,ppid,user,lstart,etime,comm -p <pid>
|
||||
sudo readlink /proc/<pid>/cwd /proc/<pid>/exe
|
||||
tr '\0' ' ' < /proc/<pid>/cmdline # port, ctx, split
|
||||
sudo ls -l /proc/<pid>/fd # where stdout/stderr go
|
||||
sudo ss -tnp | grep <port>
|
||||
awk '{print $14+$15}' /proc/<pid>/stat; sleep 10; awk '{print $14+$15}' /proc/<pid>/stat
|
||||
```
|
||||
|
||||
Abandoned **only if all** hold: utime+stime **identical** across the 10 s
|
||||
sample (frozen CPU, not just low); **zero** established connections; bind
|
||||
scope `127.0.0.1` (loopback scratch, invisible to endpoint monitors); log
|
||||
file (fd 1/2 target) last written long ago, last lines showing success not a
|
||||
hang. Active CPU or live connections → someone's running session. Stop and
|
||||
ask.
|
||||
|
||||
**4. Reclaim**
|
||||
|
||||
```bash
|
||||
cp -a <logfile> <logfile>.preserved-YYYYMMDD # evidence outlives the kill
|
||||
kill <pid> # SIGTERM first; exits <1 s
|
||||
rocm-smi --showmeminfo vram | grep Used # expect ~60 MiB baseline
|
||||
```
|
||||
|
||||
Never `pkill -f` a pattern present in your own command line.
|
||||
|
||||
**5. Restore and verify — health 200 alone is NOT proof**
|
||||
|
||||
```bash
|
||||
sudo systemctl start <svc>
|
||||
for i in $(seq 1 60); do curl -s -m 5 -o /dev/null -w '%{http_code}\n' \
|
||||
http://127.0.0.1:<port>/health; sleep 10; done # 503 while loading -> 200
|
||||
sudo tr '\0' ' ' < /proc/<MainPID>/cmdline | grep -oE '\-\-(tensor-split|split-mode|ctx-size|port|spec-type|parallel|flash-attn)'
|
||||
```
|
||||
|
||||
Prove with a **real completion** from off-host — content, reasoning, spec
|
||||
counters:
|
||||
|
||||
```bash
|
||||
curl -sS -m 180 http://<host>:<port>/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"<alias>","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":64,"temperature":0}'
|
||||
# choices[0].message.content / .reasoning_content / timings.draft_n, draft_n_accepted
|
||||
```
|
||||
|
||||
Record the runtime fingerprint (`<bin> --version` → `b<build>, commit <sha>`)
|
||||
and confirm it is the pinned runtime. Append a dated entry to
|
||||
`serverAppConfigs-docs/log.md`, run `./tests/run.sh`, commit, push (origin is
|
||||
the local Gitea on Dankclaw — push allowed).
|
||||
|
||||
**Gotchas seen here:** dankserver's login shell is ZSH — never start an
|
||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||
timeout. First request after a halogen (`frank-large`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-large`:
|
||||
read logs **without** sudo (per-user storage). If the restore never happened,
|
||||
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||
|
||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||
|
||||
Any scratch/experimental vLLM container on a host that also runs a production
|
||||
LLM service (`frank-primary`, `frank-large`, `frank-small`) MUST stop the
|
||||
production unit first, and the window script must verify it. Two independent
|
||||
failure modes if skipped:
|
||||
|
||||
1. **Expert-offload planner dies on host RAM.**
|
||||
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
||||
plan time. A production rank worker holds tens of GiB of unevictable
|
||||
pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod
|
||||
boot sees ~86 GiB instead of ~239 and dies at plan time:
|
||||
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
||||
Looks like a planner bug; is pure RAM contention. Signature: container
|
||||
exits within ~60 s, after `Resolved architecture: …` but before worker
|
||||
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
|
||||
gun.
|
||||
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
|
||||
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
|
||||
only once the prod unit is down.
|
||||
|
||||
Window skeleton (dankserver):
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-primary.service
|
||||
sudo systemctl mask --runtime frank-primary.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
|
||||
sleep 3
|
||||
podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; }
|
||||
# ... launch scratch container, watch logs ...
|
||||
podman rm -f <scratch> # stuck vLLM ignores SIGTERM
|
||||
sudo systemctl unmask --runtime frank-primary.service
|
||||
sudo systemctl reset-failed frank-primary.service
|
||||
sudo systemctl start frank-primary.service
|
||||
# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink
|
||||
```
|
||||
|
||||
The deploy restores the unit after bench windows, so stop is sufficient
|
||||
there; `mask --runtime` is for scratch boots that must survive a stray
|
||||
restart. Always end a window by re-checking the conjunction before
|
||||
reporting — if a restore after failure doesn't happen automatically, prod
|
||||
stays DOWN. See the repo's benchmarking policy (AGENTS.md) and
|
||||
[[LLM Bench Windows and Measurement]].
|
||||
|
||||
## Iterating on a fleet service before commit
|
||||
|
||||
The repo's contract is git → `deploy.sh` → host, but live testing needs the
|
||||
new file on the host first, and the naive way (copy into the host checkout)
|
||||
blocks the very deploy that would install it. The loop:
|
||||
|
||||
1. Edit locally; run `./tests/run.sh`.
|
||||
2. Ship working files **for testing**:
|
||||
`cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>'`
|
||||
(then `chmod +x` if executed — `cat >`/scp drop the exec bit and a
|
||||
container entrypoint fails status 126, "found but not executable").
|
||||
3. Test against the live service; iterate.
|
||||
4. **Before committing**, prove the host copy is byte-identical to what you
|
||||
are about to commit (`sha256sum` both ends) — otherwise step 6 loses work.
|
||||
5. Commit and push.
|
||||
6. Clean the host checkout so the deploy can pull: untracked hand copies →
|
||||
`mv` to `/tmp/handcopy/`; modified tracked files → `git checkout -- <paths>`
|
||||
(safe only because of step 4); confirm `git status --short` prints nothing.
|
||||
7. `./scripts/deploy.sh <target> --service <svc>`.
|
||||
|
||||
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
|
||||
working tree files would be overwritten by merge" — exactly the file you
|
||||
shipped); `git checkout --` silently eats uncommitted work without step 4; a
|
||||
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
|
||||
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
|
||||
container restart with no deploy; root-owned files from a container write
|
||||
need `chown` back before git operations; and **a restart is not a
|
||||
verification** — verify the *specific* claim (new code path's own output,
|
||||
file checksum on the host, the API serving the resource).
|
||||
|
||||
Why not keep a scratch copy on the host? The deployed artifact must be the
|
||||
committed one. Probes may live anywhere; runtime state lives in git.
|
||||
|
||||
## Incident window reconstruction
|
||||
|
||||
Use when the question is **"what was running when it broke?"** and the answer
|
||||
must come from artifacts, not recollection. Three independent clocks,
|
||||
cross-checked:
|
||||
|
||||
1. **Per-run duration** — scripts wrap commands in `time` → `real 4m19.129s`.
|
||||
2. **Log mtime** — last write of the log.
|
||||
3. **Journal first-failure instant** — first error line from the affected
|
||||
service.
|
||||
|
||||
`start ≈ mtime − real` for the last run in a log; each run nests inside the
|
||||
previous one's window; the run whose `[start, end]` contains the journal's
|
||||
first-failure instant is the answer.
|
||||
|
||||
```bash
|
||||
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
|
||||
ssh <host> 'grep -ah "^real" <logdir>/*.log'
|
||||
ssh <host> 'grep -aE "^####|^\+ /path/to/binary" <logdir>/<name>.log | sed "s/\(.\{130\}\)/\1\n /g"'
|
||||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
|
||||
```
|
||||
|
||||
Traps (all hit for real): `/tmp` does not survive a reboot — launch
|
||||
experiment scripts from a persistent path (`/home/dank/models/logs/<tag>.log`
|
||||
survived while `/tmp/x4*.sh` died); a log's mtime is not proof a run finished
|
||||
(truncation with the fd still held keeps growing); long command lines are
|
||||
display-truncated — wrap before reading or you report parameters never used;
|
||||
off-by-one filenames (`x4b.log` vs `x4b-experiment.log`) silently skip
|
||||
evidence — list the directory and match exactly; **overlap is a hypothesis,
|
||||
not an observation** unless a sampler proves it; do not invent the cause —
|
||||
reconstruct the *window* and keep the *mechanism* separate (`journalctl -k |
|
||||
grep -i oom`); publish "trigger not established" when the proof is absent.
|
||||
|
||||
Fleet-specific: container healthchecks shell out to the **podman CLI**, so
|
||||
the first `invalid internal status … could not find any running process` in
|
||||
the user journal is an early, per-minute indicator of rootless-podman state
|
||||
death — it timestamps the incident to within one healthcheck interval (how
|
||||
the 18:58:30 Dankclaw break was pinned to a bench still loading).
|
||||
|
||||
Worked example (2026-09-16, Dankclaw): log mtimes x4e-run 18:58:29 /
|
||||
x4f-run 18:59:56; durations E1 `real 4m53.946s`, F `real 5m24.785s` (exit 143
|
||||
= killed); journal first failure 18:58:30 ⇒ **F** was in flight (launched
|
||||
18:54:31, inside its window).
|
||||
|
||||
## Wake-on-LAN for a sleeping fleet host
|
||||
|
||||
A fleet host is asleep (100% ping loss; `ip neigh show <ip>` shows STALE or
|
||||
FAILED) and you need it up — typically the FrankBook-Pro Mac mini
|
||||
(192.168.0.112, **MAC `de:6b:6d:c1:52:23`**, sleeps and breaks the e2e
|
||||
`11-admin` spec).
|
||||
|
||||
- **Find the MAC**: `ip neigh show <ip>` — even a STALE entry carries it;
|
||||
fallback Pi-hole/DHCP leases or fleet docs.
|
||||
- **Send the packet**: `6×0xFF + 16×MAC` = 102 bytes, UDP to port **9** (and
|
||||
7), broadcast. The dev workstation has **two interfaces on 192.168.0.0/24**
|
||||
(Wi-Fi + wired USB-Ethernet, default route metric 100) — a plain
|
||||
`sendto("255.255.255.255", 9)` leaves only the default-route interface.
|
||||
Bind explicitly to each interface's local IP and target the subnet
|
||||
broadcast `192.168.0.255` as well as `255.255.255.255`; burst ~15 packets
|
||||
per interface ~0.2 s apart. `wakeonlan` is not installed; use a raw UDP
|
||||
socket with `SO_BROADCAST=1`:
|
||||
|
||||
```python
|
||||
import socket, time
|
||||
MAC = "de:6b:6d:c1:52:23"
|
||||
mac = bytes(int(x,16) for x in MAC.split(":"))
|
||||
pkt = b"\xff"*6 + mac*16 # 102 bytes
|
||||
for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"),
|
||||
("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]:
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
|
||||
s.bind((local, 0))
|
||||
for _ in range(15):
|
||||
s.sendto(pkt,(dest,9)); s.sendto(pkt,(dest,7)); time.sleep(0.2)
|
||||
s.close()
|
||||
```
|
||||
|
||||
- **Poll for the wake**: a Mac mini takes **30–90 s**; `ping -c 1 -W 1` every
|
||||
5 s for ~2–3 min before concluding failure.
|
||||
- **If it does not wake** (the common case): macOS honors WoL only when BOTH
|
||||
are enabled on the mini — **System Settings → Battery → Options → "Wake
|
||||
for network access"** = ON, and the active port's **"Enable Wake for LAN"**
|
||||
(Network → [interface] → Details). If either is off, no packet from any
|
||||
source wakes it, and you cannot change it while asleep. A full power-off
|
||||
(smart plug cycled) ignores WoL entirely. Ask the user to toggle the
|
||||
setting, then re-fire.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the bench windows this playbook arms
|
||||
- [[Remote Job Runner]] — supervising the long jobs that create these windows
|
||||
- [[Container Host Reachability]] — the pasta port gate at container create
|
||||
- [[Quadlet Lifecycle]] — the unit layer behind stop/mask/start
|
||||
- [[Model Swap and Rollback]] — swap-time incidents and rollbacks
|
||||
@@ -0,0 +1,204 @@
|
||||
---
|
||||
type: reference
|
||||
title: Payload and Disk Reclaim
|
||||
description: Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index, prove nothing live references the payload, stage to trash with fingerprints, purge only on approval; plus the purge-and-refresh protocol with fetch verification.
|
||||
tags: [operations, disk, podman, payloads, huggingface, trash]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Payload and Disk Reclaim
|
||||
|
||||
Standing procedure for "clean up old models" / "disk is full" on a
|
||||
serverAppConfigs host. Order matters: **measure, prove nothing live uses it,
|
||||
stage, ask.** Staging frees nothing until the user approves the purge — say so
|
||||
up front or the request reads as unfulfilled.
|
||||
|
||||
## 1. Measure the whole filesystem, not the category asked about
|
||||
|
||||
```sh
|
||||
du -sh /home/dank/models /home/dank/.local /home/dank/.cache \
|
||||
/home/dank/comfyui-data /home/dank/GitPackages 2>/dev/null
|
||||
df -h /home/dank | tail -1
|
||||
podman system df # reclaimable image space
|
||||
du -sh ~/.local/share/containers # where rootless storage physically is
|
||||
```
|
||||
|
||||
The answer is usually NOT the category the user named. On Dankclaw
|
||||
(2026-09-28) retired models were 510G, but podman images were **279G
|
||||
reclaimable (95%)** and uv+pip caches **73G** — 352G of regenerable garbage,
|
||||
bigger than the models. Lead with that finding; it changes the decision.
|
||||
|
||||
Check for sparse files before claiming a saving:
|
||||
|
||||
```sh
|
||||
stat -c 'apparent=%s blocks=%b' <file>; du -sh --apparent-size <file>; du -sh <file>
|
||||
```
|
||||
|
||||
A 3.3G "log" (`logs/x4e-run.log`) was **4K real** (8 blocks × 512) — deleting
|
||||
it frees nothing. `du` vs `du --apparent-size` finds these instantly.
|
||||
|
||||
## 2. Classify against the runbook index
|
||||
|
||||
Every payload directory maps to a runtime in the AGENTS.md retired-runbook
|
||||
table. Three buckets:
|
||||
|
||||
- **live** — referenced by a running service's env/unit.
|
||||
- **rollback** — named as a rollback target in a runbook. ASK before
|
||||
touching: they are the difference between a one-minute flip and a 30G
|
||||
re-download.
|
||||
- **retired** — maps to a retired runtime. Safe candidates.
|
||||
|
||||
Trap: a directory can hold **both** a dead target and a live drafter
|
||||
(frank-large's DFlash2 drafter lives beside the previous target). Move the
|
||||
single file, never the directory, and say so in the manifest.
|
||||
|
||||
## 3. The live-reference check — before moving anything
|
||||
|
||||
```sh
|
||||
for d in <candidate dirs>; do
|
||||
grep -rl "$d" --include="*.sh" --include="*.env" --include="*.service" \
|
||||
--include="*.container" --include="*.yaml" . | grep -v '^./old/'
|
||||
done
|
||||
```
|
||||
|
||||
Then **read every hit**; do not count them. Most are benign (comments,
|
||||
retired fetch-script destinations, `retire_*_quadlet` helpers that move a
|
||||
quadlet not a model) — but on Dankclaw this check found a live bug:
|
||||
`serve-frank-small.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
|
||||
payload retired a week earlier. The env file always overrode it, so it was
|
||||
invisible; a missing env file would have produced a "missing asset" naming a
|
||||
model that no longer exists.
|
||||
|
||||
**Durable lesson:** `:`-defaults in a serve script are never exercised in
|
||||
production, so they rot silently. When reclaiming a payload, grep serve
|
||||
scripts too — the fallback is a live reference in all but name.
|
||||
|
||||
Also check service **env files** (`scripts/*.env`) for `MODEL_DIR`/`MODEL`
|
||||
paths, not just unit `Volume=` lines: a payload can be mounted by path and
|
||||
named by env. `models/logs` and small tool dirs are not cache — logs are
|
||||
evidence, keep them.
|
||||
|
||||
## 4. Stage, never delete
|
||||
|
||||
```sh
|
||||
STAGE=/home/dank/.AGENTTRASH/$(date +%Y%m%d)-<topic>
|
||||
mkdir -p "$STAGE"
|
||||
mv <path> "$STAGE"/
|
||||
```
|
||||
|
||||
Same filesystem ⇒ instant and recoverable — **and reclaims no space**; say so
|
||||
rather than implying space was freed. Write `MANIFEST.md` into the stage dir:
|
||||
per item — path, size, owning runtime, when retired, the `mv` command to
|
||||
restore. Record what you **deliberately kept** and why; that is what a future
|
||||
reader needs. Always check for pre-existing staged trash
|
||||
(`du -sh ~/.AGENTTRASH/*`): 118G from a 2026-09-21 cleanup sat un-purged;
|
||||
report the running total, not just the new batch. Root-owned leftovers need
|
||||
`sudo`, and check `swapon --show` first (a trashed swapfile must not be
|
||||
active).
|
||||
|
||||
## 5. Verify, then report — and do not purge
|
||||
|
||||
```sh
|
||||
systemctl is-active frank-large frank-small
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
|
||||
```
|
||||
|
||||
Report: staged items and sizes, what was kept and why, what is **still not
|
||||
freed**, remaining candidates untouched. Purge only on explicit approval.
|
||||
|
||||
## Repo-flow gotcha: do NOT push before you deploy
|
||||
|
||||
`scripts/deploy.sh` sets `BASE_REVISION=$(git rev-parse origin/main)`. If you
|
||||
pushed separately, the diff is empty and the planner reports "changed
|
||||
services: (none)" — looks like a planner bug, is not one. The tool is
|
||||
designed for the **unpushed** state: edit → test → `deploy.sh` (it pushes,
|
||||
then dispatches). If you already pushed, pull on the host instead, and reason
|
||||
about whether a restart is actually owed. Any serve-script/env change needs
|
||||
`./tests/run.sh` before commit; a `:`-default change is usually **inert**
|
||||
(env overrides it) and owes no restart — but say so explicitly.
|
||||
|
||||
# Purge and refresh protocol
|
||||
|
||||
When the task is clearing/reclaiming/**refreshing** model caches on a serving
|
||||
host, add these steps.
|
||||
|
||||
## Fingerprint before deleting
|
||||
|
||||
`ls -l --block-size=1 <dir>`; for LLM payloads compare each file against the
|
||||
Hub's declared size. Fingerprints are the only way to answer "did the refresh
|
||||
change anything" afterwards — **a refresh is also an update check**: if
|
||||
upstream never changed, the bytes come back identical, which is itself the
|
||||
finding. Sidecars are versioned **in place**: equal main-checkpoint sizes do
|
||||
not prove the repo is unchanged, and a stale sidecar is invisible from its
|
||||
name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560 while the main
|
||||
checkpoint returned byte-identical). If a file is irreplaceable upstream
|
||||
(delisted revision, gated repo), say so before deleting.
|
||||
|
||||
## Confirm re-fetchability first
|
||||
|
||||
Query the Hub anonymously; a gated repo 403s even with the host's token:
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
|
||||
| python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
|
||||
```
|
||||
|
||||
An error/empty result means it is NOT safe to delete. Filter by **quant
|
||||
name**, never `endswith('.gguf')` — mradermacher ships large quants as
|
||||
`.partNofM` fragments that filter hides.
|
||||
|
||||
## Refresh per engine — know the recovery path BEFORE deleting
|
||||
|
||||
- **frank-large (halogen)**: the engine fetches its own payload via
|
||||
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
|
||||
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
|
||||
podman refuses a mount whose source is missing
|
||||
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
|
||||
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB
|
||||
at 130–240 MB/s ≈ 10–20 min, then load.
|
||||
- **llama.cpp services**: no self-fetch; the serve script refuses to start
|
||||
without weights. Pull yourself and verify:
|
||||
|
||||
```bash
|
||||
curl -fL -C - --retry 10 -o "$DEST/<file>" \
|
||||
"https://huggingface.co/<repo>/resolve/main/<file>" # -f is mandatory
|
||||
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to LFS oid + GGUF magic
|
||||
```
|
||||
|
||||
`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — a
|
||||
gated repo once produced three 153-byte shards with identical sha256s and a
|
||||
cheerful DONE line for 104 GiB. Always `-fL -C -`, compare byte counts to the
|
||||
Hub's declared sizes, compare sha256 to the LFS oid. Assemble `.partNofM`
|
||||
fragments with `cat part1 part2 part3 > name.gguf` in numeric order (per-part
|
||||
hashes verify the whole — concatenation is deterministic). Public repos need
|
||||
no token: omit `-H "Authorization: Bearer …"` from argv entirely (it is
|
||||
world-readable in `/proc/<pid>/cmdline`).
|
||||
|
||||
## Restore, then verify behaviour — not health alone
|
||||
|
||||
Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`),
|
||||
`reset-failed`, `start` (root units need `sudo`). Watch the engine's own
|
||||
fetch/load lines, then `/health`, `/v1/models`, and a **real completion** on
|
||||
every restored endpoint. Read the engine's memory ledger (kv pool / weights /
|
||||
blocks left) rather than assuming the config still fits. Sweep the vhosts too
|
||||
(`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki` via
|
||||
`curl -sk --resolve <host>:443:192.168.0.143`) — a purge touches state the
|
||||
whole host shares.
|
||||
|
||||
## Things that bite
|
||||
|
||||
- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark** —
|
||||
the A* run commits ~116 GiB and leaves ~10 GiB available; cache churn risks
|
||||
the OOM/state failures this fleet has already seen.
|
||||
- **Reboot is the repair of last resort** if rootless podman's pause process
|
||||
dies mid-work (`invalid internal status … could not find any running
|
||||
process`); `podman system migrate` panics in 5.7.0.
|
||||
- A same-filesystem trash reclaims no space; root-owned leftovers need
|
||||
`sudo`; never `rm` without explicit permission.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Model Swap and Rollback]] — the runbook index this classification reads
|
||||
- [[Quadlet Lifecycle]] — restoring the unit after a payload refresh
|
||||
- [[Remote Job Runner]] — supervising long re-fetches
|
||||
- [[Deploy State Triage]] — the push-before-deploy trap in context
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
type: reference
|
||||
title: Remote Job Runner
|
||||
description: Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally, the script, its invariants, verification, and reporting discipline.
|
||||
tags: [operations, ssh, jobs, sentinel, detached]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Remote Job Runner
|
||||
|
||||
Use for any long-running job on a remote host (benchmarks, model loads,
|
||||
builds, sweeps) whose result must survive the session. Also whenever you catch
|
||||
yourself writing `ssh host 'tail -f log | sed …'` as a watcher.
|
||||
|
||||
## Why the obvious approach fails structurally
|
||||
|
||||
`ssh host 'tail -f log | grep/sed …'` as a background watcher fails in ways
|
||||
that all LOOK like "the job died":
|
||||
|
||||
- the session broker **aborts long-lived ssh** — the watcher vanishes and the
|
||||
result is lost;
|
||||
- a `ready` regex can match the log's own **header** on the first line — the
|
||||
supervisor reports success instantly while the job is still running;
|
||||
- a `sed -n '/PAT/,$p'` filter **exits at the first match**, ending the pipe;
|
||||
- output goes into the ssh stream, so an aborted session loses everything.
|
||||
|
||||
## The mechanism: detach on the HOST, poll a sentinel file
|
||||
|
||||
Completion is a **`.done` sentinel touched last**, not a log pattern. Status
|
||||
is then one fast independent command with no long-lived process to lose.
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# job.sh {start|status|wait|out|tail|watch} ... Jobs live in $JOBS.
|
||||
set -uo pipefail
|
||||
JOBS=/home/dank/jobs
|
||||
mkdir -p "$JOBS"
|
||||
|
||||
case "${1:-}" in
|
||||
start)
|
||||
NAME="$2"; shift 2
|
||||
[ "${1:-}" = "--" ] && shift
|
||||
rm -f "$JOBS/$NAME.done" "$JOBS/$NAME.rc"
|
||||
# setsid detaches from this session's process group: the critical bit.
|
||||
setsid nohup bash -c '
|
||||
NAME="$1"; shift
|
||||
"$@" > "'"$JOBS"'/$NAME.out" 2>&1
|
||||
echo $? > "'"$JOBS"'/$NAME.rc"
|
||||
touch "'"$JOBS"'/$NAME.done"
|
||||
' _ "$NAME" "$@" < /dev/null > /dev/null 2>&1 &
|
||||
echo "started $NAME"
|
||||
;;
|
||||
status)
|
||||
# Glob *.out (not *.done) or running jobs are invisible.
|
||||
names="${2:-$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)}"
|
||||
for n in $names; do
|
||||
if [ -f "$JOBS/$n.done" ]; then
|
||||
printf ' %-18s DONE rc=%s %s lines\n' "$n" \
|
||||
"$(cat "$JOBS/$n.rc" 2>/dev/null)" \
|
||||
"$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
|
||||
elif [ -f "$JOBS/$n.out" ]; then
|
||||
printf ' %-18s RUN %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
|
||||
else
|
||||
printf ' %-18s PENDING\n' "$n"
|
||||
fi
|
||||
done
|
||||
;;
|
||||
wait)
|
||||
NAME="$2"; LIMIT="${3:-3600}"; waited=0
|
||||
while [ ! -f "$JOBS/$NAME.done" ] && [ "$waited" -lt "$LIMIT" ]; do
|
||||
sleep 5; waited=$((waited+5))
|
||||
done
|
||||
[ -f "$JOBS/$NAME.done" ] \
|
||||
&& echo "DONE $NAME rc=$(cat "$JOBS/$NAME.rc" 2>/dev/null)" \
|
||||
|| echo "TIMEOUT $NAME after ${waited}s"
|
||||
tail -"${4:-25}" "$JOBS/$NAME.out" 2>/dev/null
|
||||
;;
|
||||
out) NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;;
|
||||
tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;;
|
||||
watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;;
|
||||
*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;;
|
||||
esac
|
||||
```
|
||||
|
||||
```bash
|
||||
job.sh start NAME -- <command> # launch detached
|
||||
job.sh status [NAME] # one-shot state
|
||||
job.sh wait NAME 900 25 # block, then print the tail
|
||||
job.sh out NAME 60 # retrieve any past result
|
||||
job.sh tail [NAME] # live follow
|
||||
job.sh watch # refreshing status table
|
||||
```
|
||||
|
||||
## Invariants
|
||||
|
||||
- **`.done` is touched LAST**, after the exit code is written — its presence
|
||||
alone means the output file is final.
|
||||
- **Glob `*.out` for status**, never `*.done` — globbing `.done` hides running
|
||||
jobs entirely, the exact blindness this fixes.
|
||||
- **Capture the exit code** (`rc=7` proved a self-test worked; without it a
|
||||
job can "complete" having failed).
|
||||
- **Never put the payload in a line you grep for** — the sentinel is the only
|
||||
completion signal.
|
||||
- **Never `pkill -f <pattern>`** where the pattern also matches your own ssh
|
||||
command line — it kills the session. Distinctive names or `pkill -x`.
|
||||
- **Write helper scripts to a file and `scp` them.** Triple-nested
|
||||
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
||||
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line);
|
||||
every failure disappeared once the script was a file.
|
||||
|
||||
## Verify before trusting it (3am law)
|
||||
|
||||
```bash
|
||||
job.sh start selftest -- bash -c 'echo one; sleep 2; echo two; exit 7'
|
||||
job.sh start slowtest -- bash -c 'echo start; sleep 30; echo end'
|
||||
sleep 4; job.sh status # selftest DONE rc=7; slowtest RUN
|
||||
```
|
||||
|
||||
Both paths, including the failure case, and the rc round-trip — a runner that
|
||||
only proves the happy path is not trustworthy.
|
||||
|
||||
## Reporting discipline
|
||||
|
||||
- Give the user a real terminal when the run is long:
|
||||
`alacritty --title "bench - dankserver" -e bash -c 'ssh -o
|
||||
ServerAliveInterval=30 -t dank@192.168.0.235 "bash /home/dank/job.sh tail"' &`
|
||||
- State the MECHANISM, not vibes: the job is durable on the host;
|
||||
`job.sh out NAME` retrieves it later. Do not narrate "still loading"
|
||||
repeatedly — check once, act, report the resolved result.
|
||||
- When the user asks for status, one fast command, not a narration.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the sweeps this runner executes
|
||||
- [[Deploy State Triage]] — the two-process deploy this lesson generalizes
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
type: reference
|
||||
title: SSH and Remote Access
|
||||
description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session.
|
||||
tags: [ssh, remote, fish, gui, diagnosis]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# SSH and Remote Access
|
||||
|
||||
## Passwordless SSH bootstrap without sshpass/expect
|
||||
|
||||
Use when you must install your pubkey on remote hosts but `sshpass` and
|
||||
`expect` are not installed and you only have a password. Modern OpenSSH
|
||||
(≥ 8.4) can drive the password prompt from a helper program, so the password
|
||||
flows through the **process env only** — never a file, never argv.
|
||||
|
||||
1. **Generate a key** (if none):
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
|
||||
```
|
||||
2. **Create an askpass helper** that echoes the password from an env var (the
|
||||
helper itself stores NO secret):
|
||||
```bash
|
||||
printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
|
||||
chmod 700 /tmp/askpass.sh
|
||||
```
|
||||
3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY:
|
||||
```bash
|
||||
env DEPLOY_PW='<password>' \
|
||||
SSH_ASKPASS=/tmp/askpass.sh \
|
||||
SSH_ASKPASS_REQUIRE=force \
|
||||
setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
|
||||
```
|
||||
Password via the transient `env` argument only.
|
||||
4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names
|
||||
(`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`).
|
||||
5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` —
|
||||
BatchMode fails fast if a password would be required; a clean success
|
||||
proves the key works.
|
||||
|
||||
Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
|
||||
TTY is present and prompts interactively (hangs in a non-interactive tool);
|
||||
`setsid` detaches the controlling terminal — required on older OpenSSH even
|
||||
with `force`; the helper must be executable (`chmod 700`) and print ONLY the
|
||||
password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
|
||||
be rejected first ("Too many authentication failures"). Passwordless **sudo**
|
||||
on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`).
|
||||
|
||||
## Remote execution on a fish login shell
|
||||
|
||||
Running remote commands or shipping files over SSH to a host whose login
|
||||
shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN
|
||||
`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the
|
||||
workstation).
|
||||
|
||||
1. **Auth**: `SSHPASS='<pw>' sshpass -e ssh -o
|
||||
PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'`
|
||||
— password via env, never argv (ps-visible).
|
||||
2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login
|
||||
shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at
|
||||
*parse* time on fish (`Missing end to balance this for loop`). Always wrap
|
||||
compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b ||
|
||||
echo MISSING; done"'` (escape inner `$`).
|
||||
3. **File shipping**: never combine `cat > /remote/path <<EOF` with a
|
||||
trailing `bash -s` in the same ssh command — `cat` drains stdin, so the
|
||||
interpreter silently runs on empty stdin and prints nothing (a silent
|
||||
no-op that *looks* like success). Ship real bytes via stdin redirect:
|
||||
`ssh host 'cat > /remote/path' < localfile`, then verify byte-truth with
|
||||
`md5sum` on both ends.
|
||||
4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish
|
||||
(`$status` there) — wrap in bash or drop the status echo; ssh's own exit
|
||||
code propagates to the caller anyway.
|
||||
|
||||
Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first
|
||||
probe died on fish loop parsing; a `cat >` ship silently swallowed the
|
||||
heredoc meant for `bash -s`, deploying wrong bytes until md5-compared.
|
||||
|
||||
Related: dankserver's login shell is ZSH — never start an unquoted word with
|
||||
`=` in an ssh one-liner (`echo ===` aborts the whole chain).
|
||||
|
||||
## GUI hang — remote diagnosis over SSH
|
||||
|
||||
Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||||
|
||||
1. **Find the smoking gun** — the kernel hung-task reporter already logged
|
||||
it:
|
||||
```bash
|
||||
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
|
||||
```
|
||||
Look for `INFO: task X blocked for more than N seconds` and `blocked on a
|
||||
semaphore likely last held by task Y:PID` (modern kernels name the lock
|
||||
holder outright).
|
||||
2. **Full stacks** for victim and holder:
|
||||
```bash
|
||||
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
|
||||
cat /proc/<pid>/status | head -5 # State: R (spinning) vs D (blocked)
|
||||
```
|
||||
Victim `D` + holder `R` = livelock/deadlock with the lock held;
|
||||
`voluntary_ctxt_switches` in the millions on an `R` kthread = spinning,
|
||||
not progressing.
|
||||
3. **Hardware or software?** GPU/CPU still alive, no Xid errors?
|
||||
```bash
|
||||
nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
|
||||
journalctl -b -k --no-pager | grep -i xid
|
||||
```
|
||||
Instant answers with no Xid ⇒ driver lock issue, not hardware.
|
||||
4. **Trigger window**: block start = first hung message timestamp − reported
|
||||
seconds; then inspect user-session events around it
|
||||
(`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
|
||||
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`),
|
||||
USB workers, etc.
|
||||
|
||||
**Recoverability verdict:** compositor in `D` state is unkillable and a
|
||||
spinning kernel thread holding the lock cannot be poked — **the only exit is
|
||||
reboot**: `sync && systemctl reboot --force --force` (double-force skips
|
||||
stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on
|
||||
something that can complete, e.g. storage), recovery may be possible by
|
||||
clearing that dependency first.
|
||||
|
||||
After diagnosing: map stack symbols to driver internals to name the exact
|
||||
subsystem, check upstream for the exact symbols + driver version, and propose
|
||||
targeted off-switches (`NVreg_RegistryDwords`, module params from
|
||||
`/proc/driver/nvidia/params`) rather than blanket driver swaps.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Fleet Ops Playbook]] — incident procedures that run over these sessions
|
||||
- [[Remote Job Runner]] — long jobs that must survive session loss
|
||||
- [[Deploy State Triage]] — the deploy flow these credentials serve
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
type: index
|
||||
description: Operational procedures for the fleet — job runners, disk and payload reclaim, container reachability, audits, SSH craft, and incident playbooks.
|
||||
tags: [operations, index]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Operations
|
||||
|
||||
Procedures for operating the fleet. The LLM-specific pages live under
|
||||
[LLM](../LLM/index.md); infrastructure and architecture under the
|
||||
`../Infrastructure/` and `../Architecture/` folders.
|
||||
|
||||
- [[Remote Job Runner]] — detached long-job execution with sentinel files;
|
||||
why ssh tail-watchers fail
|
||||
- [[Payload and Disk Reclaim]] — measure-classify-prove-stage-ask disk
|
||||
procedure, plus the purge-and-refresh protocol
|
||||
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
|
||||
the pasta port gate
|
||||
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
|
||||
stop-first, pre-commit iteration, incident-window reconstruction, WoL
|
||||
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
|
||||
posture, whole-fleet audit, bulk redaction, stash audits
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
|
||||
login shells, GUI-hang diagnosis over ssh
|
||||
|
||||
## Related
|
||||
- [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)
|
||||
— monitoring config as code
|
||||
- [Deploy State Triage](../Architecture/Deploy%20State%20Triage.md) — the
|
||||
deploy plumbing
|
||||
- [Quadlet Lifecycle](../Architecture/Quadlet%20Lifecycle.md) — the unit layer
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
type: reference
|
||||
title: PICO-8 Asset Standards
|
||||
description: Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction, detail flow and hierarchy, sheet organization, and the cohesion checklist.
|
||||
tags: [pico8, game-dev, pixel-art, standards]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# PICO-8 Asset Standards
|
||||
|
||||
Ensure every PICO-8 asset (tileset, spritesheet, character sheet, or asset
|
||||
sheet) is cohesive, production-grade, and ready to drop into a game without
|
||||
rework.
|
||||
|
||||
## Palette discipline
|
||||
|
||||
- Use the PICO-8 16-color palette exclusively.
|
||||
- Establish a dominant color family (2–3 related hues) for the main subject.
|
||||
- Reserve accent colors for highlights, danger zones, and focal points only.
|
||||
- Maintain consistent color roles across all assets in a sheet (if shadow is
|
||||
dark blue on one tile, it is dark blue everywhere).
|
||||
|
||||
## Perspective & lighting
|
||||
|
||||
- Choose ONE perspective per sheet (top-down, side-view, or isometric) and
|
||||
never mix.
|
||||
- Apply light from a single consistent direction across all tiles and
|
||||
sprites.
|
||||
- Shadows follow the same shape logic on every asset (e.g. all cast shadows
|
||||
point bottom-right).
|
||||
|
||||
## Detail flow & hierarchy
|
||||
|
||||
- Start tiles/props from the palette's dominant colors, then add edge
|
||||
highlights and shading.
|
||||
- Keep tile density uniform — avoid one hyper-detailed tile next to empty
|
||||
flat tiles.
|
||||
- Props and decorations should read clearly at 8×8 or 16×16 without zooming.
|
||||
|
||||
## Sheet organization
|
||||
|
||||
- Group related assets together (terrain together, props together,
|
||||
characters together).
|
||||
- Leave consistent spacing between assets for clean cropping.
|
||||
- Label sections visually through grouping, not text.
|
||||
|
||||
## Cohesion checks (before finishing a sheet)
|
||||
|
||||
- All colors appear in the PICO-8 palette.
|
||||
- Perspective does not flip mid-sheet.
|
||||
- Lighting direction is consistent.
|
||||
- Edge treatment (dither, outline, anti-aliasing style) matches across
|
||||
assets.
|
||||
- No orphan tiles that don't connect to any neighbor.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
- Mixing 8-bit and 16-bit detail levels.
|
||||
- Using outline style on some tiles but not others.
|
||||
- Forgetting the palette constraint — colors outside the 16 PICO-8 colors.
|
||||
- Inconsistent tile edge connections (a grass tile that only connects to one
|
||||
dirt variant).
|
||||
@@ -115,6 +115,64 @@ macOS mini) as quadlets inside its podman machine — not on Dankserver.
|
||||
- [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack
|
||||
- [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices
|
||||
|
||||
## Skill Distillation (2026-10-05)
|
||||
|
||||
The managed skill library distilled into the vault, one reference page per
|
||||
skill-cluster. See [log.md](log.md) for the wave plan and results.
|
||||
|
||||
**Operations:**
|
||||
|
||||
- [[Remote Job Runner]] — detached long-job execution with sentinel files
|
||||
- [[Payload and Disk Reclaim]] — disk reclaim: measure, classify, prove,
|
||||
stage, ask; plus purge-and-refresh
|
||||
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
|
||||
the pasta port gate
|
||||
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
|
||||
stop-first, pre-commit iteration, incident-window reconstruction, WoL
|
||||
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
|
||||
posture, whole-fleet audit, bulk redaction
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
|
||||
GUI-hang diagnosis over ssh
|
||||
|
||||
**Architecture:**
|
||||
|
||||
- [[Stale Signal Audit]] — classifying green-but-dead and down-but-fine
|
||||
signals
|
||||
- [[Quadlet Lifecycle]] — quadlet units, drift, crash-loops, compose
|
||||
cutover
|
||||
- [[Deploy State Triage]] — revision markers, pending lockouts, deploy
|
||||
interruptions, shared-checkout hygiene
|
||||
|
||||
**Infrastructure:**
|
||||
|
||||
- [[Observability Provisioning]] — Prometheus/Grafana config as code
|
||||
- [[Homepage Status Probes]] — the status page's checker, vantage rules,
|
||||
and triage
|
||||
|
||||
**LLM cross-cutting (full index: [LLM/index.md](LLM/index.md)):**
|
||||
|
||||
- [[GGUF Compatibility]] · [[Llama Cpp Serving Sizing]] ·
|
||||
[[Llama Cpp Multi-Device Splits]] · [[PCIe Link and Atomics]] ·
|
||||
[[RDNA4 Container Serving]] · [[RDNA4 MoE Expert Offload VRAM Budget]] ·
|
||||
[[vLLM Boot Forensics and Config Interrogation]] ·
|
||||
[[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] ·
|
||||
[[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] ·
|
||||
[[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] ·
|
||||
[[FreeToken Builds]]
|
||||
|
||||
**Agent:**
|
||||
|
||||
- [[OMP Harness]] — compaction arithmetic, per-model samplers, preamble
|
||||
audits
|
||||
- [[Agent Docs and Edits]] — doc-structure audits, hashline edit recovery
|
||||
- [[Command Center Dotfiles]] — the ~/.omp repo-as-deployment
|
||||
|
||||
**Desktop / Research:**
|
||||
|
||||
- [[Noctalia and Niri]] — the desktop shell, palette recolor, template
|
||||
rendering, rotation pinning, iCX3 RGB
|
||||
- [[PICO-8 Asset Standards]] — cohesive PICO-8 asset sheets
|
||||
|
||||
## LLM Fleet
|
||||
|
||||
The LLM inference fleet on the two hosts. Full index:
|
||||
@@ -123,9 +181,14 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
### Deployment and Topology
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-primary-freetoken-pr587]] — the ACTIVE frank-primary runbook
|
||||
(2026-10-05): FreeToken built at PR #587 head `77a7201` on the host into
|
||||
`frank-primary-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned
|
||||
host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO; rollback is the
|
||||
tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`)
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE
|
||||
frank-primary runbook: the R9V radiance-vLLM fork (docker-pinned bundle
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle
|
||||
`2dac17a2`, docker→podman shim) serving the unsloth
|
||||
Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + blockfp8 MTP4 +
|
||||
BF16 vision on Dankserver's both R9700s — TP=2 tiered experts (80-slot
|
||||
|
||||
@@ -1,6 +1,71 @@
|
||||
# Bundle Change Log
|
||||
|
||||
|
||||
## 2026-10-05
|
||||
- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
|
||||
dankserver.** Built FreeToken at PR head `77a7201` on the host into the
|
||||
local image `frank-primary-freetoken:pr587-77a7201`
|
||||
(digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws
|
||||
bought with failures: the pip-ROCm wheel needs a `libhiprtc.so` dev
|
||||
symlink for the PR's `_cpu_moe` link, and a stale `docker login` at
|
||||
`~/.docker/config.json` poisons podman's ANONYMOUS pulls). Trio rewritten
|
||||
through git (env/serve/unit + seam + deployer): ONE R9700, experts in
|
||||
pinned host RAM behind a 0.10 GPU LRU, PLE 47.7 GiB pinned, KV pool
|
||||
299,968 tokens (multiple-of-64 — the `qsa_sparse` auto backend overrides
|
||||
page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`; models.yml flipped accordingly). Container renamed
|
||||
to the unit name (Frank's call; `frank-primary-qwen` → legacy list);
|
||||
deployer image gate → `podman image exists` (local builds carry no
|
||||
RepoDigests) and health gate reads the BODY — FreeToken 200s with
|
||||
`{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
|
||||
(vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK
|
||||
completion, reasoning split (17×24 → "408"), native `tool_calls`, vision
|
||||
"Red" probe, boot 2.3 min cold / 92 s warm, decode 17.1 tok/s
|
||||
single-stream (vLLM was ~30 on two cards; the KV contract caps the
|
||||
single-card LRU; TP=2 exists but the pynccl JIT can't compile for
|
||||
gfx1201 — probed, signature in the runbook). `/metrics` is a 404 — the
|
||||
Dankclaw Prometheus llm-fleet scrape of :7770 reads down until
|
||||
repointed. Rollback: tag `frank-primary-vllm-shapeG-20261003`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
- **Skill library distilled into the vault (waves 1–7).** All ~173 managed
|
||||
skills read in full and merged, one polished reference page per
|
||||
skill-cluster:
|
||||
- **LLM cross-cutting (16 pages, `LLM/`):** GGUF Compatibility,
|
||||
Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and
|
||||
Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget,
|
||||
vLLM Boot Forensics and Config Interrogation, vLLM Porting and Compile
|
||||
Triage, LLM Bench Windows and Measurement, LLM Performance Diagnosis,
|
||||
Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split
|
||||
Triage, FreeToken Builds, plus deltas folded into the retired runbooks
|
||||
(GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline,
|
||||
FLM pack registration, ik hybrid envelope, frank-medium split
|
||||
rebalance, frank-primary llama.cpp GGUF preflight, batch-ubatch split
|
||||
section, bench-window D-state/stale-read notes, second-endpoint
|
||||
bring-up in Model Swap).
|
||||
- **Architecture (3):** Stale Signal Audit, Quadlet Lifecycle, Deploy
|
||||
State Triage.
|
||||
- **Operations (6 + index):** Remote Job Runner, Payload and Disk
|
||||
Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit
|
||||
and Posture, SSH and Remote Access.
|
||||
- **Infrastructure (2 + updates):** Observability Provisioning, Homepage
|
||||
Status Probes; updates to Authentik (guard-chain flip recipe, `ak
|
||||
shell` mechanics, user-provisioning playbook, outpost netns probe),
|
||||
Anubis (guard-chain layer diagnosis), LAN-DNS (per-client resolver
|
||||
drop-in + browser DoH bypass, Porkbun zone ops), Swap Configuration
|
||||
(filesystem/rename caveats), OpenWebUI (introspection, verification
|
||||
battery, ComfyUI imagegen), Hermes (config-home drift, live config
|
||||
edits, auxiliary routing, password rotation — the pre-existing page was
|
||||
preserved and extended).
|
||||
- **Host pages:** FrankBook-Pro ARR-Stack filled with the API-version
|
||||
split, schema-first probing, auth-posture method, auth-config trap, and
|
||||
the qBittorrent E2E verification procedure added to [[FrankBook-Pro/qbittorrent]].
|
||||
- **Desktop/Agent/Research:** Noctalia and Niri (config layers, remote
|
||||
IPC, palette recolor, offline template render, rotation pinning, iCX3
|
||||
RGB), Agent/ (OMP Harness, Agent Docs and Edits, Command Center
|
||||
Dotfiles, index), PICO-8 Asset Standards.
|
||||
- Index updated: `LLM/index.md` gained a Cross-Cutting References
|
||||
section; Operations/ and Agent/ got index pages. AGENTS.md now points
|
||||
at the new sections. Link checker + full test suite green.
|
||||
- **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright
|
||||
project.** The `11-admin` walk included the six `*arr`/qBittorrent admin
|
||||
links that terminate on 192.168.0.112, so every deploy gate died there when
|
||||
|
||||
@@ -98,10 +98,12 @@ scrape_configs:
|
||||
# service down. Same host, same rule, so both local endpoints use the
|
||||
# gateway.
|
||||
#
|
||||
# (2026-09-26) frank-primary is llama.cpp again (the unsloth b11030-mix
|
||||
# prebuilt serving Qwen3.8-Flash-Next Q8_0) on Dankserver, a DIFFERENT
|
||||
# machine: the netavark published-port quirk above is same-host only, so
|
||||
# plain LAN routing reaches 192.168.0.235:7770 and the job returns.
|
||||
# (2026-10-05) frank-primary left this file: FreeToken (the PR587 ROCm
|
||||
# engine now serving it) has NO /metrics endpoint — the job scraped 404s
|
||||
# forever, which is a structural never-green, not an outage. The
|
||||
# netavark published-port quirk above is same-host only; plain LAN
|
||||
# routing REACHES 192.168.0.235:7770 (health answers) — only the metrics
|
||||
# path is absent. If FreeToken grows a /metrics, re-add the job here.
|
||||
- job_name: "frank-small"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
@@ -116,9 +118,3 @@ scrape_configs:
|
||||
labels:
|
||||
service: frank-large
|
||||
|
||||
- job_name: "frank-primary"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.235:7770"]
|
||||
labels:
|
||||
service: frank-primary
|
||||
@@ -1,10 +1,12 @@
|
||||
# Alerting and recording rules for the LLM fleet.
|
||||
#
|
||||
# One metric namespace: llamacpp:* — frank-small, frank-large and
|
||||
# frank-primary, all llama.cpp with /metrics (the family verified from
|
||||
# /metrics on 2026-09-16 and 2026-09-20). frank-primary is scraped across
|
||||
# the LAN at 192.168.0.235:7770 (the netavark published-port quirk that
|
||||
# forces the gateway for the Dankclaw endpoints is same-host only).
|
||||
# One metric namespace: llamacpp:* — frank-small and frank-large, both
|
||||
# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16
|
||||
# and 2026-09-20). frank-primary LEFT the family 2026-10-05: FreeToken (the
|
||||
# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was
|
||||
# deleted rather than left never-green.
|
||||
# (The netavark published-port quirk that forces the Dankclaw endpoints
|
||||
# onto the gateway is same-host only.)
|
||||
#
|
||||
# Every expression is keyed to a series that EXISTS. Nothing matches on
|
||||
# absence: when an endpoint's runtime leaves the fleet, its selectors are
|
||||
@@ -110,6 +112,6 @@ groups:
|
||||
description: >-
|
||||
Prometheus cannot reach the LLM endpoint. frank-small and
|
||||
frank-large are root units on Dankclaw: `systemctl is-active
|
||||
frank-small` / `frank-large`. frank-primary is a root unit on
|
||||
Dankserver: `systemctl is-active frank-primary`. All three
|
||||
scraped endpoints export the llamacpp:* family.
|
||||
frank-small` / `frank-large`. Both scraped endpoints export
|
||||
the llamacpp:* family. (frank-primary left the scrape set
|
||||
2026-10-05 — FreeToken exports no /metrics.)
|
||||
+148
-319
@@ -1,26 +1,25 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
|
||||
#
|
||||
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
|
||||
# 2026-10-01 BF16 cutover, the overnight redo campaign's verdict): the serve
|
||||
# script drives `podman run` DIRECTLY (the image entrypoint execs
|
||||
# `vllm serve`), with every engine flag qualified by boot probe and carried
|
||||
# in the repo env file. The seam pins what the flags alone do not say: the
|
||||
# KV qualification law (the night ladder: 262144/245760 OOM at graph
|
||||
# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
|
||||
# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
|
||||
# clean cache + 262k windows), the automatic-offload law (the fork plans
|
||||
# the tiered-expert offload itself; no flag exists to set), the self-healing
|
||||
# law (the run returns before the engine is up, so the serve script must
|
||||
# wait on the container for Restart=on-failure to mean anything), the
|
||||
# memlock law that makes the pinned host rows legal, the deployer's
|
||||
# image/payload gates with the false-green MainPID check, and the measured
|
||||
# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
|
||||
# no-spec family measured fastest once the prefetch tax was removed; MTP
|
||||
# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
|
||||
# MNBT 2048, the CUDA-graph plan term pinned to the measured capture cost, and
|
||||
# the client contract (reasoning parser qwen3,
|
||||
# tool parser qwen3_xml, preserve_thinking on).
|
||||
# This file asserts the FreeToken (PR #587 ROCm) ROOT-UNIT shape (the
|
||||
# 2026-10-05 cutover from the tcclaviger RDNA4 vLLM fork, git tag
|
||||
# frank-primary-vllm-shapeG-20261003): the serve script drives `podman run`
|
||||
# DIRECTLY (the image carries `ft serve`, built on the host from
|
||||
# scripts/docker/frank-primary-freetoken.Dockerfile), one R9700, with every
|
||||
# engine value carried in the repo env file. The seam pins what flags alone
|
||||
# do not say: the KV-pool math (300k tokens at 24 KiB/token clears the
|
||||
# harness worst case of 177,356), the deployed single-GPU shape
|
||||
# (--tensor-parallel-size EXISTS in FreeToken — a 2-card TP shape is future
|
||||
# tuning; HIP_VISIBLE_DEVICES pins THIS shape to one card), the host-RAM
|
||||
# law (experts + PLE pin ~170 GiB — the memlock law that makes the pin
|
||||
# legal), the self-healing law (the run returns before the engine is up, so
|
||||
# the serve script must wait on the container for Restart=always to mean
|
||||
# anything), the image-existence gate (a local build carries no
|
||||
# RepoDigests — the old digest grep refused exactly this shape; pinning
|
||||
# lives in the Dockerfile), the deployer's payload gates with the
|
||||
# false-green MainPID check, and the client contract (parsers AUTO:
|
||||
# reasoning_content emission is the engine's native behavior — models.yml
|
||||
# maps reasoningContentField on the harness side, not here).
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -29,29 +28,30 @@ SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-primary.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
DOCKERFILE="$REPO_ROOT/scripts/docker/frank-primary-freetoken.Dockerfile"
|
||||
|
||||
@test "frank-primary unit, serve script and env exist in scripts/" {
|
||||
@test "frank-primary unit, serve script, env and Dockerfile exist" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -x "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$DOCKERFILE" ]
|
||||
}
|
||||
|
||||
@test "frank-primary is boot-persistent, self-healing and unprivileged" {
|
||||
@test "frank-primary unit is boot-persistent, self-healing and unprivileged" {
|
||||
# Root unit parity with frank-large/frank-small: enable is the [Install]
|
||||
# section, the restart policy survives a crash, and the tiered-expert
|
||||
# pin needs an unbounded memlock (a root unit grants it via PID1; the
|
||||
# dank session's own hard limit is 8 MiB and cannot be raised without
|
||||
# restarting user@1000.service).
|
||||
# section, the restart policy survives a crash, and the pinned host RAM
|
||||
# (expert rows + PLE table) needs an unbounded memlock (a root unit
|
||||
# grants it via PID1; the dank session's own hard limit is 8 MiB).
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
# Restart=always, not on-failure: a runtime worker OOM makes vLLM's API
|
||||
# server exit CLEAN, so ExecStart returns 0 and on-failure never fires
|
||||
# (measured 2026-10-01 — the endpoint stayed down until a human noticed).
|
||||
# Restart=always, not on-failure: the vLLM-era lesson carries over — a
|
||||
# clean front-end exit after a worker fault reads as success to systemd,
|
||||
# so on-failure never fires.
|
||||
grep -q '^Restart=always$' "$UNIT"
|
||||
! grep -q '^Restart=on-failure$' "$UNIT"
|
||||
# Exiting the engine takes as long as vLLM's drain: ExecStop runs
|
||||
# `podman stop -t 120`, so the stop timeout MUST exceed 120 s or systemd
|
||||
# kills the stop and records 'Failed with result timeout' (measured
|
||||
# 2026-10-01). Guard the relationship, not just the literal.
|
||||
# ExecStop runs `podman stop -t 120`, so the stop timeout MUST exceed
|
||||
# 120 s or systemd kills the stop and records 'Failed with result
|
||||
# timeout' (measured 2026-10-01). Guard the relationship, not the
|
||||
# literal.
|
||||
stop_t=$(awk -F= '/^TimeoutStopSec=/{print $2}' "$UNIT")
|
||||
[ -n "$stop_t" ] && [ "$stop_t" -gt 120 ]
|
||||
grep -q '^ExecStop=.*podman stop -t 120' "$UNIT"
|
||||
@@ -69,320 +69,121 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q 'FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the unit stops the container child explicitly" {
|
||||
@test "the unit stops the container child explicitly — container identity is the unit name" {
|
||||
# The serve script blocks on podman wait, which cannot stop the
|
||||
# container child; systemd stop goes through ExecStop or the engine
|
||||
# outlives the unit and holds both GPUs. The container name must match
|
||||
# the env file's identity.
|
||||
# outlives the unit and holds the GPU. Frank's 2026-10-05 call: the
|
||||
# container name IS the service name (frank-primary), and the vLLM-era
|
||||
# name frank-primary-qwen moved into the legacy cleanup list.
|
||||
grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
|
||||
grep -q 'frank-primary-qwen' "$UNIT"
|
||||
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
|
||||
grep -q 'podman stop -t 120 frank-primary$' "$UNIT"
|
||||
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary$' "$ENV_FILE"
|
||||
grep -q 'frank-primary-qwen' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve shape is direct podman run, not a launcher or shim" {
|
||||
# The tcclaviger stack is fully expressed by flags; the R9V launcher,
|
||||
# docker->podman shim and setup-state layering are the RETIRED shape.
|
||||
@test "the serve shape is direct podman run, not a launcher, shim, or the vLLM fork" {
|
||||
grep -q 'podman run' "$SERVE"
|
||||
! grep -q 'launch.sh' "$SERVE"
|
||||
! grep -q 'podman-shim' "$SERVE"
|
||||
! grep -q 'R9V_STATE_FILE' "$SERVE"
|
||||
! grep -q -- '--model-draft' "$SERVE"
|
||||
! grep -q -- '--spec-type' "$SERVE"
|
||||
# The tcclaviger vLLM fork is RETIRED — its image name and its
|
||||
# fork-specific flags must not survive in the FreeToken run line
|
||||
# (TP=2 is gone with the fork; FreeToken serves ONE GPU).
|
||||
! grep -q 'tcclaviger' "$SERVE"
|
||||
! grep -q 'tcclaviger' "$ENV_FILE"
|
||||
! grep -q -- '--tensor-parallel-size' "$SERVE"
|
||||
! grep -q -- '--kv-cache-dtype' "$SERVE"
|
||||
! grep -q -- '--max-num-seqs' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-10-02 shape D. History: the 2026-10-01 night ladder put SEQS=3
|
||||
# bf16 at 241664 (KV 732,359 / 3.03x); SEQS=6 fp8 does NOT fit (6 x
|
||||
# 241664 asked 10.70 GiB against 9.18 GiB free); SEQS=5 fp8 x 262144
|
||||
# BOOTED (1,350,182 / 5.15x) but left ~0 bytes free and a real 995-token
|
||||
# request killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes
|
||||
# free`. UTIL is a REAL arena lever (the fork plans budget = total x util)
|
||||
# but a weak one, and Frank caps it at 0.982 — never 1.00 (see
|
||||
# FRANK_PRIMARY_UTIL for the two ceiling deaths that justify the margin).
|
||||
# SEQS=2 is IMPOSSIBLE on this model, measured 2026-10-02 by
|
||||
# single-variable boot probe against the then-deployed shape: SEQS 2 fails
|
||||
# at BOTH MNBT 1024 and 2048 with torch.fx ConstraintViolationError
|
||||
# (query_start_loc.size()[0] marked dynamic but specialized to a constant
|
||||
# (3)); SEQS 3 passes at both. model_runner sizes that buffer
|
||||
# max_num_reqs + 1, so the width is SEQS + 1 and MNBT only moves the
|
||||
# compile RANGE.
|
||||
# So the shape is 3 slots with a pool that admits only ~2 full-length
|
||||
# conversations: measured live, KV_BYTES 3.15e9/rank reports 398,658
|
||||
# tokens = 1.99x for a 200704-token request (7,892 B/token at THIS shape;
|
||||
# the per-token price is shape-dependent, 7,701 at 262144). Shorter
|
||||
# requests use the spare blocks; a third LONG request queues.
|
||||
# The arena is sized by max_model_len, NOT by KV_BYTES: the planner
|
||||
# reserves registry 9190 B/token x max_model_len (2.48 GiB at 262144 ->
|
||||
# arena 6,248 slots; 1.91 GiB at 200704 -> 6,496). The pool cut buys
|
||||
# runtime headroom, not arena.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=3166500000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
|
||||
# while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
|
||||
# (arena 16.65 device-resident, NRestarts=3 at ~6.7 min each) — there it was
|
||||
# the last 2.35 GiB over the line, not the cause. Probed on the shape-D base
|
||||
# with production stopped: boots healthy in 420 s, plan reports runtime
|
||||
# overhead 5.25 GiB (not 7.60), expert cache 17.22 GiB / 7,522 slots
|
||||
# (+1,026 / +16% over 6,496). The pool is NOT cut with it (option A): the
|
||||
# pool is not an arena lever, so 3.15e9 keeps the measured 1.99x.
|
||||
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
|
||||
# GPU_MAX_HW_QUEUES caps HSA hw queues per GPU; real knob (the literal is in
|
||||
# libhsa-runtime64.so), passed inside the container as a bare name — the env
|
||||
# file keeps only the prefixed form.
|
||||
grep -q '^FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2$' "$ENV_FILE"
|
||||
# The compile cache is a BOOT-COST fix: the image ships no /root/.cache at
|
||||
# all, our boot writes 1.1 GB of vLLM AOT artifacts (4,665 files) plus
|
||||
# 229 MB of comgr JIT kernels into the ephemeral layer, and every
|
||||
# `podman rm -f` throws them away — re-paying `torch.compile took 181.20 s`
|
||||
# (173.19 s of it one graph) on every deploy, restart and reboot.
|
||||
grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
# Prefix caching is ON (reverted 2026-10-05 the same day it was
|
||||
# disabled): the fork's QFN align-mode GDN state fault (48 gdn_hip
|
||||
# state faults, slot 42 heads 3/8, at the 13th 3136-token boundary =
|
||||
# 40,768 — the "~40k corruption" report) was mitigated with
|
||||
# --no-enable-prefix-caching, but live long conversations STILL
|
||||
# degraded at ~40k with caching off and zero faults logged — cost
|
||||
# with no fix, so the historical flag returned. The knob stays
|
||||
# commented in the env file for a future fixed fork build; flip it to
|
||||
# 1 only together with a re-verification past 40,768 tokens.
|
||||
grep -qE '^# FRANK_PRIMARY_NO_PREFIX_CACHING=1$' "$ENV_FILE"
|
||||
@test "the FreeToken engine shape is in the env file" {
|
||||
# KV pool: TOKENS, not bytes. 24 KiB/token (12 full-attention layers x
|
||||
# 2 KV heads x 256 head_dim x K+V x f16) — the 36 GDN layers use fixed
|
||||
# per-seq state, not per-token KV. 299,968 tokens = 7.0 GiB and clears
|
||||
# the harness worst case (140,492-token prompt + 36,864 max_tokens =
|
||||
# 177,356); one NATIVE 262,144-token conversation + full budget also
|
||||
# fits. The value must stay a MULTIPLE OF 64: the auto attention
|
||||
# backend is qsa_sparse, which overrides page size to 64 (measured
|
||||
# first boot — 300,000 was refused). The arithmetic guard keeps future
|
||||
# resizes honest.
|
||||
grep -q '^FRANK_PRIMARY_FT_NUM_TOKENS=299968$' "$ENV_FILE"
|
||||
nt=$(awk -F= '/^FRANK_PRIMARY_FT_NUM_TOKENS=/{print $2}' "$ENV_FILE")
|
||||
[ "$nt" -ge 177356 ]
|
||||
[ $((nt % 64)) -eq 0 ]
|
||||
# First-boot conservative shape: 2 decode slots, 10% expert LRU
|
||||
# (~12 GiB of the ~120 GiB fp8 expert set), output ceiling = the
|
||||
# harness maxTokens, single R9700 at index 0.
|
||||
grep -q '^FRANK_PRIMARY_FT_MAX_RUNNING=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_FT_MOE_RATE=0.10$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS=36864$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_FT_GPU=0$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve script wires the engine flags from the env" {
|
||||
grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
|
||||
grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
|
||||
grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
|
||||
# Speculative decode, the cudagraph capture ladder and the sampling
|
||||
# defaults are all ENV-DRIVEN knobs (unset -> the flag is absent, which is
|
||||
# the shipped shape today). JSON values carry no shell-special characters
|
||||
# that this quoting cannot carry.
|
||||
grep -q -- '${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"}' "$SERVE"
|
||||
grep -q -- '${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"}' "$SERVE"
|
||||
grep -q -- '${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"}' "$SERVE"
|
||||
grep -q -- '--tensor-parallel-size 2' "$SERVE"
|
||||
# Prefix caching renders exactly ONE flag from the knob, branched
|
||||
# explicitly in the serve script (the ${VAR:--flag} double-expansion is
|
||||
# ambiguous bash — when set it parsed as a substring offset, measured
|
||||
# 2026-10-05): set -> --no-enable-prefix-caching (the fork-named
|
||||
# mitigation, tried and REVERTED the same day — live corruption
|
||||
# persisted at ~40k with zero faults), unset -> the shipped
|
||||
# --enable-prefix-caching. Both directions are spelled out so a fork
|
||||
# default change cannot silently flip the contract.
|
||||
grep -qF 'FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching' "$SERVE"
|
||||
grep -qF 'FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching' "$SERVE"
|
||||
grep -qF '"$FRANK_PRIMARY_PREFIX_CACHE_FLAG"' "$SERVE"
|
||||
# The observable contract: the exact flag word the engine receives,
|
||||
# rendered from the shipped env through the serve script's branch (the
|
||||
# same render-assert pattern the spec knobs use above).
|
||||
run bash -c '
|
||||
set -a; . "$1" >/dev/null; set +a
|
||||
if [ -n "${FRANK_PRIMARY_NO_PREFIX_CACHING:-}" ]; then
|
||||
FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching
|
||||
else
|
||||
FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
|
||||
fi
|
||||
printf %s "$FRANK_PRIMARY_PREFIX_CACHE_FLAG"' _ "$ENV_FILE"
|
||||
[ "$output" = "--enable-prefix-caching" ]
|
||||
grep -q -- '--trust-remote-code' "$SERVE"
|
||||
# DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
|
||||
# (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
|
||||
grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
|
||||
! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
|
||||
# --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
|
||||
# default (7.60 GiB = fitted + compile/graph-capture headroom), which only
|
||||
# moves the plan's `room` term and therefore the expert arena: ~1,025
|
||||
# more slots at 2.344 MiB each.
|
||||
grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
|
||||
# Shape E host/CPU wiring: the cache dir is created before it is mounted
|
||||
# (a cold boot must have a target) and the mount lands on /root/.cache,
|
||||
# whose vllm/ child is the path VLLM_CACHE_ROOT defaults to — the engine log
|
||||
# names it as `Using cache directory: /root/.cache/vllm/torch_compile_cache/…`.
|
||||
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
|
||||
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
|
||||
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
|
||||
# Shape F/G: the CUDA-graph plan term is carried as CONTAINER env (the fork
|
||||
# reads it at call time inside estimate_cudagraph_memory), and both must be
|
||||
# present — the graph term and UTIL are the ONLY plan terms this repo can
|
||||
# move, and shrinking them is what grows the arena. The base must be shipped
|
||||
# explicitly: unset, the estimator would fall back to its 256 MiB default.
|
||||
grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE"
|
||||
grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE"
|
||||
# The host region is the ONE plan term that autofits per-boot: with no cap
|
||||
# the bound is MemAvailable at that moment (boot 2026-10-05: 237.97 GiB ->
|
||||
# 159.39 GiB usable, while the running host later reads 75.4 GiB available).
|
||||
# An explicit --expert-offload-mem cap makes the plan deterministic whatever
|
||||
# the co-tenant RAM state is at boot; 120 GiB covers the measured 105.48
|
||||
# GiB of swappable experts (52.74/rank x 2) with 14% headroom.
|
||||
grep -q -- '${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"}' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120$' "$ENV_FILE"
|
||||
grep -q 'ft serve --model /models' "$SERVE"
|
||||
grep -q -- '--served-model-name "$FRANK_PRIMARY_SERVED_NAME"' "$SERVE"
|
||||
grep -q -- '--host 0.0.0.0 --port 7770' "$SERVE"
|
||||
grep -q -- '--gpu "$FRANK_PRIMARY_FT_GPU"' "$SERVE"
|
||||
grep -q -- '--num-tokens "$FRANK_PRIMARY_FT_NUM_TOKENS"' "$SERVE"
|
||||
grep -q -- '--max-running-requests "$FRANK_PRIMARY_FT_MAX_RUNNING"' "$SERVE"
|
||||
grep -q -- '--max-output-tokens "$FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS"' "$SERVE"
|
||||
grep -q -- '--moe-cache-rate "$FRANK_PRIMARY_FT_MOE_RATE"' "$SERVE"
|
||||
# Single-GPU law: visibility is pinned to one card so in-container
|
||||
# index 0 is the right R9700 (mirrors the env's FT_GPU).
|
||||
grep -q -- '-e HIP_VISIBLE_DEVICES="$FRANK_PRIMARY_FT_GPU"' "$SERVE"
|
||||
# The boot-ladder escape hatch is opt-in: unset renders NO extra argv.
|
||||
grep -q -- '${FRANK_PRIMARY_FT_EXTRA_ARGS:+$FRANK_PRIMARY_FT_EXTRA_ARGS}' "$SERVE"
|
||||
! grep -q '^FRANK_PRIMARY_FT_EXTRA_ARGS=.' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "tools and reasoning are wired — the client contract is served" {
|
||||
# 2026-10-01: the rollback-era trio served NO tool dialect and left
|
||||
# thinking inline in `content` with a dangling `</think>` and a null
|
||||
# `reasoning` field (measured: content 'We need answer...Brief.\n</think>
|
||||
# \n\n391') — the malformed thinking block clients choke on. The working
|
||||
# values are the ones ecaad36 shipped for this model and this fork's
|
||||
# registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
|
||||
# preserve_thinking keeps prior thinking blocks across turns.
|
||||
grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
|
||||
grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
|
||||
grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
|
||||
grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
|
||||
grep -q 'preserve_thinking' "$ENV_FILE"
|
||||
grep -q 'enable_thinking' "$ENV_FILE"
|
||||
@test "parsers are AUTO — the engine infers them natively, the harness maps the field" {
|
||||
# FreeToken infers the qwen3.8 reasoning/tool dialects from the model
|
||||
# family and emits thinking in `reasoning_content` natively — NO
|
||||
# --reasoning-parser/--tool-call-parser override flags. The vLLM-era
|
||||
# client-contract wiring (qwen3 parser, qwen3_xml tool parser,
|
||||
# default-chat-template-kwargs) is gone; models.yml's
|
||||
# reasoningContentField flip is the harness-side contract and lives in
|
||||
# the operator's home, not this repo.
|
||||
! grep -q -- '--reasoning-parser' "$SERVE"
|
||||
! grep -q -- '--tool-call-parser' "$SERVE"
|
||||
! grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
! grep -q -- '--default-chat-template-kwargs' "$SERVE"
|
||||
! grep -q 'FRANK_PRIMARY_REASONING_PARSER' "$ENV_FILE"
|
||||
! grep -q 'FRANK_PRIMARY_TOOL_PARSER' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
|
||||
# ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
|
||||
# expert planner moves the overflow to pinned host rows by itself (the
|
||||
# arena 6,574 slots x 2.344 MiB plus host rows). There is no
|
||||
# --enable-expert-offload flag in this arm; the legality of the pin is
|
||||
# the unit's unbounded LimitMEMLOCK.
|
||||
! grep -q -- '--enable-expert-offload' "$SERVE"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
|
||||
}
|
||||
|
||||
@test "speculative decode is an ENV KNOB — the old retirement verdict was invalid" {
|
||||
# The 2026-10-02 c1248d4 retirement of MTP compared the WRONG PAIR:
|
||||
# fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance) against the bf16 no-spec boot
|
||||
# (36.4-39.4), then bf16 proved unbootable and fp8 came back WITHOUT MTP —
|
||||
# so fp8+spec was never compared against fp8+no-spec on the same shape. The
|
||||
# no-spec incumbent runs ~30 tok/s, under the fleet's 35 tok/s target, while
|
||||
# EVERY vendor flash-next recipe ships MTP3, and our MTP4 run used the
|
||||
# default GREEDY draft sampling (probabilistic is the other fork option and
|
||||
# is what the recipes use under temp>0). So spec is a knob, not a law: the
|
||||
# shipped value is whatever a matched-pair re-measurement wins.
|
||||
grep -q 'THE RETIREMENT VERDICT WAS INVALID' "$ENV_FILE"
|
||||
# The knobs are empty in the shipped env, so the `${VAR:+…}` wiring renders
|
||||
# NO spec/compilation flag — asserted by rendering, not by grepping the
|
||||
# assignment (shape E today has neither flag in its argv).
|
||||
run bash -c 'set -a; . "$1" >/dev/null; set +a
|
||||
printf %s "${FRANK_PRIMARY_SPEC:+--speculative-config $FRANK_PRIMARY_SPEC}${FRANK_PRIMARY_COMPCONFIG:+--compilation-config $FRANK_PRIMARY_COMPCONFIG}"' _ "$ENV_FILE"
|
||||
[ -z "$output" ]
|
||||
# The capture-ladder law: the piecewise graph ladder must contain
|
||||
# num_speculative_tokens + 1, so an MTP3 shape captures [4, 8, 12] and the
|
||||
# 2026-10-02 SEQS=2 failure (query_start_loc specialized to the constant 3)
|
||||
# is why SEQS stays a multiple-compatible value.
|
||||
grep -q 'num_speculative_tokens + 1' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "KV dtype is fp8 — the incumbent, and the only one that boots" {
|
||||
# fp8 costs ~7,701 B/token at 262144 (7,892 at 200704 — shape-dependent),
|
||||
# decomposed as 6,138 attention + 1,563 mamba/side (dtype-independent);
|
||||
# bf16 costs 13,839 (12,276 + 1,563).
|
||||
# bf16 was abandoned 2026-10-01: its 2 x 262144 boot died with 0 bytes
|
||||
# free on a real request, and the SEQS=2 slot count it was designed
|
||||
# around cannot compile this model at all (see the shape test above).
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
|
||||
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
|
||||
# sweep's biggest lever, +18.7% decode vs default geometry, and the win
|
||||
# GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
|
||||
# g64x128 is past the peak; g8x16 mild. They are container ENV (the
|
||||
# fork's knobs), not engine flags — passed through with -e.
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the arena sits just under the ceiling and the KV ask is honoured" {
|
||||
# The planner sizes expert cache as budget - non-expert - KV reserve -
|
||||
# graphs - overhead, and two of those five terms are reachable from a
|
||||
# deploy: budget (= device_total x util) and graphs (VLLM_CUDAGRAPH_MEM_*).
|
||||
# UTIL has a HARD LAW, not a field cap: 0.982, never 1.00. The 1.00 attempt
|
||||
# is UNBOOTABLE on this fork — worker init raises
|
||||
# `ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)`
|
||||
# on both ranks because the worker demands free > budget and budget ==
|
||||
# total — and even if it booted, 0.13-GiB-margin shapes have died under
|
||||
# real load. So the arena tops out at 0.982 budget (31.29 GiB).
|
||||
# Graphs: charge strictly below the fork's analytic 563 MiB default while
|
||||
# still paying per graph (the measured post-capture delta is -0.06 GiB, i.e.
|
||||
# the base pool is the over-charge, but a future spec-decode shape captures
|
||||
# 18 graphs and must still be charged).
|
||||
# MNBT stays 2048: 512 doubles TTFT at 4064 ctx, 1024-4096 are decode-
|
||||
# interchangeable AND arena-identical (the registry prices overhead flat up
|
||||
# to its nbt 4096 anchor), 12288 costs 2.5% plus arena.
|
||||
# The KV ask: the pool is priced at the engine's measured 7,892 B/token at
|
||||
# this shape, and a served conversation needs 200,704 tokens, so the pool
|
||||
# must clear 400,700 tokens to keep two full-length resident.
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
|
||||
base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE")
|
||||
per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE")
|
||||
[ "$base" -eq 0 ] && [ "$per_graph" -gt 0 ]
|
||||
[ $((base + 6 * per_graph)) -lt 563 ]
|
||||
kv=$(awk -F= '/^FRANK_PRIMARY_KV_BYTES=/{print $2}' "$ENV_FILE")
|
||||
[ $((kv / 7892)) -ge 400700 ]
|
||||
}
|
||||
|
||||
@test "expert prefetch is absent BY LAW, not by omission" {
|
||||
# Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
|
||||
# prefetched slabs (36% precision) — roughly half the gather traffic was
|
||||
# waste, and the "savings" would have been installs anyway. Deleting the
|
||||
# flag moved decode 26.8 -> 40.6 tok/s. Any future flag must come with a
|
||||
# fresh A/B, so the serve script must NOT wire it unconditionally.
|
||||
! grep -q -- '--expert-prefetch' "$SERVE"
|
||||
! grep -q 'expert_prefetch' "$ENV_FILE"
|
||||
grep -q 'PREFETCH IS ABSENT BY LAW' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "TunableOp stays off" {
|
||||
# CLAV_TUNABLEOP_SWEEP silently skips under expert offload unless
|
||||
# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics
|
||||
# (repetition loops, early halts, invalid measurements).
|
||||
! grep -q 'TUNABLEOP' "$ENV_FILE"
|
||||
! grep -q 'TUNABLEOP' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the serve script gates the digest-pinned image and the payload" {
|
||||
# The image is pinned by OCI manifest digest (the floating :latest moved
|
||||
# once already). The payload gate checks config.json and the exact shard
|
||||
# count from the env — a missing shard means a guaranteed long-timeout
|
||||
# boot instead of a fast failure.
|
||||
@test "the serve script gates the image, the payload and the GPU nodes" {
|
||||
# The image gate is EXISTENCE (podman image exists): the FreeToken
|
||||
# image is built ON THE HOST — a local tag carries no RepoDigests — and
|
||||
# pinning lives in the Dockerfile (pinned PR-head SHA, base image).
|
||||
# The payload gate checks config.json and the exact shard count from
|
||||
# the env — a missing shard means a guaranteed long-timeout boot
|
||||
# instead of a fast failure.
|
||||
grep -q 'FRANK_PRIMARY_IMAGE' "$SERVE"
|
||||
grep -q 'RepoDigests' "$SERVE"
|
||||
grep -q 'podman image exists' "$SERVE"
|
||||
grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$SERVE"
|
||||
grep -q 'config.json' "$SERVE"
|
||||
grep -q 'model.safetensors.index.json' "$SERVE"
|
||||
grep -q '/dev/kfd' "$SERVE"
|
||||
grep -q -- '--device /dev/dri' "$SERVE"
|
||||
! grep -q 'tcclaviger/vllm:latest' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve script clears stale containers and self-heals via podman wait" {
|
||||
# A crashed unit can leave the container name registered forever after;
|
||||
# the legacy identities (the retired R9V container and the ad hoc
|
||||
# experiment container) hold the same GPUs and die here too. Then block
|
||||
# on the container so a crash is a nonzero ExecStart.
|
||||
# the legacy identities (the vLLM-era frank-primary-qwen container, the
|
||||
# retired R9V container and the ad hoc experiment containers) hold the
|
||||
# same GPU and die here too. Then block on the container so a crash is
|
||||
# a nonzero ExecStart.
|
||||
grep -q 'FRANK_PRIMARY_LEGACY_CONTAINERS' "$ENV_FILE"
|
||||
grep -q 'frank-primary-qwen' "$ENV_FILE"
|
||||
grep -q 'frank-primary-r9v' "$ENV_FILE"
|
||||
grep -q 'qwen-fp8' "$ENV_FILE"
|
||||
grep -q 'exec podman wait "$FRANK_PRIMARY_CONTAINER"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-primary holds the single fleet port under its own alias" {
|
||||
# The scratch identity (qwen3.8-flash-next on :7773) must never leak
|
||||
# into production: LAN publish of :7770, the harness's model id.
|
||||
# LAN publish of :7770, the harness's model id.
|
||||
grep -q '^FRANK_PRIMARY_PUBLISH=7770:7770$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SERVED_NAME=frank-primary$' "$ENV_FILE"
|
||||
}
|
||||
@@ -404,23 +205,51 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the deployer gates the image digest and payload, then health-checks" {
|
||||
# The deployer refuses when the pinned image is absent from rootless
|
||||
# storage or the payload is missing/wrong-sharded — then clears stale
|
||||
# containers (including the legacy identities), waits out the port,
|
||||
@test "the Dockerfile pins the FreeToken source and follows the ROCm install law" {
|
||||
# The engine is the PR #587 head, NOT a moving branch: a future push to
|
||||
# rocm714_torch214 would otherwise silently change the deployed engine.
|
||||
grep -q '77a7201f98f00d93f618c6cf8b81ffe8360e0899' "$DOCKERFILE"
|
||||
grep -q 'pull/587/head' "$DOCKERFILE"
|
||||
# docs/install_amd.md's build law: --no-build-isolation --no-deps (a
|
||||
# plain `pip install -e .` on ROCm can REPLACE the ROCm torch with a
|
||||
# CUDA build via its NVIDIA-authored dependency set), curated portable
|
||||
# deps, flashlib --no-deps.
|
||||
grep -q -- '--no-build-isolation --no-deps -e .' "$DOCKERFILE"
|
||||
grep -q 'flashlib==0.3.0' "$DOCKERFILE"
|
||||
# The CUDA-only backends have no ROCm builds and are intentionally
|
||||
# absent (the PR's 12 pytest failures are exactly those backends;
|
||||
# FreeToken falls back to its Triton kernels).
|
||||
! grep -q 'flashinfer' "$DOCKERFILE"
|
||||
! grep -q 'sgl_kernel' "$DOCKERFILE"
|
||||
! grep -q 'nvidia-cutlass-dsl' "$DOCKERFILE"
|
||||
# The base image is digest-pinned (the registry tag is not trusted).
|
||||
grep -qE '^FROM rocm/pytorch:.*@sha256:[0-9a-f]{64}$' "$DOCKERFILE"
|
||||
}
|
||||
|
||||
@test "the deployer gates the image, payload and containers, then health-checks" {
|
||||
# The deployer refuses when the image is absent from rootless storage
|
||||
# or the payload is missing/wrong-sharded — then clears stale containers
|
||||
# (including the legacy identities), waits out the port,
|
||||
# relinks/enables/restarts the root unit and health-gates it.
|
||||
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary$' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
|
||||
grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
|
||||
grep -q 'podman image exists' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
|
||||
grep -q 'config.json' "$REMOTE_DEPLOY"
|
||||
grep -q 'frank-primary-qwen' "$REMOTE_DEPLOY"
|
||||
grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
|
||||
grep -q 'glm-probe' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-primary.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_for_health frank-primary http://127.0.0.1:7770/health 1800' "$REMOTE_DEPLOY"
|
||||
# The health gate reads the BODY, not just the HTTP code: FreeToken
|
||||
# returns 200 with {"status":"loading"} while the engine loads (the
|
||||
# FastAPI front end binds before the scheduler is up — a bare 200
|
||||
# false-greened a deploy at 6 s while 131 shards were still paging in,
|
||||
# measured 2026-10-06). Only status:"ok" counts.
|
||||
grep -q 'curl -fsS http://127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'never reached /health status ok' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deployer will not report green on a stray port-holder" {
|
||||
|
||||
@@ -315,31 +315,28 @@ PAIRS
|
||||
|
||||
@test "prometheus scrapes every LLM endpoint that exports metrics" {
|
||||
# The LLM endpoints are the services whose behaviour this fleet argues about
|
||||
# most (KV pressure, prefix-cache reuse, queue depth). All three export
|
||||
# the llamacpp:* family since 2026-09-26, when frank-primary (Dankserver)
|
||||
# went back to llama.cpp — the unsloth b11030-mix prebuilt serving
|
||||
# Qwen3.8-Flash-Next Q8_0 — after two days of the Lucebox DS4.1 engine,
|
||||
# which exported nothing (404) and had no job.
|
||||
# most (KV pressure, prefix-cache reuse, queue depth). Both Dankclaw
|
||||
# endpoints export the llamacpp:* family. frank-primary LEFT the set
|
||||
# 2026-10-05: the FreeToken PR587 ROCm engine has NO /metrics endpoint —
|
||||
# scraping it was a structural never-green (404), so the job was deleted
|
||||
# rather than left vacuous (the Lucebox DS4.1 precedent, 2026-09-26).
|
||||
# The two Dankclaw endpoints are HOST processes reached through
|
||||
# host.containers.internal: this Prometheus is a bridge-networked
|
||||
# container, and on this netavark host a container can only reach the
|
||||
# host's LAN IP at a PUBLISHED port — a host-process bind like
|
||||
# llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it
|
||||
# answers at 169.254.1.2:7770 (why frank-small stayed red by construction
|
||||
# for days until the 2026-09-20 /targets audit). frank-primary is a
|
||||
# DIFFERENT machine, so that quirk does not apply: plain LAN routing
|
||||
# reaches 192.168.0.235:7770, and the job must say so (a gateway target
|
||||
# would point at Dankclaw's own :7770 — frank-large — and double-scrape
|
||||
# it under the wrong label).
|
||||
# for days until the 2026-09-20 /targets audit).
|
||||
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
|
||||
grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
|
||||
grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
|
||||
grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-primary not scraped"; return 1; }
|
||||
! grep -q '"192.168.0.235:7770"' "$cfg"
|
||||
grep -q '"host.containers.internal:7770,192.168.0.235:7770"' "$cfg" \
|
||||
&& { echo "frank-primary folded into a Dankclaw gateway target"; return 1; }
|
||||
for job in frank-small frank-large frank-primary; do
|
||||
for job in frank-small frank-large; do
|
||||
grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
|
||||
done
|
||||
! grep -q 'job_name: "frank-primary"' "$cfg"
|
||||
# None of them is reached through a guarded vhost: Prometheus cannot solve
|
||||
# an Anubis challenge, so a franksdank.cloud target is red by construction.
|
||||
! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
|
||||
|
||||
Reference in new issue
Block a user