docs: distill the managed skill library into the vault (waves 1-7)
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
This commit is contained in:
Frank
@@ -0,0 +1,111 @@
|
||||
---
|
||||
type: reference
|
||||
title: Agent Docs and Edits
|
||||
description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
|
||||
tags: [agents, docs, markdown, hashline, edit-recovery]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Agent Docs and Edits
|
||||
|
||||
## Doc-structure audit (after parallel agent edits)
|
||||
|
||||
Use after one or more subagents have edited markdown docs (OKF/Obsidian
|
||||
vault with `---` frontmatter), before committing. Link checkers and
|
||||
completeness seams validate *references*, not *structure* — agents reliably
|
||||
produce structurally broken markdown that still links cleanly.
|
||||
|
||||
Damage catalogue:
|
||||
|
||||
1. Banner/prose prepended **above** the opening `---` → the frontmatter is
|
||||
no longer frontmatter (title/tags/description become body text).
|
||||
2. A rewrite **appended** instead of merged → duplicate headings and a stray
|
||||
second `---` mid-file.
|
||||
3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
|
||||
`+- `, `- - `.
|
||||
4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
|
||||
5. Line-range replacement that dropped a sentence tail → dangling fragment,
|
||||
often mid-parenthesis.
|
||||
|
||||
Checks (fast, run all):
|
||||
|
||||
```bash
|
||||
head -3 <file> # must be `---` if the page has frontmatter
|
||||
grep -c '^---$' <file> # exactly 2 (opener + closer)
|
||||
grep -n '^## ' <file> | sort -t: -k2 | uniq -f1 -d # duplicate sections
|
||||
grep -rnE '^\+- |^- - |<\|' <files> # artifacts
|
||||
python3 scripts/check-vault-links.py <vault-dir>
|
||||
./tests/run.sh # docs completeness seam etc.
|
||||
```
|
||||
|
||||
Repair order (matters):
|
||||
|
||||
1. If the file was destroyed rather than edited: stash the agent version to
|
||||
`.AGENTTRASH/<YYYYMMDD-topic>/`, then `git checkout HEAD -- <file>` and
|
||||
re-apply only the intended changes by hand.
|
||||
2. If it merely duplicated: **salvage unique content first** — cut the
|
||||
one-of-a-kind section, paste it into the copy you keep, THEN delete the
|
||||
duplicate ranges. Never delete first.
|
||||
3. Re-add missing sentence tails (insert before the orphaned fragment),
|
||||
keeping list continuation lines indented.
|
||||
4. Re-run all checks above.
|
||||
|
||||
Gotchas: agents can write AFTER they appear finished (job settles, file
|
||||
changes) — re-verify structure after the last background job reports, not
|
||||
before. Editing an agent's already-damaged file with fresh line numbers is
|
||||
safe, but its snapshot tag changes on every edit — re-read before each
|
||||
subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so
|
||||
nothing moved there reaches the repo — that is intended (temporary hold, not
|
||||
archive).
|
||||
|
||||
## Hashline edit recovery
|
||||
|
||||
Failure mode: multi-op edits resolve every `PUT N.=M:` / `CUT N.=M:` index
|
||||
against the **original** file numbering, and the post-apply diff view
|
||||
**elides lines** (it shows an anchor window, not the whole file). Two traps
|
||||
follow:
|
||||
|
||||
1. A second edit call built from a diff-view snapshot uses line numbers from
|
||||
a *partial* view. Lines outside the window are invisible but still
|
||||
counted.
|
||||
2. If the first apply was itself wrong, the file shifted. The diff view
|
||||
tells you the damage, but not the true offsets.
|
||||
|
||||
Result: the repair pastes fresh text **beside** stale text instead of over
|
||||
it.
|
||||
|
||||
Procedure:
|
||||
|
||||
1. **Never carry line numbers across calls.** Re-read the target range for a
|
||||
fresh tag before every subsequent edit. A read is cheap; a duplicated
|
||||
paragraph costs a commit and a revert.
|
||||
2. **One op per call when the previous apply was imperfect.** Fixed offsets
|
||||
are only trustworthy when every preceding apply landed exactly as
|
||||
specified.
|
||||
3. **Re-read after every apply**, and verify the whole affected region — the
|
||||
diff view is a sample, not proof. Check the *boundaries*: the line before
|
||||
the edit and the line after it. Damage shows up as a repeated sentence
|
||||
straddling the seam.
|
||||
4. **Repair by CUT, not by re-PUT.** Once you know the exact stale lines
|
||||
from a fresh read, delete them. Do not try to PUT a corrected superset
|
||||
over a range whose extent you are guessing at.
|
||||
5. Verify with a content assertion, not a diff review: `grep -c` for the
|
||||
duplicated sentence, plus a syntax/compile check on any touched script.
|
||||
|
||||
Same rule for any imperative `edit` tool with an old/new match: the matched
|
||||
text must be the **entire** text being replaced. Matching a bare fragment
|
||||
inside a paragraph splices the replacement into the middle of the sentence
|
||||
and silently corrupts the file. Re-read the file afterwards to confirm.
|
||||
|
||||
### Anchored in-page links
|
||||
|
||||
Renaming a Markdown heading breaks `[[#Old Heading]]` anchors. When a
|
||||
section is retitled, grep for the old heading text and fix every anchor in
|
||||
the same pass — doc link-checkers frequently do not validate `#fragment`
|
||||
targets, so a green suite will not catch a dangling one.
|
||||
|
||||
## Related
|
||||
|
||||
- [[OMP Harness]] — the session context these repairs protect
|
||||
- [[Command Center Dotfiles]] — the repo these edits commit to
|
||||
- [[Testing Policy]] — the seams that catch broken vault structure
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
type: reference
|
||||
title: Command Center Dotfiles
|
||||
description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
|
||||
tags: [dotfiles, omp, deploy, rsync, blankpad]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Command Center Dotfiles
|
||||
|
||||
Repo `~/.omp` is itself the deployment: packages are home-relative trees
|
||||
under `dotfiles/<pkg>/`; `scripts/deploy.sh` symlinks live files into the
|
||||
repo checkout. `git pull` + `deploy.sh` is the whole propagation story.
|
||||
|
||||
## Transport decision (agreed with Frank, executed 2026-09-09)
|
||||
|
||||
- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to
|
||||
`<user>@<host>:/home/<user>/.omp/`, NOT clone-from-Gitea — `main` can be
|
||||
ahead of `origin/main`; push is a user-only trigger, so clone would ship
|
||||
stale state.
|
||||
- Deploy runs on-target as the target user: `bash
|
||||
/home/<user>/.omp/scripts/deploy.sh` (resolves the repo via `git -C
|
||||
$(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be
|
||||
present at the target path — full-replica transport guarantees this).
|
||||
|
||||
## Pre-overwrite audit (MANDATORY before first sync on a non-blank host)
|
||||
|
||||
Diff every file that `deploy.sh --check` reports as DRIFT against the repo
|
||||
copy BEFORE letting quarantine-and-replace run. Host-specific divergence
|
||||
belongs in an override package, not in quarantine:
|
||||
|
||||
1. Pull divergent live files from target
|
||||
(`ssh ... 'cat <path>' > local-staged-copy`).
|
||||
2. Secret-audit staged copies (grep token/secret/password/key shapes)
|
||||
before anything enters git.
|
||||
3. Compose `dotfiles/zz-<hostname>/` on the source machine from those live
|
||||
copies; commit to repo first.
|
||||
4. Ship the package dir via rsync, then run deploy sync — override deploys
|
||||
LAST and win over base packages.
|
||||
|
||||
## deploy.sh mechanics (verified on blankpad)
|
||||
|
||||
- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it
|
||||
first, always.
|
||||
- Default mode = sync: adopt identicals silently, link new,
|
||||
quarantine-and-replace divergents into `.quarantine/<host>/<ts>/`
|
||||
(gitignored; never rm).
|
||||
- **Override-target bug (fixed cc2c652):** `zz-<host>` packages nest
|
||||
base-package dirs internally (`zz-blankpad/niri/.config/...`); target
|
||||
derivation must strip the nested package segment, else overrides deploy to
|
||||
phantom `$HOME/niri/.config/...` while real targets keep pointing at
|
||||
canonical copies (visible symptom: check shows 0 actions but
|
||||
`readlink ~/.zshrc` points at base `zsh/.zshrc`).
|
||||
- systemd units auto-enable only when ExecStart binary resolves on that
|
||||
host; dconf packages load via `dconf load /`, never symlinks.
|
||||
- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once
|
||||
post-transfer even before first sync.
|
||||
|
||||
## ssh/sshpass transport gotchas (all bit live)
|
||||
|
||||
- Resolve the target via mDNS FQDN (`avahi-resolve -4 -n <host>.local`); a
|
||||
bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or
|
||||
firewall default-drop, not auth failure — needs a local/console fix on
|
||||
target.
|
||||
- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; an env passed
|
||||
via the tool's env parameter did not propagate to the sshpass child
|
||||
process.
|
||||
- The remote login shell may be **fish**: never send bash syntax
|
||||
(`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash -c
|
||||
"..."` or ship a script. See [[SSH and Remote Access]].
|
||||
- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A
|
||||
here-doc attached to the ssh command is consumed by `cat` — the intended
|
||||
script body becomes the file content, silently. Verify shipped bytes with
|
||||
an md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat`
|
||||
drained it (false DIVERGED).
|
||||
|
||||
## Verification of a completed deploy
|
||||
|
||||
Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped
|
||||
= idempotent steady state. Spot-check `readlink` on override files points
|
||||
into `dotfiles/zz-<host>/`, not base packages.
|
||||
|
||||
## Related
|
||||
|
||||
- [[SSH and Remote Access]] — the transport mechanics this deploy rides
|
||||
- [[Noctalia and Niri]] — the desktop packages this repo ships
|
||||
- [[Deploy State Triage]] — the serverAppConfig deploy flow (different repo)
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
type: reference
|
||||
title: OMP Harness
|
||||
description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
|
||||
tags: [omp, harness, compaction, models-yml, context, samplers]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# OMP Harness
|
||||
|
||||
The harness reaches the fleet through `~/.omp/agent/models.yml` (model
|
||||
registry; single source of truth — there is no copy on the servers) and
|
||||
`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
|
||||
this file is repo-managed; edits are harness-side only.
|
||||
|
||||
## Compaction chain arithmetic (verify before ANY window change)
|
||||
|
||||
When a fleet endpoint's context grows past its harness-declared window, OMP
|
||||
compacts by sending the conversation to a designated target model. Before
|
||||
changing any `contextWindow`/`maxTokens`, verify the chain arithmetically —
|
||||
a target whose window is smaller than the source's trigger point silently
|
||||
breaks compaction.
|
||||
|
||||
Where the contract lives:
|
||||
|
||||
- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL),
|
||||
`methodOrder`, `thresholdPercent` (70), idle settings
|
||||
- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
|
||||
`compactionModel` references
|
||||
- Implementation: omp plugin `dist/core/compaction/compaction.js`
|
||||
|
||||
The arithmetic (the coupled contract every endpoint's maxTokens comment
|
||||
encodes):
|
||||
|
||||
1. **Source trigger** = `thresholdPercent × source contextWindow`
|
||||
(0.70 × 200,704 = 140,492 for frank-primary on the current shape)
|
||||
2. **Output budget** = capped at the target model's `maxTokens`
|
||||
(32,768 for frank-small; 36,864 frank-primary; 57,344 frank-large — the
|
||||
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
|
||||
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
|
||||
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
|
||||
200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
|
||||
(frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-small).
|
||||
|
||||
Procedure (recompute from the LIVE files every time — windows rotate with
|
||||
every swap; the numbers above are the 2026-10-05 state):
|
||||
|
||||
1. Read the `config.yml` compaction block + `models.yml` entries for source
|
||||
and target. Every endpoint's `contextWindow` comment carries its own
|
||||
arithmetic — read them.
|
||||
2. Read the target checkpoint's `max_position_embeddings` (native window) —
|
||||
the served window may be a concurrency contract, not the native limit
|
||||
(`--parallel N` DIVIDES the pool; the per-slot number is the contract).
|
||||
3. If native ≥ trigger + budget, reslice the pinned pool:
|
||||
`MAX_RUNNING × CONTEXT_LENGTH ≤ MAX_TOTAL_TOKENS` (the pin exists because
|
||||
the runtime's autofit leaves too little for deferred loads + prefill
|
||||
workspace; check `nvidia-smi` headroom — expect ~0.8 GB workspace margin).
|
||||
4. Changes: host env (CONTEXT_LENGTH-style env tunables) via git →
|
||||
`./tests/run.sh` → deploy; harness `models.yml` directly (no repo copy).
|
||||
5. Verify: `/health` 200, boot log's pool line, then a long-prompt probe at
|
||||
≥ trigger height (the compaction-fit guarantee), then concurrent requests.
|
||||
|
||||
Known reconciliation trap: comments in models.yml have described compaction
|
||||
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
|
||||
dankclaw — the comment/config disagreed. The comments now name the real
|
||||
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
|
||||
truth; fix comments in the same change.** `maxTokens` never changes for a
|
||||
reslice — it caps output; `contextWindow` caps input. Boot-shape OOM
|
||||
laddering is a different problem ([[Llama Cpp Serving Sizing]]).
|
||||
|
||||
## Per-model sampler config (`compat.extraBody`)
|
||||
|
||||
`models.yml` has **no per-model sampler keys**. Sampling is global
|
||||
(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
|
||||
`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is
|
||||
`compat.extraBody`:
|
||||
|
||||
**Why it works:** `applyOpenAIExtraBody` runs **last** in the
|
||||
openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
|
||||
`Object.assign(params, extraBody)`. So extraBody overrides global settings
|
||||
and compat-policy output.
|
||||
|
||||
```yaml
|
||||
compat:
|
||||
reasoningEffortMap:
|
||||
max: xhigh
|
||||
extraBody:
|
||||
temperature: 1.0
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
min_p: 0.0
|
||||
presence_penalty: 0.0
|
||||
repetition_penalty: 1.0
|
||||
preserve_thinking: true
|
||||
```
|
||||
|
||||
Use **wire names**, not OMP camelCase. Two traps:
|
||||
|
||||
1. **`preserve_thinking` is not user-settable as a compat key.**
|
||||
`qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
|
||||
*auto-detected* `thinkingFormat` + local baseUrl, **before**
|
||||
`applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
|
||||
models.yml does **not** enable it. Custom-named models (`frank-large`)
|
||||
match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
|
||||
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
|
||||
also sets it — `Object.assign` is shallow and replaces the whole object,
|
||||
silently dropping the `reasoning_effort` kwarg the compat policy set.
|
||||
Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
|
||||
FreeToken entry currently carries a `chat_template_kwargs` extraBody with
|
||||
a comment asking to verify-and-prune — the exact trap to check at that
|
||||
endpoint's next swap.)
|
||||
|
||||
**Gotcha: silent greedy decode.** halogen has **no server-side sampler
|
||||
defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
|
||||
endpoint decodes greedy while looking configured. This is why frank-large's
|
||||
entry carries the recipe on the wire and a comment saying "do not drop it".
|
||||
|
||||
**Reasoning-effort folding.** Qwen3.8's template accepts only
|
||||
`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
|
||||
else ("Unexpected reasoning effort high"). Every endpoint's
|
||||
`reasoningEffortMap` folds all harness levels onto that set; a role whose
|
||||
level is unmapped fails on its FIRST request, and the failure names Jinja,
|
||||
not the model.
|
||||
|
||||
### Verify the exact wire body (echo-capture)
|
||||
|
||||
Don't trust the YAML — capture what OMP actually emits:
|
||||
|
||||
1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and
|
||||
returns a canned `chat.completion` (include `reasoning_content` so
|
||||
thinking paths exercise).
|
||||
2. Copy the **real** models.yml into a throwaway profile with only the
|
||||
baseUrl swapped:
|
||||
```bash
|
||||
sed 's|http://<host>:7770/v1|http://127.0.0.1:18099/v1|' \
|
||||
~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml
|
||||
```
|
||||
3. Run and inspect:
|
||||
```bash
|
||||
omp --profile wiretest --model "<provider>/<model>" -p "hi" \
|
||||
--no-tools --no-session --no-extensions --no-skills --no-rules
|
||||
jq 'del(.messages, .tools)' last-body.json
|
||||
```
|
||||
Confirm every intended field is present with the right value.
|
||||
|
||||
**Verify a field is *honoured*, not just accepted** — unknown fields often
|
||||
return 200 anyway. Prove behavior with a **prompt-token delta**: replay an
|
||||
assistant turn carrying a long `reasoning_content`, then compare
|
||||
`usage.prompt_tokens` with the flag true vs false. A large delta means the
|
||||
field changed template rendering; zero delta means it was ignored.
|
||||
|
||||
Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
|
||||
never `rm`.
|
||||
|
||||
## Static-preamble audit ("the context feels bloated")
|
||||
|
||||
What loads statically every session (measure each, don't guess):
|
||||
|
||||
1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k
|
||||
tokens — historically the largest block).
|
||||
2. Skill index — only names+descriptions load; bodies are on-demand. Sum
|
||||
description bytes across all SKILL.md frontmatter:
|
||||
```bash
|
||||
python3 - <<'EOF'
|
||||
import glob, re, os
|
||||
for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
|
||||
os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
|
||||
n=tot=0
|
||||
for sk in glob.glob(root+"/*/SKILL.md"):
|
||||
fm=re.search(r'^---.*?---', open(sk, encoding="utf-8", errors="replace").read(), re.S)
|
||||
d=re.search(r'description:\s*(.*(?:\n(?:\s{2,}.*)*)?)', fm.group(0) if fm else "")
|
||||
n+=1; tot+=len(d.group(1)) if d else 0
|
||||
print(root, n, tot)
|
||||
EOF
|
||||
```
|
||||
Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user
|
||||
skills.
|
||||
3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name
|
||||
'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest
|
||||
MEMORY.md.
|
||||
4. Harness/system prompt + tool schemas + CLI persona: fixed cost, ~20 KB;
|
||||
not prunable.
|
||||
|
||||
Known rot patterns:
|
||||
|
||||
- Near-duplicate auto-captures of the same lesson (three "authentik ORM"
|
||||
entries, the freetoken-build pair, the three job-runner skills) — dedupe
|
||||
via `manage_skill delete`/`update`.
|
||||
- Corrupted captures containing stray CJK characters (`一团ly`, `等方法`)
|
||||
mid-sentence — a known defect of the auto-capture path; delete the corrupt
|
||||
copy, keep the clean one.
|
||||
- Stale host/service names in consolidated memory (pre-rename
|
||||
`frank-default-large`) — the repo's AGENTS.md topology table is ground
|
||||
truth.
|
||||
|
||||
Fix strategy: AGENTS.md prose (measured runbook narratives) → move to
|
||||
`serverAppConfigs-docs/LLM/` runbooks, keep laws + tables in AGENTS.md (the
|
||||
repo's own docs-tree rule mandates this). Managed skills live in
|
||||
`~/.omp/agent/managed-skills` (safe to edit); user-authored skills in
|
||||
`~/.omp/agent/skills` are NEVER touched by tooling. Memory store edits:
|
||||
stage removed captures to trash first (`~/.AGENTTRASH/YYYYMMDD-topic`).
|
||||
|
||||
Baseline to beat (2026-10-05): ~35–40k tokens of preamble — AGENTS.md ~18k,
|
||||
skill index ~10k, lessons ~3.5k, harness ~6k. Proposed reductions: −12k
|
||||
(AGENTS.md slimming), −3k (skill dedupe).
|
||||
|
||||
## Managed-skill library hygiene
|
||||
|
||||
Use when auditing, pruning, or consolidating `~/.omp/agent/managed-skills`
|
||||
(the auto-captured skill store; the full index is injected into EVERY
|
||||
session's context, so dead or duplicate entries are a permanent tax).
|
||||
|
||||
The static-context budget (measured 2026-10-05):
|
||||
|
||||
| Block | Size | Notes |
|
||||
|---|---|---|
|
||||
| `AGENTS.md` (repo) | 72,179 B ≈ 18k tok | biggest single item |
|
||||
| Skill index (~173 skills) | ~40 KB ≈ 10k tok | names+descriptions only |
|
||||
| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures |
|
||||
| Harness system prompt + tool schemas | ~23 KB | fixed |
|
||||
|
||||
Audit procedure:
|
||||
|
||||
1. Build an inventory: per skill, name, mtime, body word count, description,
|
||||
first 12 headings (one pass, out to a file; read in pages).
|
||||
2. Group into: retired-runtime (host/service gone; its runbook already lives
|
||||
in `serverAppConfigs-docs/LLM/`), verbatim-duplicate pairs/triples (same
|
||||
lesson captured twice — often the SAME DAY), same-lesson families
|
||||
(N skills = one procedure), keep.
|
||||
3. Read full bodies of every merge candidate before proposing a merge;
|
||||
description alone proves overlap only for exact dupes.
|
||||
4. Present A/B/C/D verdicts for user approval. NEVER execute before the
|
||||
discussion concludes — the user gates execution explicitly.
|
||||
5. On approval: stage everything to trash first
|
||||
(`~/.AGENTTRASH/YYYYMMDD-<topic>/`), never rm. After merging: recreate
|
||||
the surviving skill via `manage_skill` update with body = superset,
|
||||
delete absorbed ones.
|
||||
|
||||
Known rot classes (this library, 2026-10-05):
|
||||
|
||||
- Same-day duplicate captures (freetoken-build-dankclaw vs
|
||||
freetoken-dankclaw-build; the three job-runner skills).
|
||||
- CUDA-skill vs its self-declared HIP twin (pcie-link-capability-probe vs
|
||||
pcie-link-probe-hip-rdna4) → merge.
|
||||
- Era ghosts: skills naming retired runtimes (freetoken, lucebox, r9v/
|
||||
radiance, old halogen-flash-server image, ik_llama, GLM-5.3 experiment).
|
||||
- Skills contradicting user standing orders (live-bench-dashboard vs the
|
||||
2026-09-29 delete-dashboards order) — flag for user, don't assume.
|
||||
- Memory store: corrupted captures with CJK noise from broken ssh commands;
|
||||
stale host/service renames.
|
||||
|
||||
Sizing verdicts that worked: bench family → 2 skills (harness vs
|
||||
discipline), not one mega-skill; keep profiler/kernel-time attribution
|
||||
standalone; fold go/no-go arithmetic skills into their probe skill
|
||||
(heterogeneous-accelerator-split-triage → PCIe probe).
|
||||
|
||||
## Related
|
||||
|
||||
- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
|
||||
- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
|
||||
- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
|
||||
- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
type: index
|
||||
description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
|
||||
tags: [agent, index]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Agent
|
||||
|
||||
Pages about the agent's own tooling and context — how omp is configured, how
|
||||
docs get edited safely, and how the agent's home is deployed.
|
||||
|
||||
- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
|
||||
`compat.extraBody`, preamble and skill-library audits
|
||||
- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
|
||||
hashline edit recovery
|
||||
- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
|
||||
transport, override packages
|
||||
|
||||
## Related
|
||||
|
||||
- [[Testing Policy]] — the seams these pages' checks ride
|
||||
- [LLM](../LLM/index.md) — the fleet these agents operate
|
||||
@@ -0,0 +1,151 @@
|
||||
---
|
||||
type: reference
|
||||
title: Deploy State Triage
|
||||
description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
|
||||
tags: [deployment, deploy-state, systemd, podman, triage]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Deploy State Triage
|
||||
|
||||
The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||||
on each target (rootless, user `dank`, format exactly two lines
|
||||
`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
|
||||
+ `mv`). It is written ONLY by a `deploy.sh` dispatch.
|
||||
|
||||
## 1. Stale marker vs live container
|
||||
|
||||
**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
|
||||
a restart after the checkout ff-pulled to a newer revision — starts the
|
||||
container from the CURRENT `scripts/<svc>.env` but does NOT advance the
|
||||
marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
|
||||
"behavior pending".
|
||||
|
||||
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
||||
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
|
||||
a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
|
||||
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
||||
possibly zero behavioral change.
|
||||
|
||||
**Procedure (read-only first):**
|
||||
|
||||
1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
|
||||
operational delta is usually one `scripts/<svc>.env`.
|
||||
2. Probe the target (ssh as `dank`, passwordless sudo present):
|
||||
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||||
- `git -C ~/serverAppConfigs rev-parse HEAD`
|
||||
- `systemctl is-active/is-enabled <svc>`;
|
||||
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
|
||||
- the on-host env file already carries the new keys (it landed via the
|
||||
ff-pull).
|
||||
3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
|
||||
`dank` (rootless store; bare `docker` is NOT the shim → "No such object";
|
||||
`sudo podman` inspects an empty store):
|
||||
- `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
|
||||
the pull ⇒ it read the new env
|
||||
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
|
||||
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
|
||||
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
|
||||
- confirm the launcher maps each tunable to its flag
|
||||
(`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
|
||||
--enable-prefix-caching`.
|
||||
|
||||
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
|
||||
live — **do NOT restart**; reconcile the marker (below). Only if the live
|
||||
container lacks the flag/env (or predates the pull) is a real
|
||||
`deploy.sh <target> --service <svc>` warranted.
|
||||
|
||||
**Reconcile (no restart)** — mirror the deployer's atomic idiom:
|
||||
|
||||
```bash
|
||||
S=$HOME/.local/state/serverAppConfigs/deploy-state
|
||||
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
|
||||
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
|
||||
```
|
||||
|
||||
Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
|
||||
`tests/e2e/.env.local`; expect the documented `11-admin` red if the
|
||||
FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
|
||||
a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
|
||||
Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
|
||||
no-op).
|
||||
|
||||
**Anti-patterns:** trusting a commit message ("X deployed") or the marker
|
||||
alone — both are bookkeeping; the container's argv/env is ground truth.
|
||||
Reading container state with `sudo podman` / bare `docker`. Restarting a
|
||||
healthy heavy endpoint to satisfy a stale marker.
|
||||
|
||||
## 2. Pending lockout
|
||||
|
||||
Symptom: `deploy.sh <target>` fails or silently stops with
|
||||
`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
|
||||
appears.
|
||||
|
||||
1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
|
||||
reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
|
||||
`echo $?`.
|
||||
2. Inspect the state store (no sudo):
|
||||
`ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
|
||||
A `status=pending` entry for a service absent from `service_is_allowed`
|
||||
in `scripts/deploy-remote.sh` = retired-service lockout.
|
||||
3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
|
||||
not-allowed pending entries to
|
||||
`~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
|
||||
continues. Verify the WARN line names the service and the state file is
|
||||
gone.
|
||||
4. On a host without the fix: move the stale entry to
|
||||
`~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
|
||||
5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
|
||||
`deploy-state/` no longer holds the retired name.
|
||||
|
||||
Related contracts: the superseded seam
|
||||
(`deployment_scoping.bats` "remote runner recognizes a pending service at
|
||||
the same revision") now asserts prune-and-continue (exit 0), not exit 2;
|
||||
`deployment_flow.bats` extracts `add_pending_services` via sed and
|
||||
HOME-overrides the trash path. **When retiring a service going forward:**
|
||||
delete or reconcile its deploy-state entry as part of the retirement commit —
|
||||
the prune is a safety net, not the intended cleanup path.
|
||||
|
||||
## 3. Interrupted or raced deploys
|
||||
|
||||
The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
|
||||
(runs on the target over ssh). **Killing the dev-side job does NOT kill the
|
||||
host-side one** — two `deploy-remote.sh` processes can end up managing the
|
||||
same unit, one deploying a revision you already superseded. Check:
|
||||
`pgrep -af "deploy-remote"` on the target, read the revision argument on
|
||||
each, kill stale ones by explicit PID.
|
||||
|
||||
A killed deploy leaves behind:
|
||||
|
||||
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
|
||||
explicit stop — start it yourself);
|
||||
- the state marker **pending** (pre-deploy or in-flight revision);
|
||||
- the e2e gate un-run (the deployer runs it last).
|
||||
|
||||
Reconcile the marker ONLY after the service itself is verified healthy — a
|
||||
marker that runs ahead of reality is worse than one that lags (the planner
|
||||
uses it to decide what is pending). Then close the loop:
|
||||
|
||||
1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
|
||||
slots, argv flags, `NRestarts`), not just health 200.
|
||||
2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
|
||||
that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
|
||||
does NOT have it; copy it in first).
|
||||
3. If the killed revision was restoring a known-good shape, confirm the
|
||||
numbers match the last measured-good boot of that shape — equality is the
|
||||
proof, not health alone.
|
||||
|
||||
Boot failures must be read from the journal, not `podman logs`: serve scripts
|
||||
`podman rm -f` the container on every start (previous attempt lost), and
|
||||
`podman logs` on a container still *Stopping* returns the OLD configuration's
|
||||
numbers — which looks exactly like your change having no effect. Key on
|
||||
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
|
||||
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
|
||||
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
|
||||
procedure.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Deployment Flow]] — the deploy pipeline itself
|
||||
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
|
||||
@@ -0,0 +1,209 @@
|
||||
---
|
||||
type: reference
|
||||
title: Quadlet Lifecycle
|
||||
description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
|
||||
tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Quadlet Lifecycle
|
||||
|
||||
Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
|
||||
working container MUST have a self-healing, boot-persistent Quadlet unit —
|
||||
`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
|
||||
deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
|
||||
A container that runs only from compose dies silently at reboot. This page
|
||||
is the operating manual for that lifecycle.
|
||||
|
||||
## 1. Ownership audit — find the containers that will not survive a reboot
|
||||
|
||||
```bash
|
||||
for c in $(podman ps --format '{{.Names}}'); do
|
||||
printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
|
||||
done
|
||||
```
|
||||
|
||||
- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
|
||||
show-user <user> -p Linger` = yes.
|
||||
- **label absent, compose service exists** → the classic gap: dies at reboot.
|
||||
Check the publish address while there — compose forms often bind the
|
||||
wildcard (an unauthenticated UI/API on every interface).
|
||||
- **label present AND a compose service with the same name** → two owners; a
|
||||
stray `podman compose up` can create a duplicate that fights the unit for
|
||||
the name and the port. Remove the compose service, leave an explanatory
|
||||
comment (repo convention).
|
||||
|
||||
## 2. Cutover: compose service → Quadlet unit
|
||||
|
||||
Checklist (precedents to copy:
|
||||
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
|
||||
`dankclaw/observability/quadlet/`):
|
||||
|
||||
1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
|
||||
line for line: `Image=` (`localhost/` prefix for locally built images +
|
||||
`Pull=never`), `ContainerName=` (Caddy and scrape configs address
|
||||
containers by name), `Network=`, `User=0:0` when the image's own user
|
||||
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
|
||||
`Exec=` for compose `command:`, `PublishPort=`, health keys,
|
||||
`[Service] Restart=always` + `RestartSec=10`, `[Install]
|
||||
WantedBy=default.target`. **Never `systemctl --user enable` the generated
|
||||
unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
|
||||
is healthy; explicit enable fails with "transient or generated").
|
||||
2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
|
||||
it (Caddy reaches containers by name over the shared network; the wildcard
|
||||
publish is a guard-chain bypass).
|
||||
3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
|
||||
`systemctl --user restart <name>.service` → `wait_for_health`; retire any
|
||||
legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
|
||||
`~/.config/systemd/user/<svc>.service` that shells out to
|
||||
`podman compose up` collides BY NAME with the generated unit);
|
||||
`podman rm -f <container>` so the unit's container can take the name.
|
||||
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
|
||||
AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
|
||||
ordered. Add a `test-changed` case above the generic
|
||||
`*/quadlet/*.container` fallback.
|
||||
5. Retire the compose entry in the SAME change — two owners is how a stale
|
||||
duplicate comes back.
|
||||
6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
|
||||
wildcard publish, container name matches Caddy/scrape references, health
|
||||
command, deployer installs+restarts+waits, never `rm`, never `enable`.
|
||||
Migrate any older assertion that read the compose file for devices/flags.
|
||||
7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
|
||||
active/generated, container `healthy`, endpoint answers,
|
||||
`list-dependencies default.target` shows the unit, off-host probes on
|
||||
previously published ports now refuse, and the container no longer carries
|
||||
`io.podman.compose.project`.
|
||||
|
||||
**Traps that cost real time:**
|
||||
|
||||
- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
|
||||
no shell → the check fails forever. `--health-cmd` won't take exec form
|
||||
either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
|
||||
with `--format docker` — the default OCI format has no HEALTHCHECK field
|
||||
and drops it with only a warning.
|
||||
- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
|
||||
gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
|
||||
the unit dies with `stat /root/.config: permission denied`. Use a
|
||||
per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
|
||||
handler with a single-variable filter, validated so a missing secret fails
|
||||
loudly.
|
||||
- `Network=host` exporters scraped by containerized Prometheus must bind the
|
||||
host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
|
||||
for the homepage probe — two `--web.listen-address` flags, never the
|
||||
wildcard.
|
||||
- alloy 1.x refuses the `ui` config block; the unit must pass
|
||||
`--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
|
||||
- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
|
||||
no supplementary groups).
|
||||
- When the last service leaves a compose file, the compose-validity seam
|
||||
fails (requires ≥1 service): keep the file as comments and relax the seam
|
||||
deliberately, or delete the file and fix the planner's compose-diff path —
|
||||
decide with the owner, the deploy tooling reads it.
|
||||
|
||||
## 3. Drift / crash-loop diagnosis (5 minutes, in order)
|
||||
|
||||
Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
|
||||
looks right; or a service quietly down for days because a restart policy kept
|
||||
something running. Cause: the running container was created from an OLDER
|
||||
definition — a compose/unit edit does not touch an existing container, and
|
||||
the restart policy faithfully resurrects it forever.
|
||||
|
||||
1. **Who started it:** `podman inspect <name> --format 'unit={{index
|
||||
.Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
|
||||
created={{.Created}}'`; compare `created` with when the definition changed
|
||||
(`git log -p --follow <config>`). Created-before-the-edit is the signature.
|
||||
2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
|
||||
`{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
|
||||
for containers started with `--device`/`--group-add` — a false negative
|
||||
that sends you hunting the wrong root cause.
|
||||
3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
|
||||
(a capability error like `No HIP GPUs are available` beside a missing
|
||||
device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
|
||||
instance reached 9,146 restart attempts on stale config.
|
||||
|
||||
Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
= the §2 cutover. **Verify the capability, not the status:** `podman exec
|
||||
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
|
||||
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
|
||||
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
|
||||
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
|
||||
|
||||
## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
|
||||
|
||||
- **Hard-fail keys** — the whole unit silently fails to generate
|
||||
(`systemctl --user cat <unit>` shows "could not be found"): any
|
||||
dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
|
||||
in group 'Container'`); `Command=` is unsupported, use `Exec=`.
|
||||
- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
|
||||
spaces into the healthcheck argv; a quoted probe
|
||||
`/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
|
||||
command-not-found → container reads **unhealthy while the DB is up**.
|
||||
HealthCmd must contain only space-free tokens
|
||||
(`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
|
||||
unhealthy but logs look fine, compare the baked
|
||||
`{{.Config.Healthcheck.Test}}` array against the intended command before
|
||||
believing the status.
|
||||
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
|
||||
`daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
|
||||
"unsupported key"`, `systemctl --user cat probe.service` (shows the
|
||||
generated ExecStart verbatim), then remove + reload.
|
||||
|
||||
## 5. Rootless Quadlet log retrieval
|
||||
|
||||
Step 0 — establish ownership: `systemctl --user status <name>.service`
|
||||
(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
|
||||
`--user` knows the unit, EVERY command runs as that user with NO sudo.
|
||||
|
||||
| Need | Command |
|
||||
|---|---|
|
||||
| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
|
||||
| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
|
||||
| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
|
||||
| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |
|
||||
|
||||
Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
|
||||
recreation** — the journal is the only durable record. Traps:
|
||||
`sudo podman logs` → "no such container" (root's store does not contain
|
||||
rootless containers; same for `sudo podman ps`); a system-level
|
||||
`journalctl -u <name>` returns nothing for a user unit; during first-start
|
||||
weight downloads health reports unhealthy BY DESIGN (check the payload file
|
||||
is growing). Verbose vendor switches are a config change — through git and
|
||||
deploy.sh, never a live env poke.
|
||||
|
||||
## 6. Compose bring-up with ground truth (rootless hosts)
|
||||
|
||||
1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
|
||||
check every namespace (`sudo podman ps -a`) and native units
|
||||
(`systemctl list-units`) before declaring a service absent. Compare
|
||||
compose mount sources against the real filesystem — compose files name
|
||||
phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
|
||||
`/mnt/entropy`). Fix compose through git first, deploy second.
|
||||
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
|
||||
or creation fails `statfs …: no such file or directory`.
|
||||
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
|
||||
`{{.State.Health.Status}}`, then the rest. Fresh containers answer
|
||||
000/refused for the first seconds — poll.
|
||||
4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
|
||||
request for nonexistent/private repos ("manifest unknown"/token len 0 =
|
||||
repo or tag does not exist, not auth). Query the registry API
|
||||
(`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
|
||||
`/v2/<ns>/<repo>/tags/list`).
|
||||
5. **Read the interface from inside the image** before guessing flags/envs:
|
||||
`podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
|
||||
means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
|
||||
entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
|
||||
flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
|
||||
configured it serves cwd read/write for everyone (an unauthenticated
|
||||
world-writable share).
|
||||
6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
|
||||
mangled pattern is not absence — re-verify the target exists.
|
||||
7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
|
||||
(host 1000 == container 0): drop the user line, run as container-root.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Deployment Flow]] — the deployer that installs these units
|
||||
- [[Stale Signal Audit]] — the audit that catches drift's victims
|
||||
- [[Container Network Namespaces]] — reachability between containers
|
||||
- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
|
||||
- [[Infrastructure/Observability]] — same class, twice
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
type: reference
|
||||
title: Stale Signal Audit
|
||||
description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
|
||||
tags: [testing, observability, audit, signals, gates]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Stale Signal Audit
|
||||
|
||||
Run after any change and before trusting a green gate. The productive question
|
||||
is **which signal here cannot possibly be correct?** Every pass on this repo
|
||||
(4 of 4) found real defects — including one in the rule file every future
|
||||
agent reads first.
|
||||
|
||||
## The surfaces, and what each vantage point can reach
|
||||
|
||||
| Surface | Producer's vantage point | Can reach |
|
||||
|---|---|---|
|
||||
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
|
||||
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
|
||||
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
|
||||
|
||||
Anything else — another host's loopback, a container-internal port never
|
||||
published — is unreachable no matter how healthy the service is. The failure
|
||||
mode is always the same shape: a signal that **cannot** succeed by
|
||||
construction, which teaches the operator to ignore the surface it lives on.
|
||||
|
||||
## 1. Permanently-red probes (reachability)
|
||||
|
||||
For every monitor/probe/alert target ask: *can this checker physically reach
|
||||
that endpoint?* Seen classes: a card probing another host's container-internal
|
||||
port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
|
||||
`/metrics` that is disabled (Gitea 404).
|
||||
|
||||
- A scrape cannot express "is this HTTP service answering?" — the right probe
|
||||
is the **blackbox exporter** in probe mode (which also gives guarded vhosts
|
||||
a real uptime signal, since Prometheus cannot solve an Anubis challenge).
|
||||
- **Fix in the direction of honesty, never of silence:** unreachable target →
|
||||
mark the homepage card `wip: true` with the reason in `description` (the
|
||||
checker skips `wip`; `wip` is the existing convention for "real service,
|
||||
not probeable"), or repoint the job at something reachable. NEVER open a
|
||||
database port on the LAN to make a badge green, and never delete a scrape
|
||||
job — an absent scrape and a healthy one are indistinguishable.
|
||||
- Alert rules must not depend on always-red signals — scope them to services
|
||||
that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
|
||||
exactly that reason).
|
||||
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
|
||||
`connection refused` to `unsupported Content-Type "text/html"` means the
|
||||
service came up and only the scrape is wrong. Read the error, not just
|
||||
up/down.
|
||||
|
||||
## 2. Vacuous assertions (highest severity — the gate is already dead)
|
||||
|
||||
A seam that greps a file passes *by absence* once the file is retired:
|
||||
`grep` exits 2 on a missing path and `!` inverts it.
|
||||
|
||||
```bash
|
||||
grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
|
||||
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
|
||||
```
|
||||
|
||||
Fix shape — one place asserts the invariant, each local site keeps a live
|
||||
check:
|
||||
|
||||
```bats
|
||||
@test "no <host> compose file competes with the quadlet units" {
|
||||
local cf="$REPO_ROOT/<host>/compose.yaml"
|
||||
[ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
|
||||
}
|
||||
# at the per-service site: still fails if the entry returns
|
||||
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
|
||||
```
|
||||
|
||||
**Prove teeth:** recreate the offending file
|
||||
(`printf 'services:\n <svc>:\n image: x\n' > <path>`), run the seams,
|
||||
confirm RED, then move the file to trash. A re-armed check you have never
|
||||
seen fail is still unverified.
|
||||
|
||||
## 3. Gates that never run
|
||||
|
||||
Playwright specs in `tests/e2e/` are matched by **number prefix** in
|
||||
`playwright.config.ts`; an unregistered spec silently never runs and the
|
||||
suite count stays the same. Count tests before/after;
|
||||
`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
|
||||
|
||||
## 4. Validate the checker before acting on it
|
||||
|
||||
A checker's output is a hypothesis about another tool's semantics. Modelling
|
||||
Obsidian with naive text matching produced 45 "broken links" of which 3 were
|
||||
real — the rest were case-sensitivity (a `qBittorrent` link to
|
||||
`qbittorrent.md`), path-qualified resolution (a path-qualified
|
||||
`folder/Note` link resolves by path SUFFIX, not
|
||||
stem), and `%20` URL-encoding in markdown links. Fix the checker first,
|
||||
re-derive the report, only then edit documents. The result lives in
|
||||
`scripts/check-vault-links.py` with those three rules in its docstring,
|
||||
gated by the `no unresolved links in the docs vault` seam. Same class: a
|
||||
`find … | head -1` selector that reads the wrong file; a truncated listing
|
||||
that makes you "fix" a file that exists.
|
||||
|
||||
## 5. Prose invalidated by your own change
|
||||
|
||||
After retiring a file or adding a feature, grep for EVERY reference, not just
|
||||
code:
|
||||
|
||||
```bash
|
||||
grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
|
||||
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
|
||||
```
|
||||
|
||||
This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
|
||||
after it was restored with GPU access; a unit said "there are no alerting
|
||||
rules" hours after five were added; a docs page documented three scrape jobs
|
||||
as working when they could never report up. **Clean cutover includes prose** —
|
||||
docs, unit docstrings, and the repo's rule file (the most expensive place to
|
||||
be stale).
|
||||
|
||||
## Order and verification
|
||||
|
||||
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
|
||||
gates, docs/rule files, dashboard panels).
|
||||
2. For each, ask what the signal CAN express, not what it says.
|
||||
3. Validate any checker before editing the things it flags.
|
||||
4. Fix at the root; never weaken an assertion to make it pass.
|
||||
5. Gate the class so it cannot return (a seam), then prove the seam fails on
|
||||
the defect.
|
||||
|
||||
Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
|
||||
Prometheus panel expression against the API (0 series on a panel that should
|
||||
show data is a defect; 0 `ALERTS` when nothing fires is not); read the
|
||||
deployed `/srv/homepage/status.json` and triage every `state != up` entry
|
||||
against reality; query `/api/v1/targets?state=active` and print
|
||||
`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
|
||||
Report per finding: the surface, why the signal could not have been correct,
|
||||
the fix, and the red/green evidence — flag any regression you introduced
|
||||
yourself in the same breath (one was found this way: a tightened bind killed
|
||||
a reverse-proxied vhost while every gate stayed green, because the spec
|
||||
covering that hostname did not exist).
|
||||
|
||||
## Related
|
||||
|
||||
- [[Testing Policy]] — the seam inventory this audit walks
|
||||
- [[Observability]] — the scrape/probe surfaces it audits
|
||||
- [[Quadlet Lifecycle]] — the ownership labels the probes read
|
||||
@@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto
|
||||
The npx package cache lives in the container layer, so a recreation (not just a
|
||||
restart) pays one first-call npm download; subsequent calls are cached.
|
||||
|
||||
## Config-home drift ("cannot connect" to models that answer curl)
|
||||
|
||||
`podman exec <container> printenv HERMES_HOME` for each hermes container. If
|
||||
the webui reads the image default (`/opt/data`) while the gateway reads
|
||||
`/data`, the webui runs the image-default template config (`provider: auto` →
|
||||
openrouter.ai with no key = connect failure).
|
||||
|
||||
The recreation trap is the same one the volumes table shows: image `VOLUME`
|
||||
points get a fresh anonymous volume on every container recreation
|
||||
(`podman run --replace --rm`) — edits inside such a mount are lost on
|
||||
restart/redeploy. Compare volume ids before/after a restart to confirm.
|
||||
|
||||
Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
|
||||
root), deploy service-scoped. Verify with the app's own code, not guesses:
|
||||
|
||||
```python
|
||||
from api import config as C; C.resolve_model_provider("<model-id>")
|
||||
# -> (model_id, provider, base_url); bare ids map to the owning provider by
|
||||
# scanning the providers: dict models allow-list; @provider:model forces one.
|
||||
hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
|
||||
# -> ProviderDef with real base_url; accepts api/url/base_url keys.
|
||||
```
|
||||
|
||||
Named providers (`providers:` dict) are a catalog + connection source;
|
||||
`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
|
||||
neither holds the provider name, the request has no endpoint and fails to
|
||||
connect. Probe the endpoint with the **exact served model id** — the retired
|
||||
FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
|
||||
with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
|
||||
checks miss this class entirely.
|
||||
|
||||
## Live config edits
|
||||
|
||||
Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
|
||||
uid 109999; all reads/edits need `sudo`). The repo file
|
||||
`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
|
||||
live — always diff the live file first; never treat the example as truth**.
|
||||
|
||||
1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
|
||||
(in-dir `.AGENTTRASH/` also exists for removals).
|
||||
2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
|
||||
the live file are tool-generated — preserve them.
|
||||
3. **Sweep stale references** across the whole data dir:
|
||||
`sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
|
||||
--include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
|
||||
historical — leave them.
|
||||
4. **Derived caches:** `webui/models_cache.json` is keyed by a config
|
||||
fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
|
||||
is stale. Let it regenerate lazily or trash it — never hand-edit.
|
||||
5. **Restart the model-consuming units** (they parse config at startup):
|
||||
`systemctl --user restart hermes-webui.service hermes.service`.
|
||||
`hermes-dashboard.service` is unaffected by routing changes.
|
||||
6. **Verify behaviorally**: one tiny completion per endpoint with the exact
|
||||
served id — POST `/v1/chat/completions` and read the body, never just the
|
||||
status code (endpoint names rotate; see the LLM knowledge base).
|
||||
7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
|
||||
live shape, redacting secrets (`password_hash`, phone account stay
|
||||
live-only), then `./tests/run.sh` before commit.
|
||||
|
||||
`provider: custom` + `base_url` entries route by base_url and send the model
|
||||
id verbatim; named providers resolve via their `api:` URL. **Renaming a
|
||||
served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
|
||||
config in the same change** or Hermes requests 404 on nonexistent models —
|
||||
this exact gap caused the 2026-09-05 outage.
|
||||
|
||||
## Auxiliary routing against local backends
|
||||
|
||||
When a Hermes feature that calls an auxiliary task (e.g. title generation)
|
||||
fails against a local endpoint, the usual cause is a **request shape the
|
||||
local runtime does not implement**, not a routing mistake.
|
||||
|
||||
Diagnose with the real call path — run the actual function inside the
|
||||
container with the app's own constants (the probe cannot drift from
|
||||
production):
|
||||
|
||||
```bash
|
||||
base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
|
||||
'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
|
||||
podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
|
||||
podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
|
||||
```
|
||||
|
||||
Copy the file AFTER any restart (`--replace --rm` discards the container
|
||||
layer). The CLI at `/opt/hermes/hermes` uses system python and dies
|
||||
`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
|
||||
|
||||
```python
|
||||
import sys; sys.path.insert(0, "/opt/hermes")
|
||||
from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
|
||||
from agent.auxiliary_client import call_llm
|
||||
resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
|
||||
temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
|
||||
```
|
||||
|
||||
What actually breaks, and the fix:
|
||||
|
||||
- **`response_format` (json_object / json_schema) is a hard 400 on every
|
||||
local backend here** — neither FreeToken nor halogen implements constrained
|
||||
decoding. Config cannot override it: `call_llm` merges the task's
|
||||
`extra_body` first and then `update()`s the caller's, so the hardcoded one
|
||||
wins. Only temperature and max_tokens have reactive retry paths.
|
||||
- **Fix: move that one task to the Responses transport**, which never
|
||||
forwards `response_format`:
|
||||
|
||||
```yaml
|
||||
auxiliary:
|
||||
title_generation:
|
||||
provider: dankserver-small # names track the live config; diff first
|
||||
model: frank-small
|
||||
api_mode: codex_responses # real per-task key
|
||||
extra_body:
|
||||
reasoning:
|
||||
effort: none # mandatory, see below
|
||||
```
|
||||
|
||||
- **Disable thinking for any cap-limited task.** The adapter omits reasoning
|
||||
only when config says `reasoning.enabled: false`. The
|
||||
`reasoning_effort: none` *shorthand* produces exactly that and leaves the
|
||||
server default (thinking ON) — the 64-token title budget gets spent on the
|
||||
think: empty content, `finish_reason: length`. The explicit
|
||||
`extra_body.reasoning.effort: none` instead sends
|
||||
`{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
|
||||
honors.
|
||||
- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
|
||||
proxy needed for the alternate transport.
|
||||
|
||||
Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
|
||||
it must pass from config alone (that is the production path); keep the live
|
||||
file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
|
||||
config change is docs+example only, no service deploy needed.
|
||||
|
||||
Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
|
||||
`apply_instant_title`, so disabling a broken title task also removes the
|
||||
derived title — it does not degrade gracefully the way a failure does. A
|
||||
local call can also fail at the model level: a 200 with an error-text body,
|
||||
or a chatty non-title answer. Always read the raw content, not the status.
|
||||
|
||||
## Webui password rotation ("Hermex")
|
||||
|
||||
Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
|
||||
`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
|
||||
on match, 401 on miss. No username field. `/health` answers 200
|
||||
unauthenticated and `/` answers 302 — **never verify auth via those**; probe
|
||||
login with a deliberately wrong password (expect 401) and the real one
|
||||
(expect 200). Details on the webui surface: [[Hermes-WebUI]].
|
||||
|
||||
1. Run the whole probe as a heredoc'd bash script
|
||||
(`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
|
||||
inline `${…}` even inside single-quoted ssh args; never interpolate curl
|
||||
`-w %{http_code}` inline.
|
||||
2. Update BOTH env files with
|
||||
`sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
|
||||
- `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
|
||||
- `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
|
||||
quadlet's `[Container] EnvironmentFile=` is what the container reads at
|
||||
create; rotating only the root file leaves the running container serving
|
||||
the old secret.
|
||||
3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
|
||||
env; no deploy needed if the unit file itself didn't change).
|
||||
4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
|
||||
client at `http://192.168.0.143:8787` (webui publishes on the LAN
|
||||
interface since commit `4443897`; off-network ingress is SSH tunnel only —
|
||||
user `tunnel`, port **22**, key-only auth, forward remote end
|
||||
localhost:8787).
|
||||
|
||||
Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
|
||||
see rootless containers; run `podman exec hermes-webui printenv
|
||||
HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
|
||||
600). If rotating because a secret leaked into transcripts/logs, rotate in
|
||||
place via a script that fetches the live value from the running container
|
||||
(`printenv`) instead of transcribing it.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes-Dashboard]] — the LAN browser management UI
|
||||
|
||||
@@ -64,6 +64,49 @@ The Ollama connection is disabled (`ollama.enable=false`).
|
||||
> repository. Model swaps, context sizing, and spec tuning happen there. See
|
||||
> the [LLM knowledge base](../LLM/index.md).
|
||||
|
||||
## Container introspection (DB and backend python)
|
||||
|
||||
- Working dir for imports: `/app/backend`; package name `open_webui`
|
||||
(underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
|
||||
BEFORE importing any `open_webui` module (import-time hard requirement);
|
||||
the secret lives at `/app/backend/.webui_secret_key`.
|
||||
- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
|
||||
module (`podman exec open-webui python3 -c "import sqlite3;
|
||||
conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config
|
||||
keys use parameterized LIKE queries (avoids shell quoting).
|
||||
- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` —
|
||||
`Models.get_all_models()` is a coroutine (await it); model objects are
|
||||
pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
|
||||
`status`/`enabled` attribute — use `is_active` from the DB.
|
||||
- **API auth for verification:** `create_token` lives in
|
||||
`open_webui.utils.auth` (NOT `open_webui.utils.utils`);
|
||||
`get_verified_user` requires an `id` claim (not just `sub`);
|
||||
`SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
|
||||
container:
|
||||
|
||||
```
|
||||
podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \
|
||||
python -c "from open_webui.utils.auth import create_token; \
|
||||
print(create_token({\"id\":\"<UID>\",\"sub\":\"<UID>\",\"email\":\"…\",\"role\":\"admin\"}))"'
|
||||
```
|
||||
|
||||
(admin UID from the `user` table).
|
||||
|
||||
## Verification battery
|
||||
|
||||
1. `GET /api/v1/models/list?limit=100` → returns EXACTLY the intended
|
||||
models. Response key is `items`, not `data`.
|
||||
2. `POST /openai/chat/completions` per model with a trivial prompt ("Reply
|
||||
with exactly one word: pong") → each returns content `pong`. Reasoning
|
||||
models need `max_tokens>=100` — the thinking block eats the budget.
|
||||
3. A 500 "Server Connection Error" → `podman logs open-webui` for
|
||||
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
|
||||
`host.containers.internal`), not a model problem.
|
||||
|
||||
**Durability:** the model table + connections are DB-only (not git-managed);
|
||||
a fresh data volume loses them. For reproducibility, add an idempotent
|
||||
first-start seed script or document the setup in the service doc.
|
||||
|
||||
## Web search
|
||||
|
||||
Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
|
||||
@@ -147,6 +190,23 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
|
||||
|
||||
`podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
|
||||
|
||||
## Image generation (ComfyUI)
|
||||
|
||||
Image generation rides the local [[ComfyUI]] integration, not the chat
|
||||
picker — ComfyUI's models never appear in the chat model picker (that is
|
||||
chat LLMs only), so ComfyUI needs its own image-generation connection
|
||||
configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
|
||||
serves `:8188`.
|
||||
|
||||
Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
|
||||
route to `POST /api/v1/images/generations`; the `comfyui` engine is still
|
||||
upstream) with `Authorization: Bearer <token>` and body
|
||||
`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
|
||||
carries base64 image data or a URL.
|
||||
|
||||
Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
|
||||
on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
|
||||
|
||||
## Retired instances
|
||||
|
||||
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
|
||||
|
||||
@@ -117,6 +117,18 @@ frank-large `health=200` throughout.
|
||||
the model is loaded. Trust the engine's `host memory left for everything
|
||||
else` line, not `free`.
|
||||
|
||||
- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
|
||||
directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
|
||||
snapshots in the swapfile path — check `df -T` before planning, or
|
||||
`mkswap` will reject the file.
|
||||
- **Renaming an *active* swapfile works (inode-based) but leaves a stale
|
||||
path in `/proc/swaps`** — swap it off first, then rename, then `swapon`
|
||||
(which is exactly what the step-3 consolidation does). Migration is
|
||||
observable while it runs: in the 2026-09-12 run `swapoff` of 4 GiB took
|
||||
65 s and used-RAM rose only ~3 GiB (pages touched during migration), the
|
||||
rest relocating on disk. If used-RAM climbs toward the whole swapped
|
||||
amount, let it finish rather than aborting.
|
||||
|
||||
## Trash
|
||||
|
||||
The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
---
|
||||
type: reference
|
||||
title: Noctalia and Niri
|
||||
description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
|
||||
tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Noctalia and Niri
|
||||
|
||||
Noctalia (`/usr/bin/noctalia`) is the desktop shell on niri, running on both
|
||||
the workstation and blankpad. Since the 2026-09-10 unification both run the
|
||||
**bare `noctalia` v5 binary** (niri `spawn-at-startup "noctalia"`); the old
|
||||
Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is
|
||||
retired on both hosts.
|
||||
|
||||
## Config layers
|
||||
|
||||
- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g.
|
||||
`config.toml`). Main config: `~/.config/noctalia/config.toml`; the niri
|
||||
config includes `noctalia.kdl` for display settings.
|
||||
- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and
|
||||
**overrides declarative values for the same keys — never hand-edit it**;
|
||||
UI/IPC writes land there.
|
||||
- Main reference: `noctalia-reference` facts — IPC `noctalia msg <command>`
|
||||
replaced the `qs -c noctalia-shell ipc call` wrapper; app templates live at
|
||||
`/usr/share/noctalia/assets/templates/` (gtk, ghostty, alacritty, hyprland,
|
||||
sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango,
|
||||
umbriel, foot, emacs, kitty); palette generation `noctalia theme [image]`.
|
||||
|
||||
## Remote IPC over ssh (the D-Bus trap)
|
||||
|
||||
`noctalia msg <command>` resolves the running instance via the D-Bus session
|
||||
bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain
|
||||
ssh exec prints `error: noctalia is not running` even when the shell IS
|
||||
running — do not conclude "shell dead" from this error. Working remote
|
||||
procedure (runs in the systemd user-manager context where the bus address
|
||||
exists):
|
||||
|
||||
```
|
||||
systemd-run --user --pipe --wait noctalia msg <command> [args]
|
||||
```
|
||||
|
||||
Verified: `msg status` returns live JSON state; unknown panel ids return an
|
||||
error listing all valid panel ids (clipboard, control-center, launcher,
|
||||
polkit, session, setup-wizard, test, tray-drawer, wallpaper).
|
||||
|
||||
Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
|
||||
`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
|
||||
`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
|
||||
`media next|previous|playPause` (graceful no-op, rc 0, with no active
|
||||
player), `brightness-up [target] [step]`, `brightness-down`,
|
||||
`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
|
||||
`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
|
||||
`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
|
||||
`screenshot-fullscreen`.
|
||||
|
||||
**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
|
||||
resolves to XDG Pictures via user-dirs.dirs; an explicit directory key
|
||||
overrides. Fleet default pattern: point `directory` at the repo-tracked
|
||||
wallpapers dir so git-tracked wallpapers serve the shell without copying
|
||||
files. After writing config.toml over ssh the file watcher hot-reloads
|
||||
automatically; `config-reload` via systemd-run forces it immediately. Note:
|
||||
the *current* wallpaper stays the built-in default until the next rotation
|
||||
or an explicit `wallpaper-next`; `wallpaper-get` then reporting a path
|
||||
inside the configured directory is the confirmation the config took effect.
|
||||
|
||||
**Lock chain (unified):** swayidle-lock.service ExecStart on both hosts:
|
||||
`/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock'
|
||||
before-sleep '…' after-resume '…'`. `msg` resolves via the session bus, so
|
||||
no WAYLAND_DISPLAY pinning is needed in unit context.
|
||||
|
||||
## Custom-palette recolor (without restarting the session)
|
||||
|
||||
Why custom, never the community cache:
|
||||
`~/.local/state/noctalia/community-palettes/*.json` is a **sync cache** —
|
||||
`.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches
|
||||
from `https://api.noctalia.dev/palettes`, so edits there get reverted.
|
||||
Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
|
||||
(`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never
|
||||
synced. Missing/invalid file → silent fallback to the builtin palette.
|
||||
|
||||
Palette JSON shape = the community file's shape: `dark`/`light` maps of
|
||||
`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
|
||||
`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
|
||||
(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
|
||||
`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
|
||||
-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
|
||||
*generator* format, not this one. Copy an existing community JSON and mutate
|
||||
with `jq`.
|
||||
|
||||
Procedure:
|
||||
|
||||
1. Read the live scheme first — never assume: `noctalia msg color-scheme-get`
|
||||
(e.g. `community Gruber Darker`); `noctalia msg theme-mode-get`.
|
||||
2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml
|
||||
/tmp/noctalia-settings-backup-$(date +%s).toml`.
|
||||
3. Build the palette from the current one (jq keeps structure):
|
||||
|
||||
```sh
|
||||
mkdir -p ~/.config/noctalia/palettes
|
||||
jq '.dark.mPrimary = "#f43841"
|
||||
| .dark.terminal.normal.yellow = "#f43841"
|
||||
| .dark.terminal.bright.yellow = "#ff4f58"
|
||||
| .dark.terminal.foreground = "#f43841"' \
|
||||
"$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \
|
||||
> ~/.config/noctalia/palettes/"Gruber Red.json"
|
||||
```
|
||||
|
||||
Prefer a hex already present in the palette (its own `mError` red) so the
|
||||
theme stays internally consistent. Set BOTH `normal` and `bright`
|
||||
variants, and both `dark` and `light` if the user toggles modes.
|
||||
4. Switch (persists `theme.source = custom` + `theme.custom_palette`):
|
||||
`noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
|
||||
`color-scheme-get`.
|
||||
5. Re-render every configured template: `noctalia msg templates-apply`.
|
||||
Regenerates `~/.config/ghostty/themes/noctalia`,
|
||||
`~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
|
||||
starship, etc. Symlinked-into-dotfiles paths show up as modified files
|
||||
worth committing.
|
||||
6. Reload surfaces: compositor — `niri msg action load-config-file`
|
||||
(`reload-config` is NOT a valid niri subcommand); ghostty — see the trap
|
||||
below.
|
||||
|
||||
Verification: `ghostty +show-config | grep -E '^(foreground|palette)'`
|
||||
(proves the theme file resolves — note it prints only the delta from
|
||||
defaults; cross-check with `ghostty +show-config --default` before
|
||||
concluding a key is unset); `noctalia config validate` → `✓ Config is
|
||||
valid`; visual via `noctalia msg screenshot-fullscreen` (saves to
|
||||
`~/Pictures/screenshot_<ts>.png` and ALSO overwrites the clipboard with the
|
||||
PNG — warn the user); `grep -c <old-hex> ~/.config/noctalia/palettes/*.json`
|
||||
→ 0 for no leftovers.
|
||||
|
||||
Track it in the dotfiles repo (`~/.omp`, the command-center —
|
||||
`scripts/deploy.sh` is a stow-equivalent over `dotfiles/<pkg>/` trees):
|
||||
mirror the shape for the new palette so `git pull` + `deploy.sh` is the
|
||||
whole propagation story — move the JSON into
|
||||
`~/.omp/dotfiles/noctalia/.config/noctalia/palettes/` and symlink it back
|
||||
into `~/.config/noctalia/palettes/`; re-set the custom palette (re-reads
|
||||
through the symlink). Do the link by hand for one file instead of running
|
||||
`deploy.sh` — sync mode quarantines and replaces every divergent live file.
|
||||
|
||||
Traps:
|
||||
|
||||
- **NEVER `kill -USR1` ghostty to reload.** It does not reload config on
|
||||
this fleet — it kills/respawns the terminal (took out Frank's window
|
||||
2026-09-13). Correct: the default keybind `ctrl+shift+,` = `reload_config`
|
||||
(per `ghostty +show-config --default`). Existing windows keep old colors
|
||||
until reloaded; new windows inherit immediately.
|
||||
- Palette hex values are NOT a proxy for what the user sees: in
|
||||
`Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change
|
||||
the yellow" means the accent too, not just terminal index 3.
|
||||
- Saturating terminal text makes ANSI 1/9 indistinguishable from default
|
||||
text (grep highlights flatten) — flag it and offer a lighter red.
|
||||
- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats
|
||||
white-on-color here, ~3.9:1 — keep `mOnPrimary` dark).
|
||||
- Revert = `noctalia msg color-scheme-set community "<original name>"`; the
|
||||
community cache is untouched by the whole flow.
|
||||
- Generated theme files symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0`
|
||||
`noctalia.css`, ghostty theme) show up as tracked modifications after
|
||||
`templates-apply`; commit them together with the new palette JSON so every
|
||||
machine gets the same recolor.
|
||||
|
||||
## Offline template render (when templates-apply never ran)
|
||||
|
||||
Failure signature: niri `failed to read included config from
|
||||
~/.config/niri/noctalia.kdl: No such file or directory` — the desktop runs
|
||||
but with DEFAULT config (niri 26.04 falls back, does not exit): no custom
|
||||
keybinds/rules, no noctalia autostart, default output placement. ghostty:
|
||||
`theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`.
|
||||
alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing.
|
||||
|
||||
Root cause pattern: upstream dotfiles commits theme switches (niri include,
|
||||
ghostty `theme = noctalia`, alacritty import) that depend on noctalia
|
||||
TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from
|
||||
the local palette) and are intentionally NOT in the repo; on any host where
|
||||
`templates-apply` never ran, they don't exist.
|
||||
|
||||
Fix (offline, works while a session runs) — built-in templates live at
|
||||
`/usr/share/noctalia/assets/templates/` (declared in `builtin.toml`); render
|
||||
each needed one with the live palette:
|
||||
|
||||
```bash
|
||||
# niri include (plain host-local file; do NOT symlink into repo — regenerated per host)
|
||||
noctalia theme --theme-json ~/.config/noctalia/colors.json \
|
||||
--render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl"
|
||||
mkdir -p ~/.config/ghostty/themes && \
|
||||
noctalia theme --theme-json ~/.config/noctalia/colors.json \
|
||||
--render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia"
|
||||
mkdir -p ~/.config/alacritty/themes && \
|
||||
noctalia theme --theme-json ~/.config/noctalia/colors.json \
|
||||
--render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml"
|
||||
```
|
||||
|
||||
All template tokens (primary, surface, error, primary_container, …) resolve
|
||||
from the flat 16-key colors.json; container tokens are derived, never blank.
|
||||
|
||||
Verify: `niri validate` → rc 0; `ghostty +validate-config` → rc 0; next
|
||||
login journal shows `loaded config from config.kdl` with NO parse WARN,
|
||||
custom output placement (`niri msg outputs` vs display.kdl), noctalia
|
||||
autostarted (`pgrep noctalia`).
|
||||
|
||||
Traps: **`noctalia msg templates-apply` on 5.0.1 returns "ok" but applies
|
||||
NOTHING** (this build's opt-in config schema does not act on it) — do not
|
||||
trust its exit code, use the explicit `--render` calls above. Niri config
|
||||
errors at boot are NOT fatal — the compositor keeps running on defaults,
|
||||
which reads as "some settings silently stopped working", not a crash.
|
||||
Regenerate after any palette change; the rendered files are static
|
||||
snapshots.
|
||||
|
||||
## Niri output rotation and pinning
|
||||
|
||||
Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
|
||||
|
||||
**Ground truth first:** `niri msg outputs` (connector names, current mode,
|
||||
scale, transform, logical pos/size) and `niri --version`. Output **names are
|
||||
connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
|
||||
the EDID strings printed before them are decoration. On blankpad the live
|
||||
config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
|
||||
and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
|
||||
repo path (that is what gets committed); check with `ls -ld`.
|
||||
|
||||
**Transform direction is counter-clockwise** (`niri msg output <NAME>
|
||||
transform --help`): `90` = content rotated 90° CCW ⇒ the panel's **top
|
||||
bezel sits on the LEFT**; `270` = top bezel RIGHT; `180` = upside down;
|
||||
`flipped-90`/`flipped-270` = mirrored variants. `90` vs `270` is a
|
||||
physical-stand fact no tool can read (portable monitors expose no
|
||||
orientation in EDID/MCCS) — pick one, apply, and hand the user the one-liner
|
||||
to flip; don't stall on the question.
|
||||
|
||||
Two traps when editing the config:
|
||||
|
||||
1. **An `output "X" { … }` block resets every property you don't restate to
|
||||
its default.** Pinning position on `eDP-1` without repeating `scale 1.25`
|
||||
silently drops the laptop panel to scale 1.0. Restate `scale` (and
|
||||
`mode`) in every block you touch.
|
||||
2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning
|
||||
only the externals shoved `eDP-1` from `0,0` to `3936,0`. Pin every
|
||||
output's position to keep the left-to-right order. Positions are logical
|
||||
pixels at each output's own scale; x-offsets chain (eDP-1 logical width
|
||||
1536 → DP-2 at 1536 → +1200 portrait width → HDMI-A-1 at 2736):
|
||||
|
||||
```kdl
|
||||
output "DP-2" { scale 1 transform "90" position x=1536 y=0 }
|
||||
output "HDMI-A-1" { scale 1 transform "90" position x=2736 y=0 }
|
||||
output "eDP-1" { scale 1.25 position x=0 y=0 } // MUST restate scale
|
||||
```
|
||||
|
||||
Apply and verify:
|
||||
|
||||
```bash
|
||||
niri validate -c ~/.config/niri/config.kdl # exit 0
|
||||
niri msg action load-config-file # live reload; NOT reload-config
|
||||
sleep 1; niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
|
||||
```
|
||||
|
||||
Success looks like `Logical size: 1200x1920` (swapped) plus
|
||||
`Transform: 90° counter-clockwise` on each rotated panel, positions matching
|
||||
the intended chain. One-off rotation without touching config (temporary):
|
||||
`niri msg output DP-2 transform 90`. Commit via the dotfiles repo
|
||||
(`~/.omp`, `main`; push to the local Gitea permitted): `git pull --ff-only`
|
||||
→ add the display.kdl → commit → push. Pitfalls: `reload-config` is not a
|
||||
niri subcommand; never SIGUSR1 a terminal to "reload" (niri config reloads
|
||||
do not touch ghostty themes); no man page for niri on this host — `niri
|
||||
<sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
|
||||
|
||||
## GPU RGB over I2C (EVGA iCX3)
|
||||
|
||||
Turn RGB on/off (or read it back) on an EVGA FTW3 / iCX3-controller GPU
|
||||
(the 3090 on dankclaw) using the GPU's own nvidia i2c bus. No display
|
||||
engine, no X, no daemon — the mode is EEPROM-saved and survives reboot.
|
||||
|
||||
- **Find the controller:** the nvidia i2c driver exposes the card's lighting
|
||||
ports as `/dev/i2c-N`; on the 3090 FTW3 that is ports 1,5,6,7,8 →
|
||||
i2c-1..i2c-5, and only i2c-1 carries a live slave (NVIDIA port 1). Probe
|
||||
each bus. The iCX3 MCU answers at address **0x2D**. `/dev/i2c-*` are
|
||||
`root:i2c 660` — run with `sudo` (dank has passwordless sudo).
|
||||
- **Protocol (raw I2C_RDWR, NOT the SMBus ioctls)** — write the register
|
||||
byte first, then the payload:
|
||||
- enable `0xB2` write `04 C6 EB EA 15` (required once before mode writes)
|
||||
- mode `0xC0` write `09 m0 m1 m2 m3 .. .. .. .. SYNC`; m0..m3 are the four
|
||||
zone modes, `MODE_OFF = 0x00`; OFF = `09 00 00 00 00 FF FF FF FF 00`
|
||||
- save `0x90` write `04 9E EB 00 90` (commit to EEPROM — the
|
||||
reboot-persistence step)
|
||||
- fw `0xB1` block read (firmware version)
|
||||
- Read the mode back: write reg `0xC0`, then read 16 bytes; bytes 1..4 are
|
||||
the zone modes (0x00 = OFF), bytes 6..9 the per-zone LED counts.
|
||||
- **Verify persistence:** after a reboot, read the mode — all four zone
|
||||
modes `0x00` means the EEPROM save held (confirmed 2026-09-19 post-reboot).
|
||||
- **Tool:** a ~90-line I2C_RDWR reader/writer lived at `/tmp/evga_raw.c`
|
||||
(`evga_raw <bus> mode|off|save|enable|fw`) — `/tmp` is wiped by reboot, so
|
||||
recreate it if needed; OpenRGB's `EVGAGPUv3Controller.cpp` carries the
|
||||
same protocol plus the full colour API.
|
||||
- Gotchas: raw I2C_RDWR only (the iCX3 answers the raw form, not the SMBus
|
||||
ioctls); the `enable` write must precede the first mode write in a
|
||||
session; without the `save` write the mode is RAM-only and resets on power
|
||||
cycle.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Command Center Dotfiles]] — the stow-equivalent deploy this page links into
|
||||
- [[Hermes]] — the agent that drove the CDP daemon alongside this desktop
|
||||
- [[GGUF Compatibility]] — unrelated but shares the "read the file, not the name" lesson
|
||||
@@ -5,3 +5,115 @@ description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Reada
|
||||
tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
|
||||
timestamp: 2026-09-04T00:00:00Z
|
||||
---
|
||||
|
||||
# *arr Stack
|
||||
|
||||
The *arr automation stack runs on **FrankBook-Pro** (the Mac mini) as quadlets
|
||||
inside its podman machine; all egress rides the gluetun WireGuard netns. The
|
||||
per-service pages ([[Prowlarr]], [[Sonarr]], [[Radarr]], [[Lidarr]],
|
||||
[[Readarr]], [[qBittorrent]]) carry deployment facts; this page holds the
|
||||
cross-cutting API and auth-posture procedures.
|
||||
|
||||
## API version split (memorize before any *arr HTTP call)
|
||||
|
||||
- Sonarr/Radarr: `/api/v3/…`; Lidarr/Readarr/Prowlarr: `/api/v1/…`.
|
||||
- Prowlarr is port **9696** — 8090 is Jackett.
|
||||
- **Empty body = endpoint absent**, not a failure: probe endpoints by status
|
||||
code first; v1 has no `/api/v1/profile` endpoint at all.
|
||||
- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object
|
||||
(GET it, patch `settings`, PUT the whole object back). An array body fails
|
||||
with `Path: $ | BytePositionInLine: 1`.
|
||||
- API keys live in each container's config:
|
||||
`podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
|
||||
They are local-only secrets — they stop being secret the moment a vhost
|
||||
serves the API unauthenticated.
|
||||
|
||||
## Schema-first probing (validators are free documentation)
|
||||
|
||||
Send the FULL schema template object with fields filled; v1 validators reject
|
||||
partials, and each error names the next required field — peel validators one
|
||||
live call at a time instead of guessing payload shapes:
|
||||
|
||||
- `'Fields' must not be empty` → send a full schema copy, not `fields:[]`.
|
||||
- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
|
||||
required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
|
||||
- `DefaultMetadataProfileId must be > 0` → fetch real ids from
|
||||
`GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
|
||||
`GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
|
||||
profiles.
|
||||
- `targetType null` → configContract not registered in that build (this
|
||||
Prowlarr build has no Prowlarr-plugin indexer; integration is
|
||||
Torznab-family only).
|
||||
- `testingPackageId` in a PUT body downgrades validation failures to
|
||||
warnings.
|
||||
- `syncLevel` is enum `ApplicationSyncLevel` — omit it or use the int; a
|
||||
string is rejected.
|
||||
|
||||
**Verification rules:** `POST /downloadclient` returns the COLLECTION, not
|
||||
the created object — verify via GET. Test endpoints return verdicts in the
|
||||
BODY at HTTP 200 (`{}` = pass, a list = fail) — judge from bodies, not
|
||||
status codes. Command POSTs (`MoviesSearch`, `ApplicationSync`) run
|
||||
server-side and respond only when the work finishes: curl `-m` timeouts
|
||||
(HTTP 000) during a search are NOT failures — poll queue/history afterward
|
||||
instead of re-firing; warm small commands (`RefreshMonitoredDownloads`)
|
||||
answer in ~20 ms, so a hang means live work in progress, not a dead API.
|
||||
|
||||
**DB as arbiter:** when endpoint shapes are opaque, stop the container and
|
||||
edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly,
|
||||
then start. The indexers table columns are
|
||||
Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the
|
||||
enable flag column is `Enable` (not `EnableRss`). Prowlarr's real application
|
||||
settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
|
||||
`/api/v1/applications` GET responses carry `fields: []`).
|
||||
|
||||
## Auth posture probe (ground truth is behavior, not config flags)
|
||||
|
||||
`AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in
|
||||
`config.xml` is ambiguous on paper; on the wire it means a login screen with
|
||||
**zero users** — first-account-wins.
|
||||
|
||||
1. **Confirm which host runs the service before probing** (`podman ps` on
|
||||
the target, not the box you ssh'd into first — the arrs live on
|
||||
FrankBook-Pro).
|
||||
2. Probe an API endpoint that bypasses the SPA shell, locally and through
|
||||
any vhost: `curl -s -o /dev/null -w "%{http_code}"
|
||||
http://<host>:<port>/api/v3/system/status`. **SPA roots (`/`) always
|
||||
serve a 200 shell — never conclude "open" or "gated" from `/`.**
|
||||
401 = gated; 200 + JSON = API fully open (anyone on the net can drive it
|
||||
with curl).
|
||||
3. For config truth, dump the whole file
|
||||
(`podman exec <svc> cat /config/config.xml`), not a piped
|
||||
`grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the
|
||||
pipeline hides a miss and you conclude "empty field" when the grep itself
|
||||
failed.
|
||||
4. Decision framing: when posture = gated-with-zero-users, present it as a
|
||||
decision menu (None behind the front-door wall vs External/authentikarr),
|
||||
not a silent change — first-account-wins means whoever logs in first owns
|
||||
the instance.
|
||||
|
||||
## The `None` + `Enabled` auth-config trap (v1 .NET apps)
|
||||
|
||||
`<AuthenticationMethod>None</AuthenticationMethod>` +
|
||||
`<AuthenticationRequired>Enabled</AuthenticationRequired>` is
|
||||
self-contradictory: the UI Save is permanently disabled, and any save throws
|
||||
`DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every
|
||||
page after restart. The valid no-auth representation is **absence of the
|
||||
`<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix =
|
||||
delete the key from `/config/config.xml`
|
||||
(`sed -i "/<AuthenticationRequired>/d"`), restart the container. Tell-tale:
|
||||
one v1 app boots clean while siblings 500-sprawl → diff their config.xml
|
||||
auth blocks; the healthy one shows the correct shape. (Prowlarr runs
|
||||
`Forms` + `Enabled` healthily — the crash combo is `None`+`Enabled` only.)
|
||||
|
||||
General rules that saved the session these came from: container
|
||||
`State=running` ≠ app serving — read `podman logs --tail 12 <c>` before
|
||||
theorizing; when a UI gate blocks saving, go under it (these apps keep
|
||||
plaintext XML config on disk — read/edit via `podman exec`).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Prowlarr]] — indexer feeds and the v2 sync model
|
||||
- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] — the managers
|
||||
- [[qBittorrent]] — download client, 4.6.7 pin, E2E verification
|
||||
- [[FrankBook-Pro]] — the host and its podman machine
|
||||
- [[Homepage Status Probes]] — how the e2e gate treats this sleeping host
|
||||
|
||||
@@ -111,6 +111,90 @@ Restart=always
|
||||
probe one ssh-context out returns the residential IP — a false leak alarm.
|
||||
- Ensure umask is set to `022` so downloaded files are readable by arr apps.
|
||||
|
||||
## E2E verification after any pipeline change
|
||||
|
||||
Run after ANY change to the torrent pipeline — WireGuard credential rotation,
|
||||
qB version change, gluetun reconfig, category re-seeding. Proves the full
|
||||
grab path (tunnel egress → qB download → category save path → clean
|
||||
removal). All commands from a dev box via `ssh frankbook-pro`; inside the
|
||||
Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
|
||||
`podman machine ssh`.
|
||||
|
||||
**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
|
||||
(NOT core — `core` is the podman-machine VM user, reached only via
|
||||
`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
|
||||
2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
|
||||
low MaxAuthTries: when password-bootstrapping use `-o
|
||||
PreferredAuthentications=password -o PubkeyAuthentication=no
|
||||
-o IdentitiesOnly=yes` or the key offers exhaust the attempts before the
|
||||
password is tried.
|
||||
|
||||
**1. Egress gate — MUST run inside gluetun:**
|
||||
|
||||
```bash
|
||||
podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
|
||||
```
|
||||
|
||||
Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config); judge by org,
|
||||
never country alone. NEVER run this one ssh-context out (bare `wget` on the
|
||||
Mac host): it silently returns the residential Comcast IP and reads as a
|
||||
hard-gate leak that does not exist. A residential reading ⇒ re-run inside
|
||||
the container before diagnosing.
|
||||
|
||||
**2. Login — generic cookie parse.** The QPW fetch is flaky (~1 in N returns
|
||||
empty); assert length before use — an empty-password run once produced a
|
||||
Forbidden loop that cost hours:
|
||||
|
||||
```bash
|
||||
QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
|
||||
[ ${#QPW} -eq 20 ] || exit 1
|
||||
SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
|
||||
http://127.0.0.1:8080/api/v2/auth/login \
|
||||
| grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
|
||||
curl -b "$SID" … # pass the WHOLE match
|
||||
```
|
||||
|
||||
WebUI from the LAN: `http://192.168.0.112:8080` (published on the gluetun
|
||||
unit). Username `admin`; password = the machine-local `QBIT_WEBUI_PASSWORD`
|
||||
(20 chars, generated once at provisioning by the deploy handler — never in
|
||||
the repo). Parse-bug 403s never reach qB as failed logins — no ban accrues;
|
||||
server-side truth is `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log`
|
||||
(`WebAPI login success/failure` + source IP). Repeated REAL failures accrue
|
||||
toward qB's 5-failures/3600 s IP ban.
|
||||
|
||||
**3. Add a legal test torrent.** Blender open-movie torrents (~300 MB, e.g.
|
||||
Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file
|
||||
(`-F "file=@…"`) and `category=Finished Torrents`. Never leave a real-looking
|
||||
grab behind.
|
||||
|
||||
**4. Poll and verify.** Poll `GET /torrents/info` (~20 s): state
|
||||
`downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading`
|
||||
at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents`
|
||||
(qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT
|
||||
inherit category paths — one more reason for the 4.6.7 pin). Confirm on
|
||||
disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is
|
||||
normal for the tunnel.
|
||||
|
||||
**5. Cleanup — delete only your artifact:**
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
|
||||
--data-urlencode "hashes=<literal-hash>" \
|
||||
--data-urlencode "deleteFiles=true" \
|
||||
http://127.0.0.1:8080/api/v2/torrents/delete # expect 200
|
||||
```
|
||||
|
||||
Target the known hash literally — the user's real grabs share the directory
|
||||
and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir
|
||||
gone, all other entries intact.
|
||||
|
||||
**Version-coupling constraints:** qB is pinned to 4.6.7 (Readarr's EOL
|
||||
client rejects qB ≥ 5's `204` login at runtime — see the pin rationale
|
||||
above). Do not unpin while Readarr exists. After ANY qB version flip,
|
||||
regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k
|
||||
iterations) — a 5.x-written hash line makes 4.6.7 reject every login.
|
||||
Rewrite while qB is stopped.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ARR-Stack]] — full *arr stack documentation
|
||||
|
||||
@@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
|
||||
distinguish challenge from deny at this release; grep the body for
|
||||
`Access Denied` / `Making sure you` instead.
|
||||
|
||||
## Guard-chain layer diagnosis (a user reports a guarded page "broken")
|
||||
|
||||
Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
|
||||
"broken / links gone / not loading" and the fault must be localized to
|
||||
server-side or client-side/session. Chain topology:
|
||||
`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
|
||||
→ Caddy :8017 (@home/@data handlers) → app/static`.
|
||||
|
||||
Layer-by-layer checks (run from the host via ssh):
|
||||
|
||||
1. **Caddy unit + guard containers**: `systemctl --user is-active
|
||||
caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
|
||||
'anubis|authentik|caddy'` — expect `Up … (healthy)`.
|
||||
2. **Anubis challenge page validity**: `curl -sk
|
||||
https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
|
||||
— a healthy challenge contains `"anubis_challenge"` JSON with
|
||||
`difficulty` and the main JS.
|
||||
3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
|
||||
-iE 'error|exception|traceback|500|provider'` — routine dramatiq task
|
||||
logs are normal.
|
||||
4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
|
||||
its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
|
||||
connection refused, which is EXPECTED, not a fault):
|
||||
`podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
|
||||
www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
|
||||
`/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
|
||||
this container has NO `-H` flag — use `--header "Host: …"`.)
|
||||
5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
|
||||
parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
|
||||
|
||||
Tool/behavior notes: the anubis container is distroless — `podman exec anubis
|
||||
sh` fails (`sh: executable not found`); inspect via its logs or the served
|
||||
challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
|
||||
podman IP) for ALL clients — rootless-podman published-port SNAT;
|
||||
clients are indistinguishable by IP and clearance is cookie-based. Do NOT
|
||||
mistake that for the bug. Homepage render: `app.js` does
|
||||
`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
|
||||
failure the catch sets `#summary` to `status unavailable: <err>` and the
|
||||
grid stays EMPTY — which a user reads as "links gone / not loading." A
|
||||
30-service status.json with a 9-service services.json renders fine
|
||||
(stateByName lookup).
|
||||
|
||||
**Decision:** if every layer is healthy (challenge valid, outpost clean,
|
||||
handlers 200, files render), the fault is client-side/session. STOP
|
||||
thrashing the live system — repeated failed browser challenges share the
|
||||
masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
|
||||
symptom (challenge spinning? login loop? empty grid? "status unavailable"
|
||||
text?) or the authentik login to drive the page in a real browser.
|
||||
|
||||
## Verify live
|
||||
|
||||
```
|
||||
|
||||
@@ -342,6 +342,168 @@ The silent re-auth path was proven end-to-end: after deleting
|
||||
`qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
|
||||
state and passed with no login prompt (2 tests, 2.9 s).
|
||||
|
||||
## Guard-chain flip recipe (adding a guarded vhost)
|
||||
|
||||
Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
|
||||
client → TLS edge → Anubis :8923 → outpost decision → internal
|
||||
`http://:8017` listener → Host-matched handler → real upstream.
|
||||
|
||||
**Order (never reorder):**
|
||||
|
||||
1. **Provider first.** Create the outpost ProxyProvider before touching
|
||||
routes, or authed traffic hits the outpost error page instead of the app.
|
||||
Fleet convention for `external_host` is the exact browser-facing origin
|
||||
(the inventory table carries both bare origins and trailing slashes;
|
||||
routing matches on hostname, verified 2026-09-02).
|
||||
`internal_host = http://caddy:8017`, all three flows set, and an
|
||||
Application attached — the two provisioning rules above.
|
||||
2. **Route flip in ONE commit/deploy**: the vhost becomes
|
||||
`reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
|
||||
inside the `http://:8017` listener add `@<svc> host <vhost>` +
|
||||
`handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
|
||||
edge, `:8017` routing table, and provider list change together.
|
||||
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
|
||||
|
||||
**Verification (all three):**
|
||||
|
||||
- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
|
||||
challenge page; grep the body for `challenge|not a bot|anubis` (~8
|
||||
markers on a live challenge). A 502 here instead of the challenge = chain
|
||||
mis-wired.
|
||||
- Post-auth routing proof: grep the LIVE config inside the container —
|
||||
`podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
|
||||
after the deploy prints its reload line; confirm the reload actually ran,
|
||||
don't trust exit 0.
|
||||
- Outpost reachability: `podman exec authentik-server getent hosts caddy`
|
||||
resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
|
||||
|
||||
**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
|
||||
000 (connection refused) — the post-auth listener lives in the caddy
|
||||
container's netns; probe inside it or grep the config. Non-browser clients
|
||||
cannot solve the JS challenge — before flipping, audit service-to-service
|
||||
traffic: ingestion/push paths must target internal addresses (`loki:3100`),
|
||||
never a public hostname of a guarded service. Exempt-by-construction
|
||||
categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
|
||||
endpoints, services with their own complete auth — document the exemption
|
||||
in the vhost comment.
|
||||
|
||||
## Driving the ORM (`ak shell`)
|
||||
|
||||
All authentik object work happens through the server container's ORM shell —
|
||||
no repo change, no deploy (these providers are NOT repo-managed; capture new
|
||||
provider+application pairs as a blueprint or setup script to avoid drift).
|
||||
|
||||
```bash
|
||||
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
|
||||
from authentik.core.models import User, Group
|
||||
...
|
||||
EOF
|
||||
```
|
||||
|
||||
- The heredoc IS the program: `ak shell` reads all of stdin and execs it
|
||||
(heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
|
||||
one).
|
||||
- Bootstrap prints JSON log lines, then
|
||||
`### authentik shell (2026.8.3)` and `229 objects imported automatically…`
|
||||
— pipe through `sed -n '/229 objects/,$p'` to drop the noise.
|
||||
- 229 models are pre-imported. Import paths (each wrong guess costs a failed
|
||||
exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
|
||||
`PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
|
||||
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
|
||||
`LDAPProvider` → `authentik.providers.ldap.models`;
|
||||
`authentik.core.models.application` does NOT exist as an import path; there
|
||||
is no `authentik.policies.group` module in this build.
|
||||
- For live DB truth prefer the postgres container
|
||||
(`podman exec authentik-postgres psql -U authentik -d authentik -At -c
|
||||
"select * from …"`) — **query with `select *` and stop guessing column
|
||||
names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
|
||||
FieldError — `type` moved to subclass tables; the outpost `enabled` column
|
||||
may not exist under that name; provider pk is `oauth2provider_ptr_id`).
|
||||
|
||||
## User provisioning playbook (user-level account)
|
||||
|
||||
There is no `User.create_user` classmethod, and `is_superuser` is NOT a
|
||||
constructor kwarg (it is a property derived from group membership) — passing
|
||||
it raises `TypeError: User() got unexpected keyword arguments`:
|
||||
|
||||
```python
|
||||
from authentik.core.models import User, Group
|
||||
u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
|
||||
u.set_password("...") # password arrives on stdin only, never argv
|
||||
u.save()
|
||||
for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
|
||||
u.groups.add(Group.objects.get(name=g))
|
||||
```
|
||||
|
||||
`path` defaults to `users` (regular). The access map (verified live
|
||||
2026-10-04): there are NO per-app policy bindings on the proxy providers
|
||||
(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
|
||||
access is decided entirely by GROUP MEMBERSHIP consumed downstream —
|
||||
|
||||
- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
|
||||
regex gate) and the homepage admin panel; regular users get only
|
||||
`category: user` panels from `services.json`.
|
||||
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
|
||||
"Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
|
||||
"Open WebUI roles" property mapping → `roles` claim. Verify live with
|
||||
`u.app_entitlements(app)`.
|
||||
- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
|
||||
(filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
|
||||
- `Non-Admin Users` = the plain-user marker group (precedents: alex,
|
||||
qa-e2e).
|
||||
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
|
||||
`openwebui_admins`, `authentik Admins`, `authentik Read-only`.
|
||||
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
|
||||
Plex, Seerr, Marreta) need no per-user group: any authenticated user
|
||||
passes the outpost.
|
||||
|
||||
Verify login behaviorally — `authenticate()` is the exact PasswordStage code
|
||||
path:
|
||||
|
||||
```python
|
||||
from django.contrib.auth import authenticate
|
||||
assert authenticate(request=None, username="ridley", password="...") # -> User
|
||||
assert authenticate(request=None, username="ridley", password="nope") is None
|
||||
```
|
||||
|
||||
Look flows up by slug, not name:
|
||||
`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
|
||||
"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
|
||||
Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
|
||||
enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
|
||||
|
||||
More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
|
||||
can be null (guard with `getattr`); `LDAPProvider` has
|
||||
`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
|
||||
`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
|
||||
`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
|
||||
(use `type(p).__name__`); `User` has no `.status`.
|
||||
|
||||
Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
|
||||
every outpost session fleet-wide; purge the stale `proxysession` rows
|
||||
instead (section above).
|
||||
|
||||
## Probing the outpost from inside the netns
|
||||
|
||||
Rootless podman gives each container its own netns; **host `ss -ltn` cannot
|
||||
see listeners inside other containers' namespaces** — a missing listener on
|
||||
the host is an observation artifact, not a fault. Distroless images have no
|
||||
shell (`podman exec authentik-server sh` fails), so probe by joining the
|
||||
namespace:
|
||||
|
||||
```
|
||||
podman run --rm --network container:<container> --entrypoint /bin/busybox \
|
||||
docker.io/library/busybox:latest \
|
||||
wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
|
||||
```
|
||||
|
||||
- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
|
||||
triggers globbing (`no matches found`).
|
||||
- Boot transience: immediately after a restart the same probe can return 503
|
||||
(configure task pending); poll until the answer changes before concluding.
|
||||
- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
|
||||
version — the field is `.HostConfig.NetworkMode`.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
---
|
||||
type: reference
|
||||
title: Homepage Status Probes
|
||||
description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
|
||||
tags: [homepage, status, probes, e2e, homepage-status]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Homepage Status Probes
|
||||
|
||||
The homepage is a static site —
|
||||
`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
|
||||
— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
|
||||
`scripts/deploy-remote.sh` refuses other targets). A checker
|
||||
(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
|
||||
each card's target and writes `/srv/homepage/status.json` (mode **0644** so
|
||||
Caddy can serve it — losing the mode 403s the page).
|
||||
|
||||
## The checker's vantage and its three false-down shapes
|
||||
|
||||
The checker probes TCP/HTTP **from the host netns**. It only has three states
|
||||
— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
|
||||
never render honestly and must be dropped or re-pointed, not "fixed".
|
||||
|
||||
1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
|
||||
only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
|
||||
card says down while PMS answers `200` over HTTPS. Fix:
|
||||
`"method":"tcp"` (connect is the honest liveness signal; the public URL
|
||||
stays HTTP because Caddy re-keys).
|
||||
2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
|
||||
compose→quadlet move moved the dashboard to `9119` and the webui to
|
||||
`8787`; nothing listened on 8090, so a healthy stack read `down`.
|
||||
3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
|
||||
`systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
|
||||
and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
|
||||
netns-scoped, so from the host these are unreachable *by construction* —
|
||||
no probe value makes them honest. Drop those cards.
|
||||
|
||||
A host-vantage probe can only reach a listener **published to the host**
|
||||
(rootlessport/pasta); unpublished ports are invisible from the host netns.
|
||||
See [[Container Host Reachability]] for the netns rules.
|
||||
|
||||
## Triage procedure
|
||||
|
||||
1. Read the card's probe target in `shared/homepage/services.json`; note
|
||||
method (`tcp` vs `http`).
|
||||
2. Does that port have a **host-visible listener**?
|
||||
`podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
|
||||
`podman inspect <c> --format '{{json .NetworkSettings.Networks}}
|
||||
{{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
|
||||
published nothing → structurally unprobeable from the host).
|
||||
3. Probe from the **correct vantage**: from a bridge-net container the host
|
||||
is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
|
||||
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
|
||||
which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
|
||||
bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
|
||||
they expand **inside** `podman exec bash -c`.
|
||||
4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
|
||||
SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
|
||||
killer) vs false down (running + unreachable target).
|
||||
5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
|
||||
only exposure is an internal netns; never invent a probe target.
|
||||
6. Verify by re-running the checker path, not by eyeballing: after deploy,
|
||||
re-read `/srv/homepage/status.json` and diff the `down` list
|
||||
before/after.
|
||||
|
||||
Removing a card's `probe` key without a code change crashes the checker
|
||||
(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
|
||||
fix. The `description` field is consumed only by `app.js` `descHTML()`, so
|
||||
adding a `details` object (role/spec/inputs/use) is schema-safe:
|
||||
`tests/homepage_config.bats` checks only name/category/section/probe.
|
||||
|
||||
## Local render check (before every deploy)
|
||||
|
||||
The page fetches `services.json` + `status.json` at load; `file://` is
|
||||
CORS-blocked, so serve over HTTP:
|
||||
|
||||
```bash
|
||||
cd shared/homepage
|
||||
python3 -c "import json; s=json.load(open('services.json')); \
|
||||
json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
|
||||
python3 -m http.server 8899 --bind 127.0.0.1
|
||||
```
|
||||
|
||||
Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
|
||||
(panels render after the async fetch; checking immediately returns 0 panels),
|
||||
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
|
||||
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
|
||||
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
|
||||
visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
|
||||
time) — harmless.
|
||||
|
||||
The scratch `status.json` is untracked and NOT git-ignored — move it to
|
||||
`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
|
||||
|
||||
CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
|
||||
`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
|
||||
`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
|
||||
reset so `?bare=1` strips it to black.
|
||||
|
||||
## Deploy
|
||||
|
||||
```
|
||||
./scripts/deploy.sh dankclaw --service homepage --skip-e2e
|
||||
```
|
||||
|
||||
Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
|
||||
datetime), restarts the `homepage-status` timer. Verify live:
|
||||
`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
|
||||
|
||||
**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
|
||||
asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
|
||||
Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
|
||||
no remote wake) times out that test — a pre-existing environmental failure,
|
||||
not a regression. All non-Mac-mini e2e tests pass.
|
||||
|
||||
## e2e raw-IP *arr failures — probe from the browser's box, not a peer
|
||||
|
||||
When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
|
||||
Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
|
||||
so a healthy probe from dankclaw proves nothing about the **browser's** path:
|
||||
|
||||
```bash
|
||||
# 1. From the box that RUNS the browser:
|
||||
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
|
||||
# 2. From a LAN peer:
|
||||
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
|
||||
--max-time 6 http://192.168.0.112:$p/; done'
|
||||
# 3. Liveness (ICMP, not TCP):
|
||||
ping -c 2 -W 3 192.168.0.112
|
||||
```
|
||||
|
||||
| peer probe | dev-box probe | ping | meaning |
|
||||
|---|---|---|---|
|
||||
| 200 | 200 | ok | transient — re-run the spec |
|
||||
| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
|
||||
| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
|
||||
|
||||
Third row: confirm the Mac is innocent through the dankclaw jump
|
||||
(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
|
||||
`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
|
||||
--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
|
||||
between the dev box and the Mac.
|
||||
|
||||
Don't hold a deploy hostage to a known-environmental red gate: verify the
|
||||
deployed service **directly** (its own ledger/health/real completion), state
|
||||
the gate was skipped or partially red and why (`--skip-e2e` prints the
|
||||
warning, so the skip is recorded), and report the residual gap plainly
|
||||
("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
|
||||
probe returns 200; deploy verified by other means"). Do not weaken or skip
|
||||
the spec — it guards a real SSO-plus-RBAC contract.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Container Host Reachability]] — the netns rules behind the vantage
|
||||
- [[Observability Provisioning]] — the monitoring stack reading the same fleet
|
||||
- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
|
||||
- [[Testing Policy]] — the seam/e2e contract this page's checks live in
|
||||
@@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the
|
||||
| HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
|
||||
| Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
|
||||
|
||||
## Per-client fix: resolver drop-in + browser DoH bypass
|
||||
|
||||
Symptom: a LAN machine gets `connection timeout` (curl rc=28) on
|
||||
`https://*.franksdank.cloud` while the server is healthy. Root cause: the
|
||||
public DNS hands out the WAN IP `71.227.227.172`, which has **no 443
|
||||
port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN
|
||||
answer → SYN to a filtered port → timeout. There are TWO resolver layers to
|
||||
check, in order:
|
||||
|
||||
**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in
|
||||
sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route
|
||||
servers:
|
||||
|
||||
```bash
|
||||
dig +short www.franksdank.cloud @192.168.0.143 # expect 192.168.0.143
|
||||
dig +short www.franksdank.cloud @1.1.1.1 # expect 71.227.227.172 (the trap)
|
||||
getent hosts www.franksdank.cloud # what the OS actually resolves
|
||||
curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \
|
||||
https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n' # 200 = server fine
|
||||
|
||||
sudo mkdir -p /etc/systemd/resolved.conf.d
|
||||
sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
|
||||
[Resolve]
|
||||
DNS=192.168.0.143
|
||||
Domains=~franksdank.cloud ~franksdank.casa
|
||||
EOF
|
||||
sudo systemctl restart systemd-resolved && resolvectl flush-caches
|
||||
```
|
||||
|
||||
The `~` (route-only) prefix is load-bearing: it routes only those domains to
|
||||
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
|
||||
`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
|
||||
|
||||
**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
|
||||
Layer 1 is verified but the BROWSER still times out, the browser is doing
|
||||
its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR
|
||||
provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it:
|
||||
`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead
|
||||
end). Diagnosis details:
|
||||
|
||||
- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT
|
||||
`~/.mozilla/firefox/` — find it via the running process if the default
|
||||
path is empty.
|
||||
- `grep network.trr. <profile>/prefs.js` — **no entries = shipped default,
|
||||
and since Firefox 113 the shipped default is DoH ON.**
|
||||
|
||||
Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy
|
||||
& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** →
|
||||
`franksdank.cloud` (and `franksdank.casa`). The exception list = domains
|
||||
that do NOT use DoH; they fall back to the system resolver, which has the
|
||||
route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome:
|
||||
Settings → Privacy and security → Security → "Use secure DNS" → off, or
|
||||
"Use the same settings as your system". Do NOT hand-edit `prefs.js` while
|
||||
the browser is running — it holds prefs in memory and rewrites the file on
|
||||
exit, clobbering the edit.
|
||||
|
||||
This is a per-client fix: every LAN client that must reach the homelab by
|
||||
name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not
|
||||
bypass it with DoH. The server side needs no change. Verified on blankpad
|
||||
2026-09-21: the drop-in was present since 09-12, curl returned 200, and
|
||||
Firefox 155 with default (ON) DoH was the remaining hole.
|
||||
|
||||
## Porkbun zone ops (franksdank.cloud)
|
||||
|
||||
Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so
|
||||
the API key/secret sourced from `/home/dank/serverAppConfigs/.env`
|
||||
(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe
|
||||
scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' <
|
||||
/tmp/script.sh`.
|
||||
|
||||
Endpoints (all POST, JSON body):
|
||||
|
||||
- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud`
|
||||
— auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong
|
||||
here). Response shape is **top-level `.records[]`**, NOT
|
||||
`.content.records[]` as some docs claim — parse with
|
||||
`jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
|
||||
- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
|
||||
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
|
||||
`type`, `content`, `ttl`, `prio`).
|
||||
|
||||
Quirks that bite (verified live 2026-08-31):
|
||||
|
||||
1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud`
|
||||
yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare
|
||||
label (`ai`) and the API appends the zone itself. Verify after every
|
||||
rename with a retrieve before declaring done.
|
||||
2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of
|
||||
rapid mutations makes some calls return `<!DOCTYPE html>...` where jq
|
||||
then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do
|
||||
not assume the call succeeded — sleep ~8 s and retry the failed ids;
|
||||
verify by re-listing the zone at the end.
|
||||
3. Deletes/renames are per-record-id: always retrieve first, map name→id,
|
||||
then mutate by id.
|
||||
|
||||
Rename procedure (vhost rename = DNS + Caddy together):
|
||||
|
||||
1. `retrieve` → collect ids for the renamed names.
|
||||
2. `edit` each id with the new **relative** label, preserving
|
||||
type/content/ttl.
|
||||
3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
|
||||
4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the
|
||||
Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test
|
||||
fails the suite if any Caddyfile vhost lacks an addn-hosts A record on
|
||||
192.168.0.143.
|
||||
5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed
|
||||
paths). Capture the suite's true exit with
|
||||
`out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit
|
||||
code behind `tail`'s own status and has shipped untested commits here.
|
||||
6. Verify live: `dig +short <name> @192.168.0.143` then
|
||||
`curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302;
|
||||
first hit triggers dns-01 issuance, which works without inbound port 80).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
type: reference
|
||||
title: Observability Provisioning
|
||||
description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
|
||||
tags: [observability, prometheus, grafana, provisioning, deployment]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Observability Provisioning
|
||||
|
||||
The monitoring stack is **provisioned as code**: dashboards, datasources,
|
||||
scrape jobs and alert rules live in the repo and ride the normal deploy flow.
|
||||
No hand-edits inside containers — they are wiped on recreate and never
|
||||
committed.
|
||||
|
||||
## The config triad
|
||||
|
||||
| Repo path | What it holds |
|
||||
|---|---|
|
||||
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
|
||||
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
||||
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
||||
|
||||
Targets and datasources reference **container names** (`loki:3100`,
|
||||
`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
|
||||
by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
|
||||
are host-netns services, not containers. Prometheus 3 refuses HTML responses
|
||||
as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
|
||||
exporter (see [[Stale Signal Audit]]).
|
||||
|
||||
## The deploy path
|
||||
|
||||
`scripts/deploy-remote.sh` carries two copier functions:
|
||||
|
||||
- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
|
||||
|
||||
`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
|
||||
`shared/grafana/**` to the `prometheus`/`grafana` services, so
|
||||
`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
|
||||
touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
|
||||
exist, the copier functions exist, and the provisioned paths match the units'
|
||||
mounts.
|
||||
|
||||
## Entry points
|
||||
|
||||
**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
|
||||
vantage first: container name if the exporter is a container on the same
|
||||
netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
|
||||
if it is an HTTP probe, not a metrics endpoint.
|
||||
|
||||
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
|
||||
Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
|
||||
Rules reload with the prometheus service.
|
||||
|
||||
**Add a datasource** — drop a YAML file into
|
||||
`shared/grafana/provisioning/datasources/`. Point it at the container name of
|
||||
the backend (`http://prometheus:9090`), not loopback.
|
||||
|
||||
**Add a dashboard** — drop the JSON into
|
||||
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
|
||||
unified storage, not (only) the SQLite table — grep the API, not the DB, when
|
||||
verifying (see the verify section of [[Fleet Audit and Posture]]).
|
||||
|
||||
## Verify through the APIs, not the filesystem
|
||||
|
||||
```bash
|
||||
# prometheus: rules and targets actually loaded
|
||||
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
|
||||
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
|
||||
# grafana: datasource + dashboard provisioned
|
||||
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
|
||||
```
|
||||
|
||||
Health answers from a stale process and provisioned resources can hide in
|
||||
unified storage — a file existing on disk is not proof the running server
|
||||
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
||||
table (refused = outage, 404/HTML = config defect).
|
||||
|
||||
## Current delivered state (2026-09-17)
|
||||
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
|
||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
||||
via `host.containers.internal`) → signal-cli. Some targets read down for
|
||||
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
||||
the blackbox exporter. See [[Observability]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — stack topology and the alert delivery chain
|
||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
|
||||
- [[Quadlet Lifecycle]] — the units that host these services
|
||||
- [[Deploy State Triage]] — how the planner selects these services
|
||||
@@ -271,6 +271,83 @@ graph TD
|
||||
6. Step up in size: 9B, then gpt-oss-20b.
|
||||
7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
|
||||
|
||||
## Registering packs absent from the catalog (NPU2)
|
||||
|
||||
Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
|
||||
to the FLM registry. Registry facts: user registry
|
||||
`/home/dank/.config/flm/model_list.json`, shape
|
||||
`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
|
||||
file_url, size, flm_min_version, files[], vlm, default_context_length,
|
||||
max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
|
||||
resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
|
||||
env var set lists custom tags (`⏬` = not downloaded).
|
||||
|
||||
1. Back up the registry JSON first (`cp -a`).
|
||||
2. Insert the entry via a python json edit + atomic `os.replace`. Set
|
||||
`files` to the FULL HF file list — **including `vision_weight.q4nx` even
|
||||
for text-only packs**: the qwen3.5-family loader opens it unconditionally
|
||||
and a missing file makes the first request HANG silently (no error log).
|
||||
3. **Do not use `flm pull <tag>` for custom entries** — it throws
|
||||
`json.exception out_of_range key '<tag>:<size>' not found` while building
|
||||
the download list. Place files manually into
|
||||
`/home/dank/.config/flm/models/<PackName>/` with
|
||||
`curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
|
||||
— plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
|
||||
weights; verify byte counts against the HF tree API before trusting a
|
||||
download.
|
||||
4. Kernel packs: copy from the same-shape family pack (shape-match
|
||||
`config.json` on hidden_size/num_hidden_layers/head_dim first) into
|
||||
`/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
|
||||
(root-owned dir).
|
||||
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
|
||||
the new process loads with all files present.
|
||||
6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
|
||||
`/v1/models` alone proves registration, not loadability.
|
||||
|
||||
Hard constraints: only ONE `flm serve` process can hold the NPU hardware
|
||||
context — a second concurrent serve process fails every load with
|
||||
`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
|
||||
0x2000003); never run scratch/verify instances while production holds the
|
||||
NPU. Tag switching happens INSIDE one serve process; the first request
|
||||
after a model switch pays ~6–11 s cold-load cost (visible in
|
||||
`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
|
||||
pattern present in your own command line — use bracket-guarded patterns
|
||||
(`[q]wen3.5:4b`) or exact PIDs.
|
||||
|
||||
## Tag load-failure diagnosis (200 with an error body)
|
||||
|
||||
`POST /v1/chat/completions` answers **HTTP 200** with a flat
|
||||
`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
|
||||
returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
|
||||
200. A status-code-only client reads this as a successful empty completion —
|
||||
**always check for the `error` key**. Two distinct causes, distinguished via
|
||||
`journalctl -u <flm-service> --full`:
|
||||
|
||||
1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
|
||||
Correct behavior, not a bug.
|
||||
2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
|
||||
downloads missing weight files on the fly (first request hangs ~90–110 s),
|
||||
then fails `Failed to load model: No such file
|
||||
'/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
|
||||
attempts fail fast (~2 s). Root cause: the serve script exports
|
||||
`FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
|
||||
kernel lookup resolves under the user dir, which only contains symlinks
|
||||
someone linked. **A tag listed in `/v1/models` is NOT proof it is
|
||||
loadable on this host.** Fix: one symlink per missing pack
|
||||
(`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
|
||||
/home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
|
||||
symlink the first request loaded and answered in 7.4 s wall.
|
||||
|
||||
Switch-cost behavior: the `model` field routes per-request (no restart
|
||||
needed); a request whose model differs from the resident one triggers a full
|
||||
reload (7–11 s wall for an 8-token completion on ~4B-class packs);
|
||||
alternating tags pays the swap each time. Single-lane NPU: concurrent
|
||||
requests queue FIFO behind the current lock holder (`NPU Locked!` →
|
||||
`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
|
||||
the journal for `Failed to load model` (the quoted xclbin path names the
|
||||
missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
|
||||
installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
|
||||
|
||||
## Incident: qwen3.6-moe unreachable (2026-08-16)
|
||||
|
||||
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
type: reference
|
||||
title: FreeToken Builds
|
||||
description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
|
||||
tags: [freetoken, build, dankclaw, vllm, mtp]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# FreeToken Builds
|
||||
|
||||
FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
|
||||
vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
|
||||
different dependency traps. Both were verified end-to-end 2026-09-11/12.
|
||||
|
||||
## Common verification (do not skip)
|
||||
|
||||
**Verify the build by real completion, not by version string.** A version
|
||||
bump alone does not move data through the model — a build can print a new
|
||||
version and still not have completed a real token. Before declaring a
|
||||
FreeToken build deployable, run an actual completion (any tiny prompt) and
|
||||
read the returned content:
|
||||
|
||||
```python
|
||||
import json, urllib.request
|
||||
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
|
||||
json.dumps({"model": "frank-small", "messages": [{"role": "user",
|
||||
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
|
||||
print(json.load(urllib.request.urlopen(req, timeout=600)))
|
||||
```
|
||||
|
||||
A 200 with a real `choices[0].message.content` is the only build green.
|
||||
|
||||
## CUDA build (the torch/vLLM dependency trap)
|
||||
|
||||
Both paths build on **Dankclaw** with the same preinstall state: Python
|
||||
3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
|
||||
download.pytorch.org/whl/nightly/cu130, installed with
|
||||
`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
|
||||
--index-url https://download.pytorch.org/whl/nightly/cu130` — use
|
||||
`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
|
||||
version). Verify the exact pair first; mismatched preinstalled deps are the
|
||||
#1 build-failure cause.
|
||||
|
||||
Then install the **matching vllm dev wheel**:
|
||||
|
||||
```bash
|
||||
uv pip install --index-strategy unsafe-best-match \
|
||||
--index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu130
|
||||
```
|
||||
|
||||
The wheel URL pattern is fixed:
|
||||
`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
|
||||
— read it back from vLLM's CI-run page (`run details →
|
||||
build_and_install_package` job, "install the vllm wheel" step). **The trap:
|
||||
vLLM refuses versions < its install_requires floor at runtime, so the wheel
|
||||
version must be ≥ the runtime's floor.** A wheel built at a lower base
|
||||
(0.5.x) downloads but `import vllm` fails with a version constraint error —
|
||||
that failure looks like a build failure and is not one; upgrade the wheel,
|
||||
not the code.
|
||||
|
||||
If `uv pip install -e . --no-deps` refuses (some repos pin the
|
||||
`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
|
||||
after `python -m venv --copies .venv && source .venv/bin/activate` and
|
||||
`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
|
||||
re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
|
||||
|
||||
## ROCm build (gfx1151 via TheRock nightly)
|
||||
|
||||
Same venv discipline, ROCm wheels from TheRock's nightly index:
|
||||
|
||||
```bash
|
||||
uv pip install --index-strategy unsafe-best-match \
|
||||
--index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
|
||||
"torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
|
||||
```
|
||||
|
||||
Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
|
||||
repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
|
||||
prebuilt extension wheels into the tree (the repo's `setup.py` skips
|
||||
compiling them when found). The ROCm build also needs
|
||||
`install-fastrak-gfx1151.sh` from the repo's scripts and
|
||||
`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
|
||||
x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
|
||||
`franksdank/freetoken-ops-gfx1151` — fetch it once and point
|
||||
`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
|
||||
the extension API changed.
|
||||
|
||||
`--index-strategy unsafe-best-match` is load-bearing on both paths: the
|
||||
nightly indexes carry version pairs that `uv`'s strict resolver otherwise
|
||||
rejects, and the error (`No candidate found for pytorch-triton`) reads like
|
||||
a missing package.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
|
||||
- [[Llama Cpp Serving Sizing]] — the harness measurement discipline
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
type: reference
|
||||
title: Heterogeneous Split Triage
|
||||
description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
|
||||
tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Heterogeneous Split Triage
|
||||
|
||||
Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
|
||||
the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
|
||||
across device A and device B" with arithmetic instead of enthusiasm.
|
||||
|
||||
## 1. Characterise the link (bytes vs latency)
|
||||
|
||||
```bash
|
||||
lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:" # capability vs actual
|
||||
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
|
||||
```
|
||||
|
||||
Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
|
||||
upshifts under load. Measure while the device is busy, or you size the design
|
||||
against the wrong number. Per-hop cost against the hidden state
|
||||
(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
|
||||
Gen4 x4.
|
||||
|
||||
- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
|
||||
for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
|
||||
never the constraint.**
|
||||
- The real budget line is **hop COUNT**: every device boundary is a
|
||||
host-side sync. A runtime that splits per *op* hits ~100 splits/token and
|
||||
pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
|
||||
assume.
|
||||
- **Design rule: partition by where the weights live, never by where the
|
||||
activations flow.** Resident weights, activations crossing.
|
||||
|
||||
## 2. Get the workload's real bottleneck (KV is usually it)
|
||||
|
||||
```
|
||||
KV/token = kv_bytes_per_position × context
|
||||
```
|
||||
|
||||
Attention reads the whole cached context every token, so KV/token scales
|
||||
with context — compare it against expert/weight traffic per token and total
|
||||
token time. At long context on a hybrid arch (few attention layers, many
|
||||
positions), **KV dominates**. Consequences:
|
||||
|
||||
- The highest-leverage placement is **KV cache on the fastest-memory device,
|
||||
together with its attention compute** — never KV alone: separating them
|
||||
puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
|
||||
hundreds of ms/token).
|
||||
- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
|
||||
attention_trunk ≤ VRAM`.
|
||||
|
||||
## 3. What placement never buys
|
||||
|
||||
Prefill is compute-bound and expert weights are read once per batch, so
|
||||
moving them to faster memory buys nothing while adding a serial stage.
|
||||
Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
|
||||
(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
|
||||
GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
|
||||
`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
|
||||
decode gain to within ~2 points — usable for decode projections, and it
|
||||
assumes sequential stages (true at batch 1; no pipelining).
|
||||
|
||||
## 4. Is the second device even a separate pool?
|
||||
|
||||
On a UMA APU, check before assuming additivity:
|
||||
|
||||
```bash
|
||||
cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small)
|
||||
cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM
|
||||
rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees
|
||||
```
|
||||
|
||||
A large GTT window means the "GPU memory" is the *same* LPDDR under one
|
||||
memory controller: adding a stage there adds a compute engine, not capacity
|
||||
or bandwidth. Only the discrete card's VRAM is additive. Fencing more
|
||||
(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
|
||||
|
||||
## 5. Speculative decoding as the "cheap" split
|
||||
|
||||
A drafter on the *other* device is bandwidth-free (token ids per round, tens
|
||||
of bytes), so it is tempting. Before proposing it:
|
||||
|
||||
- the drafter needs the target's **exact tokenizer** for a plain draft model
|
||||
(compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
|
||||
must be a head **trained for that target** (EAGLE/DFlash/MTP are
|
||||
conditioned on the target's hidden states and are not portable between
|
||||
models);
|
||||
- a plain drafter costs **K sequential forwards per round**, so it must be
|
||||
10–50× faster than the target. Break-even with `r = t_draft/t_target`:
|
||||
`rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
|
||||
(a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
|
||||
- compare against the **incumbent**, not the no-draft baseline: a target
|
||||
already running its own MTP head is a high bar;
|
||||
- check the runtime accepts an external drafter at all — closed engines may
|
||||
enumerate a fixed drafter set and 400 anything else (capability gates
|
||||
keyed on the loaded checkpoint). That is a design boundary, not a config
|
||||
gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
|
||||
(draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
|
||||
|
||||
## 6. Expressing a split (llama.cpp, the usual vehicle)
|
||||
|
||||
```bash
|
||||
-dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest)
|
||||
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (KV co-location)
|
||||
```
|
||||
|
||||
- buffer types match by name at load (`ggml_backend_buft_name`), so device
|
||||
names work as `-ot` values;
|
||||
- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
|
||||
- several backends in one binary is fine (CUDA + Vulkan is the practical
|
||||
pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
|
||||
build's CMakeCache tells you what needs adding;
|
||||
- contiguous ranges are the WRONG split when KV bandwidth dominates: they
|
||||
drag half the attention layers (and their KV) back to the slow pool.
|
||||
|
||||
## 7. Hazards checklist
|
||||
|
||||
- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
|
||||
are read from; moving them is pure loss.
|
||||
- Keep `mmap` on for models with huge embedding tables (anonymous memory
|
||||
gets the process cgroup-killed).
|
||||
- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
|
||||
— check before assuming optimisations apply.
|
||||
- A brand-new backend on the second device may not implement the arch's ops
|
||||
at all: make the first measurement a **baseline of the runtime on that
|
||||
device alone** — a split's arithmetic is worthless if the baseline is
|
||||
worse than the incumbent engine.
|
||||
- A split consumes the fast device entirely — check what else it was
|
||||
serving before trading it away.
|
||||
- Benchmark discipline on production hosts: stop and mask first
|
||||
([[Fleet Ops Playbook]]).
|
||||
|
||||
## 8. Verdict shape
|
||||
|
||||
Rank and report; prefer the boring answer when it wins:
|
||||
|
||||
1. **Two independent endpoints** (zero shared traffic) — often the real
|
||||
answer to "what's a useful combination", especially over a narrow link.
|
||||
2. **One process, weights-resident, KV co-located with attention on the
|
||||
fast pool** — promising and quantifiable (+% decode at long context);
|
||||
needs a dual-backend build and a baseline measurement.
|
||||
3. **Hot-expert cache / per-token weight streaming** — affordable only if
|
||||
the miss set per token is small (a few experts; check
|
||||
`misses × bytes × tok/s` against link bandwidth) and overlapped; injects
|
||||
PCIe latency into the critical path; no mainstream runtime implements a
|
||||
dynamic expert cache.
|
||||
4. List explicitly what does NOT help: TP/collectives across unequal
|
||||
devices, KV on the far device, model loading over the link,
|
||||
prefill-motivated placements.
|
||||
|
||||
## Related
|
||||
|
||||
- [[PCIe Link and Atomics]] — the link and collective walls underneath
|
||||
- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
|
||||
- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic
|
||||
@@ -0,0 +1,271 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Bench Windows and Measurement
|
||||
description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
|
||||
tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# LLM Bench Windows and Measurement
|
||||
|
||||
Fleet law first: **STOP the production service before any GPU experiment**
|
||||
(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
|
||||
GPU and corrupts results; an unisolated GPU experiment has precedent for
|
||||
wedging a host until power cycle. Reference harnesses:
|
||||
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
|
||||
`scripts/autoresearch/frank-primary-arms.sh`.
|
||||
|
||||
## 1. The window structure (one config per arm, a driver owns the window)
|
||||
|
||||
1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
|
||||
no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
|
||||
EXITED containers and can never clear — `rm -f` the exited one first) AND
|
||||
port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
|
||||
`ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
|
||||
then polls `/health` gets **200 from the still-running PRODUCTION engine**
|
||||
— the arm "passes" with production's boot log, then `podman exec` dies
|
||||
with `container state improper`.
|
||||
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
|
||||
LLM hosts — one card, one server). The serve script takes its config via
|
||||
env-file override (`FRANK_SMALL_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
|
||||
pointing at a staged copy; the checkout is never mutated.
|
||||
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
|
||||
the loop — a refused boot dies in one poll instead of 20 min. Capture the
|
||||
serve script's FULL stdout on failure (a `tail -8` loses engine-init root
|
||||
causes; raise happens above the tail).
|
||||
4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
|
||||
SpecDecoding lines) — they survive measure crashes.
|
||||
5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
|
||||
`podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
|
||||
before the next boot; restore production from the EXIT trap and verify the
|
||||
**conjunction oracle**: health 200 + `is-active` + `is-enabled` +
|
||||
mask-symlink clean. A bare `systemctl start` is not a restore oracle.
|
||||
(`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
|
||||
enough: systemd never auto-restarts an explicitly stopped unit.)
|
||||
6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
|
||||
mixed speeds.
|
||||
|
||||
**Destructive tails (2026-10-01):** a scratch script whose tail does
|
||||
`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
|
||||
probe loop survived the driver kill — it can destroy a just-deployed
|
||||
production engine (signature: deploy exits green, then seconds-to-minutes
|
||||
later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
|
||||
crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
|
||||
and kill strays by explicit PID; killing the dev-side job does NOT kill the
|
||||
host-side `deploy-remote.sh` (two of them once managed the same unit).
|
||||
|
||||
Two failure shapes inside the window that read as "the bench died":
|
||||
|
||||
- **Stale reads from a stopping container.** `podman stop` answers while the
|
||||
engine inside is still flushing; a `podman logs` taken in that window
|
||||
returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
|
||||
`podman wait`) before reading logs, or you will write "no results" into
|
||||
the table for a run that has them.
|
||||
- **A D-state worker wedges the stop.** A bench child that wedged in the
|
||||
kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
|
||||
then the container is `Stopping` forever. Kill by container first
|
||||
(`podman kill <ctr>` → SIGKILL path), verify the process is gone from
|
||||
`podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
|
||||
window script proceed to a redeploy with a zombie bench worker holding
|
||||
VRAM.
|
||||
|
||||
## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
|
||||
|
||||
**What llama-bench cannot express — decide BEFORE designing:** no
|
||||
`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
|
||||
activation buffers fit beside the DEPLOYED KV — only the real unit answers
|
||||
that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
|
||||
(`tg` numbers are the drafterless baseline, useful as a control only), and a
|
||||
warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
|
||||
cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
|
||||
~zero depth and meaningless for long conversations.
|
||||
|
||||
**The separator trap:** `/` separates devices WITHIN one configuration, `,`
|
||||
separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
|
||||
single value 36 → everything on one card. `--tensor-split 36,29` (server) →
|
||||
`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
|
||||
|
||||
**Harness shape:** one invocation per cell (an infeasible cell calls
|
||||
`exit(1)` and kills every later cell of a cross-product invocation — and
|
||||
infeasibility is usually the question); TSV row per cell as it completes +
|
||||
`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
|
||||
--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
|
||||
intended looks fast); resume by skipping non-empty cell outputs; stop the
|
||||
service in the harness, restore from `trap … EXIT`, verify afterwards. Match
|
||||
the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
|
||||
|
||||
**Run under a transient unit, not a shell:**
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/<harness>.sh
|
||||
```
|
||||
|
||||
Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
|
||||
8 MiB and fails the load for an unrelated reason. This also survives ssh
|
||||
disconnects. **Boot rungs under the production environment, not a bench
|
||||
shell** — a rootless shell probe booted 3×262144 bf16 that production refused
|
||||
(systemd `User=` + memlock change what the pinned-expert path holds):
|
||||
`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
|
||||
Restart=no env VARFILE=… /path/serve.sh`.
|
||||
|
||||
**Verify before burning hours:** dry-run against a STUB binary with
|
||||
`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
|
||||
including one forced-`exit 3` cell — prove the TSV row, failure capture,
|
||||
resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
|
||||
nonexistent file. **Reading:** bracket the sweep (reference cell first AND
|
||||
last); check per-cell `stddev_ts` (a difference smaller than spread is not a
|
||||
result); single-sequence decode is usually flat across batch/ubatch (prefill
|
||||
knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
|
||||
smaller-batch cells); report what the grid cannot prove (drafter, slots, real
|
||||
context).
|
||||
|
||||
## 3. HTTP benches of server-side parameters
|
||||
|
||||
`llama-bench` benches the bench binary, not the running server — server-side
|
||||
params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
|
||||
HTTP:
|
||||
|
||||
- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
|
||||
`prompt_tokens / wall_time`.
|
||||
- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
|
||||
spec acceptance all stabilize over thousands of tokens); measure
|
||||
`predicted_per_second` from `timings` or `completion_tokens / wall_time`;
|
||||
temperature 0.
|
||||
|
||||
Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
|
||||
`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
|
||||
sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
|
||||
reasoning parser streams zero deltas — use `/v1/completions`.
|
||||
|
||||
## 4. The measurement law (under speculative decoding this is absolute)
|
||||
|
||||
**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
|
||||
mapping; chunk counting reported 8.3 tok/s for a config whose server counters
|
||||
said 23–26 — a 3x false regression that nearly abandoned the change that
|
||||
worked. Always:
|
||||
|
||||
```python
|
||||
decode_tok_s = usage.completion_tokens / (wall - ttft)
|
||||
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
|
||||
```
|
||||
|
||||
vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
|
||||
measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
|
||||
them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
|
||||
acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
|
||||
temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
|
||||
(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
|
||||
repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
|
||||
before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
|
||||
nearly shipped the wrong setting.
|
||||
|
||||
## 5. Honest methodology
|
||||
|
||||
- **Match the corpus to the workload.** Token soup under-reports speculative
|
||||
decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
|
||||
settings; acceptance, not step time, moves throughput. Build the corpus
|
||||
from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
|
||||
`--custom-output-len`.
|
||||
- **Salt every run** with a unique per-arm token in the first bytes — it is
|
||||
the prefix-cache key (and a proxy routing key). Without it arm 2 measures
|
||||
arm 1's warm cache.
|
||||
- **Two workload shapes need two clients:** distinct prompts (batch
|
||||
behaviour, prefill parallelism) and sequential turns with growing history
|
||||
(TTFT of the first streamed chunk per turn — the only measurement that
|
||||
exposes prefix-cache behaviour). These pull in opposite directions: a cold
|
||||
batch spreads prefill across replicas, a conversation's next turn only pays
|
||||
if it lands where its prefix is cached.
|
||||
- **Read the engine's ledger, not the card:** `Available KV cache memory` /
|
||||
`GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
|
||||
request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
|
||||
is a LIFETIME average across every prompt the replica ever saw — send the
|
||||
same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
|
||||
→ 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
|
||||
full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
|
||||
chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
|
||||
three points before treating a knob as a lever.
|
||||
- **Soak before believing:** 15–30 min of concurrent families, count failures
|
||||
AND latency; report hit/miss split and medians with min/max — the median of
|
||||
a bimodal distribution describes neither mode.
|
||||
- **Verification that is not the benchmark:** run the client the fleet
|
||||
actually uses; and a deployed unit is not proven until an unattended reboot
|
||||
brings it back.
|
||||
- **Record the "no" answers** with numbers and mechanism — placement balance,
|
||||
concurrency admission and chunk-size tweaks all measured flat or worse
|
||||
here; the wall was arithmetic (cached prefixes + live sequences share one
|
||||
pool).
|
||||
|
||||
## 6. Sweep hygiene (the arms that produced wrong verdicts)
|
||||
|
||||
- **Prefetch uniformity:** every arm in a sweep carries the same
|
||||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
|
||||
prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
|
||||
12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
|
||||
verdict.
|
||||
- **Knob-landed preflight:** for every `-e` knob, confirm
|
||||
`podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
|
||||
produces a "parity" arm that is really the default (v1 of the gather sweep
|
||||
lost all 3 arms this way; serve-script hooks are additive and can drop
|
||||
vars).
|
||||
- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
|
||||
deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
|
||||
compare against default-off, not "incumbent + one flag".
|
||||
- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
|
||||
autofit (remainder) so pool numbers are reproducible; compute need =
|
||||
tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
|
||||
states — measure, don't assume 2x).
|
||||
- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
|
||||
deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
|
||||
one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
|
||||
- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
|
||||
`mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
|
||||
|
||||
## 7. Bash laws that ate windows (each cost an arm)
|
||||
|
||||
1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
|
||||
`SPEC=3: command not found`, nothing runs. Pass literal assignments:
|
||||
`SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
|
||||
2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
|
||||
an if/else it aborts ONLY that compound statement — the script CONTINUES,
|
||||
every arm tears down clean with zero measurements. Detection law: `ls`
|
||||
the per-arm `.measure.txt` files before trusting any summary; arm echoes
|
||||
are not evidence.
|
||||
3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
|
||||
it aborts AFTER the harness stopped production. Two statements.
|
||||
4. `sed -i` renames the file; already-running bash keeps the OLD inode —
|
||||
mid-flight patches miss arms launched in the preceding minutes. Patch
|
||||
early or run a make-up window.
|
||||
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
|
||||
(`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
|
||||
fails silently. Same trap in `git commit -m`: backticks become command
|
||||
substitution.
|
||||
6. No comments INSIDE a backslash-continued command (the continuation joins
|
||||
the comment); verify multi-line command construction with `bash -n` AND a
|
||||
render test.
|
||||
|
||||
## 8. Results location and recovery (frank-primary arms)
|
||||
|
||||
| copy | path |
|
||||
|---|---|
|
||||
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
|
||||
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
|
||||
| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
|
||||
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
|
||||
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
|
||||
|
||||
`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
|
||||
constants 99.9/88.8 identical across arms by construction; **identical
|
||||
numbers across arms == stub or bug, never a measurement.** After compaction,
|
||||
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
|
||||
logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
|
||||
window.
|
||||
|
||||
## Related
|
||||
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
|
||||
- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
|
||||
- [[LLM Performance Diagnosis]] — profiling and attribution methods
|
||||
- [[Remote Job Runner]] — detached execution of long sweeps
|
||||
- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Performance Diagnosis
|
||||
description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
|
||||
tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# LLM Performance Diagnosis
|
||||
|
||||
When a served endpoint is slower than wanted and someone names a suspect
|
||||
("PCIe is saturated", "the gather is the bound"), throughput arithmetic cannot
|
||||
settle it — it is usually circular (it assumes the suspect consumes the step).
|
||||
Get data in this order; the profiler window alone has retired multi-day
|
||||
optimization plans.
|
||||
|
||||
## 1. Is decode context-bound at all? (the flat-depth probe)
|
||||
|
||||
Sweep decode across context depth. **Flat** ⇒ attention over history is not
|
||||
the cost; the problem is per-token expert movement and KV/attention tuning is
|
||||
ruled out entirely. A genuinely context-bound server shows a clear downward
|
||||
slope. Record cold vs warm separately — a small gap means the cache is
|
||||
working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304
|
||||
to 16,095 tokens.
|
||||
|
||||
## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
|
||||
|
||||
Symptom: smooth ~25–35 % utilization with 100 % peaks on an expert-offload
|
||||
MoE. This is the EXPECTED signature, not a fault:
|
||||
|
||||
```
|
||||
per decode step: compute burst ~100 ms (100 %) → expert-gather wait ~400 ms (~0 %)
|
||||
resident fraction = arena slots / total experts (e.g. 6574/24576 ≈ 27 %)
|
||||
PCIe bytes/token = miss_rate × lookups × slab_bytes (0.73 × 480 × 2.344 MiB ≈ 0.84 GB)
|
||||
predicted duty ≈ compute_ms / (compute_ms + wait_ms)
|
||||
```
|
||||
|
||||
Validate the terms against the deployed engine's boot plan
|
||||
([[RDNA4 MoE Expert Offload VRAM Budget]]). If predicted ≈ observed, the low
|
||||
average is structural. Note small-transfer reality: bulk bandwidth (91 % of
|
||||
ceiling at 1 GiB) overstates what 2–3 MiB slabs get (~72 % plus ~20 %
|
||||
per-transfer latency). Before blaming the link, measure it
|
||||
([[PCIe Link and Atomics]]) and compare the engine's effective gather rate to
|
||||
the measured slab-size bandwidth — a 40–50 % gap is the software read path,
|
||||
not the wire. `rocm-smi --showuse` reads ~0 % during the whole multi-minute
|
||||
weight load: show VRAM (`--showmeminfo vram`) as the load-progress signal.
|
||||
|
||||
## 3. Kernel-time attribution (the profiler window)
|
||||
|
||||
Run BEFORE building any replacement kernel.
|
||||
|
||||
1. Stop + mask production; boot the shape under test with ONLY the profiler
|
||||
config added. vLLM argv:
|
||||
`--profiler-config '{"profiler":"torch","torch_profiler_dir":"/prof","torch_profiler_with_stack":false,"torch_profiler_record_shapes":false,"torch_profiler_with_memory":false,"torch_profiler_dump_cuda_time_total":false}'`
|
||||
— `with_stack=false` is load-bearing (with stacks the stop hangs and the
|
||||
trace buffer grows ~40 GB → kernel OOM-kill of the engine core, anon-rss
|
||||
~95 GB; the worker-side stop can hang 5 min on ROCm/kineto). Bind-mount
|
||||
`/prof`. If a shell substitutes values into that JSON, verify the escapes
|
||||
actually landed.
|
||||
2. Window it: warm request → `POST /start_profile` → N generations of ≥128
|
||||
tokens (~15 s of traffic) → `POST /stop_profile` (client may time out at
|
||||
60 s; the trace still lands). `telemetry_report` has ONE caller: the
|
||||
gpu_worker profiler-stop path. Overhead is small enough to keep decode
|
||||
comparable (9.6 vs 9.9–10.0 tok/s measured).
|
||||
3. **Parse ON THE SERVER** (traces are 100–150 MB gzipped, GB expanded).
|
||||
Rank `ph=="X"` events with `cat in {kernel, gpu_memcpy, gpu_memset}` by
|
||||
summed `dur` (µs): report % of device time, launch count AND µs/call
|
||||
(22 µs × 708k launches is a different problem than 4 ms × 4k), plus the
|
||||
first-to-last event span (fraction of wall the device was busy). Aggregate
|
||||
by FULL mangled name (template args are the tuning signal) and by stripped
|
||||
symbol. **Trap: `gpu_user_annotation` events masquerade as GPU time** (a
|
||||
naive sum reported 63 % in one `execute_context_0(...)` entry; spans
|
||||
overlap and can exceed the wall clock).
|
||||
4. Divide by the window's token count → ms/token; if kernel time ≈ step
|
||||
time, the table IS the optimization plan.
|
||||
|
||||
Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
|
||||
|
||||
```
|
||||
34.8 % 708,500 launches vectorized_elementwise_kernel <- the actual bound
|
||||
10.0 % 42,000 MoE W4A16 GEMM
|
||||
8.9 % 83,160 router/top-k (sort-based)
|
||||
6.5 % 133,920 hyper-connection pre/post kernels
|
||||
1.05 % 20,664 <the long-suspected expert gather>
|
||||
```
|
||||
|
||||
No amount of link arithmetic would have produced those two conclusions: the
|
||||
suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
|
||||
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
|
||||
predicted a negative result in advance (spec decode cannot help a shape whose
|
||||
per-token cost scales with the token count) which matched an unexplained flat
|
||||
MTP measurement.
|
||||
|
||||
## 4. Telemetry without benchmarking (live hosts)
|
||||
|
||||
When benching is forbidden (production serving), mine the runtime's own
|
||||
per-request telemetry — it measures the real workload and costs nothing.
|
||||
|
||||
- llama.cpp (root unit, `-lv 3`): `sudo journalctl -u <unit> --since "-3h"
|
||||
--no-pager` → `slot print_timing` lines (prompt eval, eval, total, `graphs
|
||||
reused`, `draft acceptance … mean len`, `n_gen = N, tg = X t/s, tg_3s = …`).
|
||||
- halogen (rootless quadlet): `journalctl --user -u <unit>` (**never sudo** —
|
||||
per-user storage) → `serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16
|
||||
rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok`.
|
||||
- Read the envelope, never one sample: short-prompt `prompt eval` is
|
||||
overhead-dominated (57 tok @ 168 t/s) — quote the longest available (1106
|
||||
tok @ 1013 t/s); decode and acceptance are content-dependent (0.46–0.76
|
||||
measured on one unchanged build); `graphs reused > 0` / `selected slot by
|
||||
LCP similarity` prove prefix-cache reuse.
|
||||
- **Attribute PCIe without touching the model:** background `nvidia-smi dmon
|
||||
-s t -c 16 -o T`, ONE ordinary `max_tokens=200` request, `wait` at the end
|
||||
(without it ssh closes and the client dies — the journal shows `W srv stop:
|
||||
cancel task` and NO print_timing while curl still reports plausible
|
||||
status). Measured: peak ~230 MB/s against a ~7,900 MB/s link = ~3 %, 0 at
|
||||
idle — the sampled-logits stream, not the transport, is decode traffic on
|
||||
that shape.
|
||||
- One `max_tokens=200` request is production traffic and fine; anything that
|
||||
pins the card is a bench → stop-and-mask first. Check WHICH instance you
|
||||
are reading (`systemctl status | head -12`; task ids jump across
|
||||
restarts). The engine's own ledger (`KV self size`, `n_slots`,
|
||||
`n_ctx_slot`) outranks your model of it.
|
||||
|
||||
## 5. The fork's data path (before blaming the link)
|
||||
|
||||
Offload engines pick between DMA copy-in and read-through UVA dereference
|
||||
inside the GEMM. Grep the installed package for the LRU/offload module and
|
||||
read the BRANCH CONDITION, not just the env names. A token-count threshold
|
||||
decides the path, and it interacts with spec decoding: an MTP step is k+1
|
||||
tokens, so the step can silently cross the threshold onto the slow path as
|
||||
draft depth grows. Check the threshold's premise: a default tuned at
|
||||
12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
|
||||
several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
|
||||
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
|
||||
check for the skip line before concluding autotuning ran. Gather-geometry
|
||||
knobs read at import time → each arm is a full reload.
|
||||
|
||||
## 6. RAM-bandwidth-bound MoE ceiling (before recommending a quant)
|
||||
|
||||
```
|
||||
routed experts/token = top_k × n_layers
|
||||
bytes/token = routed_params × bytes_per_param + shared_expert_bytes
|
||||
ceiling (tok/s) = measured_bandwidth / bytes_per_token
|
||||
```
|
||||
|
||||
Measure the HOST's real bandwidth, never assume DIMM ratings: 4-DIMM AM5
|
||||
boards downclock to DDR5-3600 regardless of 5600 sticks (two large DIMMs beat
|
||||
four small). Single-threaded reads under-report dual-channel ~4x — use the
|
||||
multi-threaded plateau of a checksumming reader (`scripts` pattern:
|
||||
`/tmp/bw.c`, 1/2/4/8/16 threads). Worked example (DeepSeek-V4.1-Flash):
|
||||
6×40 experts × 35.4M params = 8.5B params/token; at 3 bpw + 0.53 GB shared =
|
||||
**3.7 GB/token** → 42 GB/s ⇒ **11.3 tok/s ceiling**. Sanity anchors: H100
|
||||
NVL + CPU offload 226 GB ⇒ 5.8–9.0 tok/s; 2×3090 + 768 GB DDR4 ⇒ 12.2.
|
||||
CPU compute rarely binds below 4 bpw (~5.3 flops/byte ⇒ ~223 GFLOPS at 42
|
||||
GB/s); PCIe expert STREAMING is worse than CPU compute (Gen5 x8 ~22.7 GB/s
|
||||
effective < half of RAM); prefill reads the union of a batch's experts
|
||||
(prefill ≈ bandwidth / total_expert_bytes × ubatch). A quant that fits only
|
||||
by ignoring the Engram table does not fit (Engram = sparse n-gram lookup,
|
||||
NVMe-tolerant; the routed experts are not). An iGPU GTT aperture is an ALIAS
|
||||
of the same RAM, not extra memory.
|
||||
|
||||
## 7. IBS model-access attribution (AMD CPUs)
|
||||
|
||||
Real CPU access patterns for mmap'd weights, live, no stop (established on
|
||||
frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
|
||||
`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
|
||||
analyze; `--analyze-only`).
|
||||
|
||||
1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
|
||||
from CONSECUTIVE offsets, never a type-size table** — a wrong per-element
|
||||
size silently under-covers every tensor and dumps tail samples into a junk
|
||||
bucket (first run lost 59 % of expert samples; a uniform gap/exps ratio
|
||||
~1.4 across parts is the tell). Data start = `(tell + 31) & ~31`.
|
||||
2. `/proc/<pid>/maps` VMAs (7-group regex; pgoff = group 4, path = group 7)
|
||||
matched against part files; multi-VMA per part is normal.
|
||||
3. `sudo perf record -e ibs_op// -c 50000 --data -o out.perf -p <pid> --
|
||||
sleep <dur>` WHILE a mixed workload runs (idle perf numbers mean nothing).
|
||||
`sudo perf script -i out.perf -F addr` for addresses.
|
||||
4. Attribute: bisect VMAs, `foff = pgoff + (addr − vma_start)`, bisect tensor
|
||||
starts; per-expert index = `int(lo / (span / N_EXPERTS))` (expert-major
|
||||
rows).
|
||||
5. Verify before believing histograms: gap ≈ 0, every tensor-bearing part
|
||||
nonzero, and MoE blocks should be FLAT by construction (top-k fires every
|
||||
block every token) — a non-flat block histogram means mis-attribution
|
||||
first. Established law: expert routing is per-block flat, per-EXPERT
|
||||
skewed.
|
||||
|
||||
## 8. Prefix-cache-aware routing (multi-replica serving)
|
||||
|
||||
Measure with a **sequential-turn client**, not only a batch benchmark — a
|
||||
batch of distinct prompts parallelises prefill across replicas and hides the
|
||||
problem. The two failure modes: round-robin loses every other cache hit
|
||||
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
|
||||
pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
|
||||
pinned). **The rule that gets both: route by request shape** — a request
|
||||
carrying more than one message is a continuation (agent loop re-sending its
|
||||
context) → pin to the replica that served that prefix; anything else → spread
|
||||
with a COUNTER, not a prefix hash. Add failover: demote a replica on
|
||||
connection failure and retry on a live one BEFORE writing anything to the
|
||||
client (safe pre-stream; never retry after streaming started).
|
||||
|
||||
The wall no policy fixes: **a running sequence and its cached prefix draw on
|
||||
the SAME pool.** Three concurrent 50–80k families against 122,636 tokens/card
|
||||
thrash under every placement and admission policy tried (counter placement,
|
||||
load-balanced placement, one-large-request-per-card admission, both chunk
|
||||
directions — the last COSTS KV in both directions). When the working set
|
||||
exceeds aggregate KV, say so: the levers are capacity, fewer concurrent
|
||||
large contexts, or a different platform. Traps: a lifetime hit-rate counter
|
||||
is not a per-request verdict (probe directly: same prompt twice to an idle
|
||||
replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
|
||||
`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
|
||||
against the live endpoint before dashboards. Test shape: drive the routing
|
||||
function directly with synthetic bodies (stdlib-only policy module);
|
||||
conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
|
||||
hashes differently per turn and pinning silently never happens.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
|
||||
- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
|
||||
- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
|
||||
@@ -0,0 +1,281 @@
|
||||
---
|
||||
type: reference
|
||||
title: Model Swap and Rollback
|
||||
description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
|
||||
tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Model Swap and Rollback
|
||||
|
||||
Applies to every endpoint swap on the fleet (frank-small / frank-large /
|
||||
frank-primary). Repo law: all config flows through git; the live models are
|
||||
host state under `/home/dank/models/<name>/`.
|
||||
|
||||
## 1. Fit preflight — never deploy a swap without confirming it FITS
|
||||
|
||||
1. Get the checkpoint's on-disk/weight size AND precision from the HF card
|
||||
(`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params:
|
||||
BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
|
||||
2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that
|
||||
stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload,
|
||||
layer splits). Budget weights + KV pool + OS + page cache; **if weights
|
||||
alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink
|
||||
weights.
|
||||
3. **The failure mode is a self-sustaining thrash, not a crash:** a large
|
||||
swap makes the kernel PAGE instead of OOM-killing → sshd/user space
|
||||
starved indefinitely, no recovery window opens by itself; systemd
|
||||
`Restart=always` turns it into a crash-reload-rethrash loop; `enabled` +
|
||||
`WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle
|
||||
does NOT fix it.
|
||||
|
||||
Recovery (may need console / SysRq-to-GRUB):
|
||||
|
||||
```bash
|
||||
sudo systemctl mask --now <svc> # break the reboot trap FIRST
|
||||
cd <checkout> && git fetch origin && git reset --hard origin/main
|
||||
grep MODEL_DIR <env> # confirm the fixed target
|
||||
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
|
||||
curl -s <health> # then a REAL completion
|
||||
```
|
||||
|
||||
Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
|
||||
confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
|
||||
FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
|
||||
quant by size fit, then accept the compute path the card supports. Before
|
||||
choosing a smaller quant, state WHICH of the two you are buying — headroom,
|
||||
or a design change (a smaller quant does not change what the runtime can
|
||||
express; per-tensor cross-device placement stays refused under
|
||||
`--override-tensor`, and a layer split moves only contiguous layer ranges).
|
||||
|
||||
## 2. Ground truth first
|
||||
|
||||
- Read the HF model card (runtime pin, hard constraints, recommended args)
|
||||
+ the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
|
||||
`scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
|
||||
- Check the host before downloading anything: `ls /home/dank/models/`,
|
||||
`df -h /home`, `du -sh` — bench-era or prior-deployment files often already
|
||||
exist.
|
||||
|
||||
## 3. Fetch and verify (HF payloads)
|
||||
|
||||
**Inventory by QUANT NAME, never by extension** — large quants ship as
|
||||
`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
|
||||
`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
|
||||
mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
|
||||
repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
|
||||
return 403 even with a valid token until the account accepts the gate —
|
||||
prefer the ungated copy.
|
||||
|
||||
```bash
|
||||
# token out of band: mode-600 curl config; a token passed as -H is world-readable
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
|
||||
chmod 600 /tmp/hf.curlrc
|
||||
curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"
|
||||
```
|
||||
|
||||
`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes
|
||||
the error body as the artifact (identical 153-byte shards, cheerful DONE,
|
||||
zero bytes of model). Never `pgrep -a` a running fetch (token is in
|
||||
`/proc/<pid>/cmdline`); watch progress by size:
|
||||
`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`.
|
||||
|
||||
**Verify, then assemble:** per part, byte count == Hub's declared size AND
|
||||
`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total ==
|
||||
sum of parts (compute the sum with python, not by hand) and `head -c 4` ==
|
||||
`GGUF`. The assembled file has no published hash — the per-part oids + exact
|
||||
total are the chain. Keep the fragments when disk allows (only copies
|
||||
re-verifiable against the Hub); failure artifacts and the replaced payload
|
||||
go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants
|
||||
and the local file may be the last copy.
|
||||
|
||||
Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already
|
||||
do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to
|
||||
skip all three. HF cache snapshots SYMLINK every file into `blobs/` and
|
||||
those links dangle inside containers — serve scripts and the deployer refuse
|
||||
symlinks and missing tokenizer files for exactly this reason.
|
||||
|
||||
## 4. Runtime build (when the card pins a fork/build)
|
||||
|
||||
- Clone + build with the card's exact cmake line, plus
|
||||
`-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui
|
||||
bucket) aborts llama.cpp link at 98 % on these hosts; the API server is
|
||||
unaffected.
|
||||
- Verify every flag the serve script passes exists in the new binary BEFORE
|
||||
touching the service: `llama-server --help | grep -q -- "<flag>"` for each;
|
||||
list MISSes. Read the GGUF's tensor histogram first if the payload uses a
|
||||
custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]).
|
||||
|
||||
## 5. Config via git only
|
||||
|
||||
Edit `scripts/<svc>.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve
|
||||
script header/defaults locally; `bash -n`; `./tests/run.sh` must pass;
|
||||
commit; push; `git pull --ff-only` on the target. Never edit configs on the
|
||||
remote. Renaming a model dir (`mv`) is host state and allowed. The env-file
|
||||
hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged
|
||||
arms possible — keep it intact.
|
||||
|
||||
## 6. Live validation
|
||||
|
||||
- No scratch ports — one GPU cannot host two big models. Ensure the old unit
|
||||
is truly gone first: `systemctl is-active` can hang in `stop-sigterm`
|
||||
(llama.cpp never preempts an in-flight generation); poll until the PID
|
||||
exits.
|
||||
- Manual serve for smoke: `setsid nohup bash scripts/serve-<svc>.sh >
|
||||
/tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session.
|
||||
- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows
|
||||
(`reasoning_content` non-empty); if the endpoint takes images, a
|
||||
solid-color probe through an `image_url` content part (legacy top-level
|
||||
`images` is silently ignored; probe with ≥256 px).
|
||||
- vLLM specifics: first request after boot can take ~18 s (deferred Triton
|
||||
loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
|
||||
the contract ([[vLLM Boot Forensics and Config Interrogation]]).
|
||||
|
||||
## 7. systemd handoff — THE TRAP
|
||||
|
||||
`./scripts/deploy.sh <host> --service <svc>` gates health, but a leftover
|
||||
manual server holding the port makes the health check pass while the UNIT
|
||||
crash-loops on bind. Always confirm ownership:
|
||||
`systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server`
|
||||
(or the container runtime PID); only then trust "healthy". If a real
|
||||
long-context task is queued, do the handoff anyway — client aborts are
|
||||
recoverable, drift is not.
|
||||
|
||||
## 8. Close-out (mandatory)
|
||||
|
||||
1. Docs: active runbook updated (build recipe, profile, validation numbers
|
||||
incl. real-workload prefill/decode); superseded runbook gets a RETIRED
|
||||
banner and stays as the rollback record; `LLM/index.md` mermaid + active
|
||||
list; AGENTS.md fleet table + runtime section + history chain; work-log
|
||||
entry in `serverAppConfigs-docs/log.md`.
|
||||
2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
|
||||
context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
|
||||
the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
|
||||
`max-model-len` and its concurrency line. Overselling causes silent
|
||||
client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
|
||||
engine's cap, `input` capabilities (text/image), and
|
||||
`reasoningContentField` per runtime (llama.cpp/SGLang emit
|
||||
`reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry
|
||||
that named the old endpoint.
|
||||
3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary +
|
||||
payload kept on disk, rollback numbers recorded in the runbook.
|
||||
|
||||
## 9. Revert-pin an ad hoc experiment (the three pins)
|
||||
|
||||
A campaign is "done" only when all three exist — without them a
|
||||
`podman pull`, an image update, or a lost scratch script destroys the way
|
||||
back:
|
||||
|
||||
1. **Git tag (annotated, pushed):**
|
||||
`git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline +
|
||||
runbook path + revert command>"` — a future agent must reconstruct the
|
||||
campaign from the tag message alone.
|
||||
2. **Image digest in the serve script:** floating tags move. Capture
|
||||
`podman inspect <ctr> --format '{{.ImageName}} {{.Image}}'` + `podman
|
||||
images --digests`, patch `IMAGE=` to `repo@sha256:<repodigest>` with a
|
||||
comment (image ID, date, old ref), verify `podman image exists
|
||||
repo@sha256:<digest>` and `bash -n`. Digest refs survive upstream pushes;
|
||||
`podman pull repo@sha256:<digest>` re-fetches exact bytes. One host can
|
||||
hold MULTIPLE images of one repo — pin by the digest of the RUNNING
|
||||
container, never by tag name. Never edit the pinned line back to a
|
||||
floating tag; new experiments branch from the digest.
|
||||
3. **Local alias:** `podman tag <id> <exp>-runtime:tuned-YYYYMMDD`.
|
||||
|
||||
Record scratch-script paths + full flag surface in the runbook (hook env
|
||||
vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`.
|
||||
Pure measurement sessions that changed no state need only pin 1.
|
||||
|
||||
## 10. Research verification discipline (for "can model X run here?")
|
||||
|
||||
- Scouted research summaries have shipped fabricated citations on this fleet
|
||||
(invented HF repos, plausible-but-wrong PR numbers, stale engine versions).
|
||||
NEVER copy a scout's citations into a deliverable — re-verify each
|
||||
load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it
|
||||
enters anything.
|
||||
- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim
|
||||
about our own hosts.
|
||||
- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active,
|
||||
CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early
|
||||
GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path
|
||||
(heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K).
|
||||
GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the
|
||||
default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY,
|
||||
flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in
|
||||
unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant
|
||||
ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
|
||||
MMLU-Pro vs Q8_0).
|
||||
- GGUF parsing gotcha: expert tensors are
|
||||
`blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
|
||||
a `len(parts) == 3` filter matches zero layers with no parse error. Filter
|
||||
on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
|
||||
|
||||
## 11. Adding a second LLM endpoint to a host
|
||||
|
||||
When a host must serve two models at once (an iGPU runtime plus a second
|
||||
card), or one endpoint is moving and a new one takes the freed hardware.
|
||||
|
||||
**0. Decide the shape before touching hardware.** One provider per
|
||||
HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
|
||||
— a second endpoint is a new provider block, not a second model in the
|
||||
existing one. Read the budget as an inequality: for unified memory the
|
||||
engine's own fit check is the arbiter (`budget = host − pinned weights −
|
||||
reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV
|
||||
by counting only full-attention layers — and remember `--parallel N`
|
||||
DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free
|
||||
RAM by changing the POOL or the ARENA (the reserve is a fit-check term,
|
||||
never an allocation) and drop optional charges (a vision tower is ~2 GiB)
|
||||
rather than hoping.
|
||||
|
||||
**1. Repo first, and retire in the same commit.** New service files
|
||||
(`scripts/<svc>.env`, `.service`, `serve-<svc>.sh` with hard guards for
|
||||
missing artifacts); planner entries (`SERVICE_ORDER` + the target's service
|
||||
set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the
|
||||
dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a
|
||||
suffix — and check what pointed at them: bench harnesses, a chat template
|
||||
only the old runtime read, and `scripts/test-changed.sh` mappings are the
|
||||
usual victims (a mapping to a retired test file fails the pre-commit
|
||||
wrapper silently for weeks). One bats file per service asserting
|
||||
INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
|
||||
not constants; a negative grep must strip comments first
|
||||
(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
|
||||
the very thing you removed.
|
||||
|
||||
**2. The port-bind race (rootless podman).** pasta binds the published port
|
||||
at container CREATE; a just-stopped predecessor still holds it. Order the
|
||||
handler: retire the old manager's unit → stop THIS service's own unit
|
||||
(no-op on first cutover) → wait for every socket on the port to clear
|
||||
([[Container Host Reachability]]) → install → start. Without the second
|
||||
step a redeploy deadlocks against its own gate.
|
||||
|
||||
**3. Wire the surfaces, refuse impossible signals.** Point the harness
|
||||
small/fast + image roles at the new endpoint; move the vision role WITH the
|
||||
vision tower (one owner per capability). `shared/homepage/services.json`: a
|
||||
card per endpoint, probed from the monitor's own host. Prometheus: only add
|
||||
a job if the runtime actually exposes Prometheus text (halogen publishes
|
||||
`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
|
||||
reason written down rather than left as a permanent red).
|
||||
|
||||
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
|
||||
<large> --service <small>` (order matters); per endpoint: `/health`,
|
||||
`/v1/models` reports the alias, a chat completion returns content AND
|
||||
`reasoning_content`, a `tools` request returns `tool_calls`, an image
|
||||
request is answered (or refused with a named reason), the runtime's own
|
||||
speculation counters move, and the unit is `enabled`/`generated` with linger
|
||||
on. Read the engine's boot ledger for a silent clamp. Finally spend one
|
||||
real request through the harness per changed role — routing labels lie; the
|
||||
servers' own journals do not.
|
||||
|
||||
Traps that look like model faults: `500 tools param requires --jinja flag`
|
||||
(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected
|
||||
reasoning effort <level>` (fold via `compat.reasoningEffortMap`); the same
|
||||
request works by hand but a subagent dies (subagents run at a different
|
||||
thinking level — check both); the container dies instantly although the
|
||||
predecessor "stopped" (the port race above).
|
||||
|
||||
## Related
|
||||
|
||||
- [[GGUF Compatibility]] — which runtime can load a payload at all
|
||||
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
|
||||
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint
|
||||
@@ -0,0 +1,264 @@
|
||||
---
|
||||
type: reference
|
||||
title: Tcclaviger Fork Ops
|
||||
description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
|
||||
tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Tcclaviger Fork Ops
|
||||
|
||||
The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
|
||||
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
|
||||
first-class expert-offload planner) served `frank-primary` at shape G until
|
||||
the 2026-10-05 FreeToken cutover; it remains the rollback target
|
||||
(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM
|
||||
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
|
||||
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
|
||||
[[LLM Bench Windows and Measurement]].
|
||||
|
||||
## The ad-hoc sweep harness (dankserver)
|
||||
|
||||
Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
|
||||
depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
|
||||
`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
|
||||
(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
|
||||
JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
|
||||
into every fresh container — `podman cp` files die on container recreate.
|
||||
|
||||
- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
|
||||
`mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
|
||||
ttft)`; chunk counting is INVALID under spec decode.
|
||||
- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
|
||||
`out NAME` ([[Remote Job Runner]]).
|
||||
- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
|
||||
(else repetition-halt or early EOS), finish_reason, and inspect the text —
|
||||
`bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
|
||||
loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
|
||||
Acceptance swings 48–61% run-to-run — noise; decode is the metric.
|
||||
- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
|
||||
- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
|
||||
`Application startup complete` — it is on STDERR; an unmerged grep can
|
||||
never match and burns the whole wait timeout.
|
||||
- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
|
||||
are CPU-side); VRAM climbing is the load indicator.
|
||||
- Config edit discipline: the serve script is patched by scp'd python
|
||||
scripts with asserts (anchor must exist, addition must not); `bash -n`
|
||||
after every patch.
|
||||
|
||||
## Measured knob verdicts (2026-09-29 campaign)
|
||||
|
||||
Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
|
||||
TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
|
||||
|
||||
| Knob | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
|
||||
| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
|
||||
| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
|
||||
| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
|
||||
|
||||
Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
|
||||
batch-shaped compute knobs move ~nothing.
|
||||
|
||||
The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
|
||||
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
|
||||
the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
|
||||
depth.
|
||||
|
||||
Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
|
||||
flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
|
||||
profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
|
||||
kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
|
||||
(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
|
||||
fires ONLY from the worker's profiler-stop path; worker-side stop can hang
|
||||
>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
|
||||
measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
|
||||
temporal reuse dominates, so never infer miss rates from resident-slot
|
||||
fractions.
|
||||
|
||||
## Sweep hygiene (fork-specific preflight)
|
||||
|
||||
1. **Knob landed:** for each `-e` env knob, confirm in-container
|
||||
`podman exec … env` (or the engine log's confirmed-values line). A knob
|
||||
the fork silently ignores produces a "parity" arm that is really the
|
||||
default — the v1 gather sweep lost all three arms to exactly this
|
||||
(GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
|
||||
2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
|
||||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms
|
||||
with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
|
||||
(no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
|
||||
verdict.
|
||||
3. **Spec arms must prove engagement:** engine log
|
||||
`SpecDecoding metrics: Mean acceptance length …` and
|
||||
`curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
|
||||
4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
|
||||
refused boot dies in one poll instead of 20 min of health waits. Capture
|
||||
>8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
|
||||
refusal) lives higher up.
|
||||
5. Report per-depth tables, never one number: g32×64's win grows with depth
|
||||
(54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
|
||||
rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
|
||||
rows across arms only.
|
||||
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
|
||||
noise).
|
||||
|
||||
Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
|
||||
arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
|
||||
(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
|
||||
measure-stage bug costs decode tables but not boot-log findings.
|
||||
|
||||
**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
|
||||
is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
|
||||
4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
|
||||
0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
|
||||
4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
|
||||
OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
|
||||
tokens). Acceptance is real: mean length 2.4–2.8, per-position
|
||||
0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
|
||||
exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
|
||||
|
||||
## Serving a model larger than VRAM (dankserver recipe)
|
||||
|
||||
Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB
|
||||
RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives
|
||||
a bogus 186.7 GiB (the Raphael iGPU is in the list — see
|
||||
[[Llama Cpp Serving Sizing]] for the reading discipline).
|
||||
|
||||
**Kernel gate — this decides which checkpoint can work at all.**
|
||||
`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*`
|
||||
but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with
|
||||
BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup
|
||||
otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`,
|
||||
`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no
|
||||
kernel (the class exists, the r4d.so entry does not); RFA has neither. Check
|
||||
the checkpoint's quant against this list BEFORE downloading anything.
|
||||
|
||||
Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve
|
||||
"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`.
|
||||
Known-good invocation:
|
||||
|
||||
```bash
|
||||
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
|
||||
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
|
||||
--cap-add SYS_PTRACE \
|
||||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||||
-v /home/dank/models/<model>:/models:ro \
|
||||
docker.io/tcclaviger/vllm:latest /models \
|
||||
--served-model-name <name> --port 7770 \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 32768 \
|
||||
--kv-cache-memory-bytes 2147483648 \
|
||||
--max-num-seqs 4 \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--enable-prefix-caching --trust-remote-code
|
||||
```
|
||||
|
||||
`-p 7770:7770` is not included, so the API is only reachable via
|
||||
`podman exec` — add `-p` if host-side curl is needed. Load ~5 min;
|
||||
`/health` answers connection-refused the whole time; watch for
|
||||
`Application startup complete` in `podman logs <ctr>`. Verified working:
|
||||
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131
|
||||
shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10,
|
||||
multimodal, native 262,144).
|
||||
|
||||
Traps that each cost a boot:
|
||||
|
||||
1. **Mount the whole `/dev/dri`, never individual render nodes.** Render
|
||||
nodes without the primary `card*` nodes → HIP enumerates two agents and
|
||||
refuses a context on either (`RuntimeError: No CUDA GPUs are available`)
|
||||
while `device_count()` cheerfully returns 2. Confirmed by direct A/B.
|
||||
2. **memlock must be raised** (done on Dankserver via
|
||||
`/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any
|
||||
large pinned allocation fail as a misleading
|
||||
`torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`;
|
||||
it cannot be raised from inside a container or root's own shell — only
|
||||
PAM applies it at login. Matters because `--expert-host-mlock` defaults
|
||||
True.
|
||||
3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** —
|
||||
the planner sizes the arena against the budget and leaves KV the
|
||||
remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed
|
||||
at 0.64 GiB.
|
||||
4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the
|
||||
budget subtracts the KV reserve first, so a smaller context shrinks the
|
||||
reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV;
|
||||
32768 → 0.34 GiB).
|
||||
5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token
|
||||
planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost).
|
||||
Fix by pinning the pool: `--kv-cache-memory-bytes`.
|
||||
6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
|
||||
structurally orthogonal to `--expert-precise-memory` (verified
|
||||
in-container 2026-10-05: precise-memory only swaps the GPU overhead term,
|
||||
expert_offload_mem only sets the host bound; they rejoin only inside
|
||||
`place_experts`, and the plan rewrites `expert_offload_mem` back to the
|
||||
cap the plan met, so plan and allocation cannot drift).
|
||||
7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one.
|
||||
8. `podman run -d` without `-a` sends no stdout to a redirect — the log file
|
||||
only receives the container ID; use `podman logs`.
|
||||
|
||||
Status reads: use `scripts/dankstatus.sh`, never hand-rolled
|
||||
`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a
|
||||
TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and
|
||||
awk prints `0.0` — a false "host out of memory" that invites an unnecessary
|
||||
abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv
|
||||
(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count
|
||||
to the Hub's declared size; `-C -` resumes truncated shards.
|
||||
|
||||
**Check whether the stack supports the model natively before porting an
|
||||
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
|
||||
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
|
||||
through two image tags to bring an overlay up was entirely wasted work.
|
||||
|
||||
## Compile-cache persistence (survives `podman rm -f`)
|
||||
|
||||
Serve scripts that recreate the container discard the writable layer, so
|
||||
every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s`
|
||||
cold, of which 173 s was one graph).
|
||||
|
||||
1. **Prove it and locate it:** fresh container vs live
|
||||
(`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
|
||||
`podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
|
||||
f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
|
||||
`/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
|
||||
`/root/.cache/comgr`.
|
||||
2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
|
||||
both trees, safe whenever the image ships no such directory (step 1
|
||||
proves it; otherwise you would be hiding baked state). Wire as an opt-in
|
||||
tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
|
||||
`${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
|
||||
checkout.
|
||||
3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
|
||||
N s`, warm `Directly load the compiled graph(s) for compile range (…)
|
||||
from the cache` with `torch.compile took` in single-digit seconds.
|
||||
4. Traps: the cache key includes the compile RANGE — an MNBT change costs
|
||||
ONE recompile, not a permanent tax; do not claim a proportional boot cut
|
||||
(time to `/health` is dominated by weight loading: 490 s warm vs 188 s
|
||||
cold compile); stale entries are misses, not wrong answers (version +
|
||||
config hashed); if the serve script is seam-gated, assert the mkdir and
|
||||
the exact mount target.
|
||||
|
||||
## Plan-line verification checklist (after any fork deploy)
|
||||
|
||||
```
|
||||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
|
||||
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
|
||||
```
|
||||
|
||||
`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate
|
||||
these columns to `17.`. Verify: new budget/graphs/arena in the plan;
|
||||
`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
|
||||
memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
|
||||
exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
|
||||
real margin; `systemctl show frank-primary -p NRestarts --value` = 0;
|
||||
`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
|
||||
completion from a PEER — `curl localhost:7770/health` inside an ssh session
|
||||
on the host can return `000` and looks like an outage that isn't.
|
||||
|
||||
## Related
|
||||
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
|
||||
- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
|
||||
- [[Model Swap and Rollback]] — where the fork sits in the swap history
|
||||
- [[Remote Job Runner]] — the job.sh harness referenced above
|
||||
@@ -123,3 +123,27 @@ Stop and disable the root unit, trash it, restore the halogen quadlet from
|
||||
`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
|
||||
`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
|
||||
the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
|
||||
|
||||
## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
|
||||
|
||||
Why this deployment could not be tuned into a production endpoint: decode
|
||||
3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
|
||||
1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
|
||||
The decode root cause is structural: ~100 CPU↔GPU backend splits per token
|
||||
with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
|
||||
dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
|
||||
`pause` (ggml_barrier spin): threads wait on the serial critical path.
|
||||
Memory placement (mmap/anon/THP/deferred) does not matter — all variants
|
||||
landed 3.8–4.6.
|
||||
|
||||
- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
|
||||
per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
|
||||
gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
|
||||
stay dead.
|
||||
- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
|
||||
4× slower than ik's iqk) — do not switch engines for speed.
|
||||
- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
|
||||
NextN head is stripped (no `blk.48` tensors).
|
||||
- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
|
||||
decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
|
||||
an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.
|
||||
|
||||
@@ -134,6 +134,59 @@ slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement
|
||||
0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on
|
||||
both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.
|
||||
|
||||
## The split: re-measure it at the deployed shape
|
||||
|
||||
The batch and the tensor split are **coupled**: compute buffers are not
|
||||
symmetric across the two cards, so any batch change moves the balance point.
|
||||
Order of work — sweep batch with llama-bench (above), THEN re-measure the
|
||||
split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading
|
||||
the real service with a per-config env override and reading `rocm-smi`. A
|
||||
llama-bench number cannot decide the split: it has no `--parallel`, sizes
|
||||
its own context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
|
||||
|
||||
The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND
|
||||
the port to be free (`ss -ltn | grep :7770` until gone) before the next
|
||||
load, or the transient server dies with `couldn't bind HTTP server socket`.
|
||||
Free bytes follow `card = weights_share + KV_share + constant + ubatch_term`,
|
||||
where the ubatch term costs the drafter-carrying card roughly **1.85×** the
|
||||
other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by
|
||||
measurement only: one `-ts` step is one layer, worth ~0.43 GiB on a DeltaNet
|
||||
boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is
|
||||
about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91;
|
||||
36,29 @512 → 6.01/7.32; **35,30 @512 → 6.44/6.90 (deployed)**.
|
||||
|
||||
Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` at the new
|
||||
deployed point and re-anchor the per-card constants; check neighbouring
|
||||
splits still FAIL the balance gate or the seam has stopped discriminating
|
||||
(keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy,
|
||||
then verify live: flags in `/proc/<pid>/cmdline`, `/slots`, VRAM matching
|
||||
the pre-deploy measurement, a tool call, two concurrent requests,
|
||||
`draft acceptance` lines on BOTH slots, 0 ROCm errors.
|
||||
|
||||
**Per-card utilization (who is hot, and the split lever)** — a different
|
||||
question from batch/prefill. Measure decode tok/s AND per-card
|
||||
`rocm-smi --showuse` at 2–3 neighbouring splits in ONE session. Pitfalls
|
||||
(each cost a re-run): run the transient unit as ROOT (a `pkill -9
|
||||
llama-server` as dank auto-restarts the service and `systemctl` as dank
|
||||
fails "Access denied"); warm up 3 throwaway requests before the timed one
|
||||
(first 384-token decode after load measured 1.77 tok/s against a true ~32);
|
||||
`rocm-smi --showmeminfo vram --csv` columns are `$2`=Total, `$3`=**Used** —
|
||||
parsing the wrong column spins the settle gate the full 600 s; the util% is
|
||||
sampled post-hoc (which card finished last), so treat it as "which card is
|
||||
the pipeline tail", not a duty cycle.
|
||||
|
||||
**What the split does and does not buy (measured 2026-09-17, 393216 /
|
||||
MTP n=3):** the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29
|
||||
all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at
|
||||
35:30 because the MTP drafter + output layers + heavier compute buffers are
|
||||
pinned to the card holding the LAST layers — not because PCIe is the
|
||||
bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch).
|
||||
36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput
|
||||
cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing
|
||||
preference is that the card with the FASTEST PCIe lane should be the more
|
||||
utilized one; deploying it requires re-fitting the seam's slack/floor and
|
||||
re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **`llama-bench` cannot run the drafter, the slots or the deployed context.**
|
||||
|
||||
@@ -45,6 +45,98 @@ timestamp: 2026-09-30
|
||||
> because the fork compiles its own Qwen trunk and not its GLM one. See
|
||||
> [The measured profile](#the-measured-profile-and-the-real-wall).
|
||||
|
||||
## DFlash2 target-side port — the four laws and the KV ladder
|
||||
|
||||
Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder`
|
||||
/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot.
|
||||
|
||||
**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`):
|
||||
|
||||
1. **Patch the instance the runner calls.** `model_runner.py:425` calls
|
||||
`set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model —
|
||||
GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper;
|
||||
`self.language_model = Glm5NextForCausalLM`). Members placed only on the
|
||||
CausalLM fail the check on the conditional — same error text, wrong file.
|
||||
Patch BOTH wrappers.
|
||||
2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is
|
||||
`@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__`
|
||||
(including base-protocol members) with `getattr_static`, breaking on the
|
||||
first miss. Both wrappers need, as class attributes:
|
||||
`supports_eagle3: ClassVar[Literal[True]] = True`,
|
||||
`has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`.
|
||||
Verify in-image with stock typing — no boot needed.
|
||||
3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five
|
||||
Protocol-derived bases; adding `SupportsEagle3` anywhere dies with
|
||||
"Cannot create a consistent method resolution order" (sim-proven: the
|
||||
ordering rejects it in every base position). The attribute route avoids
|
||||
the MRO, so the two methods must be written by hand: the CausalLM
|
||||
unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional
|
||||
delegates to `self.language_model` (pixtral.py:498 pattern);
|
||||
`get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`,
|
||||
with the drafter config's `target_layer_ids` winning when present.
|
||||
4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call
|
||||
— GLM's loop carries the single-stream state entering the layer.
|
||||
|
||||
**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`):
|
||||
|
||||
1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
|
||||
2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads
|
||||
whole token rows) but CAN grow block_size when the max page is an exact
|
||||
multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to
|
||||
the `lcm` of all MLA-family pages first (patch 02).
|
||||
3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's
|
||||
divisible branch grew KpoolTailSpec's block; its kernel pins block ==
|
||||
index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded`
|
||||
instead of growing (patch 03).
|
||||
4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why
|
||||
GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs
|
||||
exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend
|
||||
`_get_kv_cache_groups_glm5_next` to partition
|
||||
HiddenStateCacheSpec + drafter attention specs; the drafter forms its
|
||||
own uniform group (per-group pages, NO unification → the MLA block
|
||||
stays at the engine-auto-set 2,432 from interface.py:918, kpool guard
|
||||
`block_size % 512 == 0` unstressed); layout returns a 10-tuple
|
||||
(+draft_group, +hidden_groups); the builder sizes ONE shared backing
|
||||
allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) =
|
||||
aliased pool + appended draft/hidden tensors (patch 04). After 04 the
|
||||
capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and
|
||||
the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL
|
||||
graphs on both TP ranks).
|
||||
5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter
|
||||
KV) + 5×hidden×2 (aux hidden cache) keyed on
|
||||
`speculative_config.method in {dflash, dspark, extract_hidden_states}`.
|
||||
NEVER key on `use_aux_hidden_state_outputs` — it lives on the
|
||||
gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan
|
||||
time, so the check silently fails and the reserve stays 13,600
|
||||
(patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 +
|
||||
5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.
|
||||
|
||||
**Verify cheaply before booting:** smoke-test the pure KV logic on CPU
|
||||
torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`,
|
||||
register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
|
||||
`from __future__ import annotations`, and run synthetic GLM+draft+hidden
|
||||
spec sets through the patched grouping/layout/builder functions. A 7-minute
|
||||
boot per hypothesis is the alternative.
|
||||
|
||||
**Window discipline:** stop + mask `frank-primary` FIRST and gate on
|
||||
MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
|
||||
reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
|
||||
host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
|
||||
to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once
|
||||
under pipefail via SIGPIPE before log-save, losing the capture. The abort
|
||||
path must restore prod (unmask + daemon-reload + reset-failed + start),
|
||||
never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts
|
||||
the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method":
|
||||
"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's
|
||||
echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH.
|
||||
Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter
|
||||
torch.compile is the frontier (~380 s).
|
||||
|
||||
**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s
|
||||
with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s
|
||||
prefill. Reconcile with a clean single-request measurement before trusting
|
||||
either.
|
||||
|
||||
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
|
||||
hardware — its conclusion was drawn from throughput arithmetic and is superseded
|
||||
by this page's kernel-time measurement),
|
||||
|
||||
@@ -110,6 +110,32 @@ retires the lucebox user quadlet to dated home trash
|
||||
truly free, relinks + enables + restarts the root unit, and waits up to
|
||||
1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).
|
||||
|
||||
### Preflight the GGUF without downloading it
|
||||
|
||||
GGUF metadata lives in the header and HF honours ranged GETs, so a few MB
|
||||
answers every sizing question before a 13 GB transfer
|
||||
(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64
|
||||
n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY
|
||||
= u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers
|
||||
forget). Walk arrays by byte size even for a summary: a naive parser
|
||||
desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read:
|
||||
`general.architecture`, `<arch>.block_count`, `<arch>.context_length`,
|
||||
`<arch>.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp`
|
||||
build carries `blk.<n>.nextn.*`; `--spec-type draft-mtp` then loads the
|
||||
head against the MAIN model's weights, and `--model-draft` would load the
|
||||
file twice), `<arch>.head_count_kv`, `<arch>.key_length`/`value_length`,
|
||||
`<arch>.full_attention_interval`, `<arch>.ssm.state_size`/`group_count`/
|
||||
`time_step_rank`, and for the projector `clip.projector_type` +
|
||||
`clip.has_vision_encoder`.
|
||||
|
||||
KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)):
|
||||
only full-attention layers take KV
|
||||
(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B,
|
||||
q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot
|
||||
buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the
|
||||
MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size`
|
||||
between slots — it does not multiply.
|
||||
|
||||
## Verification (2026-09-26 deploy)
|
||||
|
||||
- `/v1/models` reports `frank-primary`.
|
||||
|
||||
@@ -146,3 +146,70 @@ systemctl --user start frank-medium`) and stop/mask frank-primary. The
|
||||
_Filled in after the cutover: health, real completion, tool call, Open WebUI
|
||||
completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
|
||||
state._
|
||||
|
||||
## Dual-R9700 follow-up (2026-09-28 probe era)
|
||||
|
||||
After this single-card runbook was retired, the DS41 engine got a dual-R9700
|
||||
attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
|
||||
`unsupported model architecture 'deepseek41'`). Findings that outlive the
|
||||
probe:
|
||||
|
||||
**Budget both cards explicitly, or you waste half the box.** The engine
|
||||
sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
|
||||
every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
|
||||
looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
|
||||
cache of things still streaming from NVMe. Pin all four:
|
||||
|
||||
| env | card | meaning |
|
||||
|---|---|---|
|
||||
| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
|
||||
| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
|
||||
| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
|
||||
| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
|
||||
|
||||
Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
|
||||
card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
|
||||
|
||||
**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
|
||||
`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
|
||||
gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
|
||||
tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
|
||||
secondary=hip:1`. It is the only lever that materially raises resident
|
||||
expert count, and resident count is the entire bottleneck. Confirm every run
|
||||
with `expert ownership:` and `hybrid experts ready:` lines.
|
||||
|
||||
**The placement file must live under the mounted models dir.** JSON
|
||||
`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
|
||||
`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
|
||||
keeps what fits its budget and demotes the rest. Anywhere outside
|
||||
`/models/...` gives `cannot open expert placement` and a SILENT single-tier
|
||||
fallback — a silent-wrongness failure, not a crash.
|
||||
|
||||
**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
|
||||
exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
|
||||
is refused there. With ~22% of experts resident each token pulls ~187 cold
|
||||
experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
|
||||
the model is larger than the box's RAM and you are in page-cache thrash at
|
||||
the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
|
||||
with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
|
||||
decode, not a broken run.
|
||||
|
||||
**Harness traps, each of which produced a wrong number:** never `env -i` the
|
||||
podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
|
||||
PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
|
||||
before the container exists — pass container env with `-e`); never put a
|
||||
large prompt on curl's argv (`Argument list too long` at ~128 KB — use
|
||||
`--data-binary @file`); size prompts by growing the file until `wc -c`
|
||||
reaches the byte target AND assert depth against the engine's own
|
||||
`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
|
||||
that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
|
||||
30%); the server runs as the container's PID 1, so tear down with
|
||||
`podman rm -f`, never host `pkill`; drop page cache between arms
|
||||
(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
|
||||
every later one is a lie.
|
||||
|
||||
Read the placement log — it is the whole diagnosis
|
||||
(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
|
||||
hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
|
||||
… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
|
||||
mapped, so they do not evict the expert page cache).
|
||||
|
||||
@@ -190,6 +190,70 @@ repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
|
||||
on :7770 (`--enable-prefix-caching` in the container argv, health 200,
|
||||
live completion answering).
|
||||
|
||||
## Authoring a new profile (the qualification pipeline)
|
||||
|
||||
Order matters at every step:
|
||||
|
||||
1. **Package manifest** `packages/models/<pkg>/package.json` — target shards
|
||||
+ aux artifacts with sha256. Verify every target's byte count AND sha256
|
||||
after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`).
|
||||
2. **Placement bootstrap** from GGUF headers: parse all shards' tensor
|
||||
tables (a KV-only shard1 is normal for split parts; each part carries its
|
||||
own tensor table). Per-expert full per layer =
|
||||
(`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices
|
||||
rank0 = full×416//640, rank1 = full−rank0. Hot counts =
|
||||
floor(measured_budget / per-expert total). **Anchor budgets to the
|
||||
MEASURED hot_bytes of the qualified sibling profile, never invented.**
|
||||
3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then
|
||||
`expert_budget.validate_manifest(m)` +
|
||||
`expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>)`
|
||||
(runtime/catalog are keyword-only). Declared hot_bytes must match
|
||||
computed or it raises.
|
||||
4. **PLE derive** (containerized `tools/prepare_ple.py` with
|
||||
`--expected-bytes/--expected-type/--expected-shape`); setup skips
|
||||
re-derivation when the file exists with matching size.
|
||||
5. **profile.env**: set values BEFORE `source dual-r9700/profile.env`
|
||||
(set-first-wins; an empty string does NOT survive `:=`).
|
||||
|
||||
The canonical bootstrap.json wrapper schema is exactly
|
||||
`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path,
|
||||
artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the
|
||||
wrapper `id` MUST equal the id in the profile.json placement descriptor
|
||||
(setup rejects mismatches: "does not match profile placement"). Never
|
||||
fabricate qualification metadata — use
|
||||
`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`.
|
||||
|
||||
GGUF parsing laws for the bootstrap: expert tensor names are
|
||||
`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is
|
||||
parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes
|
||||
`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte
|
||||
extents from consecutive OFFSETS, never a type-size table (mis-buckets tails
|
||||
silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536),
|
||||
54,400,261,120 B, an entire shard.
|
||||
|
||||
Setup/start: `./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image
|
||||
sha256:<digest> --local-image --reuse-from <sibling-model-dir>
|
||||
--accept-model-license` (aux artifacts reuse byte-identical files; F16 vs
|
||||
Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed
|
||||
placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises.
|
||||
The host-only doctor FAILs when another instance holds the cards (it wants
|
||||
3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second
|
||||
instance CANNOT coexist, so qualification requires a stop+mask window for
|
||||
the incumbent, then unmask+restart.
|
||||
|
||||
**Qualification gate (unqualified → qualified):** first start runs the
|
||||
workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning
|
||||
proof (`VmLck` ≈ PLE bytes in `/proc/<container-pid>/status` when
|
||||
RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If
|
||||
headroom is violated, lower hot counts via re-trim and re-derive
|
||||
MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook.
|
||||
|
||||
Remote-execution gotchas from the same windows: `nohup X &` inside an ssh
|
||||
command races ssh teardown — the script can die before writing a byte; run
|
||||
long remote scripts synchronously, or setsid + disown, or via a system
|
||||
unit. zsh login shells abort on unquoted `===` separators; grep globs need
|
||||
quoting through ssh+zsh layers.
|
||||
|
||||
## Ops
|
||||
|
||||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||||
|
||||
@@ -407,6 +407,42 @@ frank-small, port 7771]
|
||||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||||
|
||||
## Cross-Cutting References (skill distillation, 2026-10-05)
|
||||
|
||||
One polished reference page per skill-cluster, distilled from the managed
|
||||
skill library into the vault (see [log](../log.md) for the wave plan):
|
||||
|
||||
- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
|
||||
of a loader; how the runtime is chosen by the file
|
||||
- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
|
||||
laddering, llama-bench discipline
|
||||
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
|
||||
laws, drafter placement
|
||||
- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
|
||||
heterogeneous-pair verdicts
|
||||
- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
|
||||
lever, and why free VRAM is not arena
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
|
||||
died, and interrogating a running fork's config
|
||||
- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
|
||||
versions/forks; torch.compile/dynamo breakage triage
|
||||
- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
|
||||
the window script laws
|
||||
- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
|
||||
endpoints, IBS attribution, prefix-cache routing
|
||||
- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
|
||||
research verification, second-endpoint bring-up
|
||||
- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
|
||||
verdicts, large-MoE recipe, compile-cache persistence
|
||||
- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
|
||||
proposals (the no-go verdict format)
|
||||
- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
|
||||
dependency traps
|
||||
|
||||
These sit alongside the operational pages under
|
||||
`../Operations/index.md` and `../Agent/index.md`.
|
||||
|
||||
## Related
|
||||
|
||||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
type: reference
|
||||
title: Container Host Reachability
|
||||
description: The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate.
|
||||
tags: [podman, netavark, networking, rootless, reachability, caddy]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Container Host Reachability
|
||||
|
||||
Two failure families on rootless podman + netavark hosts (Dankclaw), both
|
||||
reading as "service down" when the service is up and the *path* is wrong.
|
||||
Fleet source of truth: `serverAppConfigs-docs/Architecture/Container Network
|
||||
Namespaces.md` (commit `2e179b0`).
|
||||
|
||||
## Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly
|
||||
|
||||
A container cannot reach a `Network=host` service at the host's LAN IP.
|
||||
netavark DNATs **published** ports only: from a bridge container,
|
||||
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
|
||||
gateway `10.89.1.1` are both refused. This is what 502'd
|
||||
`hermes.franksdank.cloud` until the Caddyfile upstream became
|
||||
`host.containers.internal` on 2026-09-16 — its vhost comment had claimed the
|
||||
LAN IP was open from that bridge; it never was.
|
||||
|
||||
Probe that proves it in one command:
|
||||
|
||||
```bash
|
||||
ssh <host> "podman run --rm --network <net> --entrypoint sh \
|
||||
docker.io/library/busybox:latest -c \
|
||||
'for p in <target-port> <published-port>; do
|
||||
nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD
|
||||
done'"
|
||||
podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK'
|
||||
```
|
||||
|
||||
Published OK + target BAD ⇒ quirk confirmed. Note busybox `nc -z` reports any
|
||||
failure as failure — use it for bulk sweeps; use `/dev/tcp` (or `nc -v`) when
|
||||
refusal-vs-timeout matters.
|
||||
|
||||
Fix: point the service's host-facing URL at
|
||||
`http://host.containers.internal:<port>` in compose/quadlet env, redeploy via
|
||||
git (`deploy.sh <target> --service <svc>`), verify
|
||||
`podman exec <svc> printenv <URL_VAR>` and the connect log line (~40 s).
|
||||
**Do not publish the daemon port** to make the LAN IP work — publishing is
|
||||
exactly what makes other ports refuse.
|
||||
|
||||
## Quirk 2 — addressability is per netns, not per host
|
||||
|
||||
- `NetworkMode=host` → the container **is** the host netns: dials all of
|
||||
`127.0.0.1`, binds any free port.
|
||||
- bridge/named net → own netns; only published ports are reachable, via the
|
||||
network's **gateway IP**. Unpublished host listeners refuse **instantly
|
||||
(RST)** — no firewall rule, no stale DNAT, `nft`/`iptables -t nat` empty. Do
|
||||
not hunt for REJECT rules; probe.
|
||||
|
||||
Probe sequence (read-only):
|
||||
|
||||
```bash
|
||||
podman inspect <name> --format '{{.HostConfig.NetworkMode}}'
|
||||
sudo ss -ltnp 'sport = :<port>' # host netns only — why 0.0.0.0 can still be unreachable
|
||||
podman exec <name> cat /proc/net/route # default route = the gateway IP to dial
|
||||
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'
|
||||
```
|
||||
|
||||
Interpretation without blurring:
|
||||
|
||||
- **instant refusal** = nothing listening in a reachable namespace (or
|
||||
rootlessport rejected an unpublished port map).
|
||||
- **timeout** = no route to that address at all. `host.containers.internal`
|
||||
(`169.254.1.2`) is an alias on the **default `podman` net only**; containers
|
||||
on a named network time out against it.
|
||||
|
||||
Tool gotchas (podman 5.x, verified live):
|
||||
|
||||
- `podman network inspect <net> --format '{{json .containers}}'` errors
|
||||
(`can't evaluate field containers in type interface {}`) — list membership
|
||||
from the container side: `podman ps -a --format '{{.Names}} {{.Networks}}'`.
|
||||
- An **empty** `Networks` field = host network or shared namespace
|
||||
(`--network=container:<other>`), not "no network".
|
||||
- Quadlet `Network=<name>.network` yields the runtime name **`systemd-<name>`**
|
||||
— probe with `podman network ls`, never the `.network` filename.
|
||||
|
||||
Fix options when a bridge-net container must reach a host-netns listener:
|
||||
(a) move it to `Network=host`, or (b) publish the port — (b) makes the port
|
||||
reachable from every namespace **and the LAN**, an exposure change needing
|
||||
explicit operator approval, never a silent fix.
|
||||
|
||||
## Related trap — the pasta port gate (rootless published ports)
|
||||
|
||||
On hosts publishing ports through pasta, the port is bound in **userspace at
|
||||
container CREATE**, not when the app binds. A predecessor stopped moments
|
||||
earlier can still hold listener/TIME_WAIT state, so the replacement dies
|
||||
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
|
||||
seconds later** under `Restart=on-failure`. The failure looks like it failed,
|
||||
then looks fine, and the missing gate stays invisible because the retry masks
|
||||
it. Observed 2026-09-16 cutting `frank-large` from a root unit to a rootless
|
||||
Quadlet.
|
||||
|
||||
Gate procedure: stop the old unit AND the new one (a redeploy's running
|
||||
replacement is itself the port-holder — without that stop the gate deadlocks
|
||||
against itself), then wait for the port to be empty across **all socket
|
||||
states**:
|
||||
|
||||
```bash
|
||||
wait_port_free() {
|
||||
local port="$1" limit="${2:-90}" i
|
||||
for ((i = 1; i <= limit; i++)); do
|
||||
if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "FAIL: port $port still held after ${limit}s:" >&2
|
||||
ss -tan "sport = :$port" >&2 || true
|
||||
return 1
|
||||
}
|
||||
```
|
||||
|
||||
`sport` catches accepted-connection server sockets and TIME_WAIT residue that
|
||||
`ss -ltn` misses; fail loudly with the socket table attached. Verify the gate
|
||||
by re-running the same deploy against a running service (idempotency) and
|
||||
assert it in seam tests (`wait_port_free` present in the deploy handler).
|
||||
|
||||
## Related
|
||||
|
||||
- [[Quadlet Lifecycle]] — the unit layer these containers run under
|
||||
- [[Container Network Namespaces]] — fleet network namespace source of truth
|
||||
- [[Observability]] — the alerting paths that depend on reachability
|
||||
- [[Deploy State Triage]] — the deploy flow that installs these configs
|
||||
@@ -0,0 +1,331 @@
|
||||
---
|
||||
type: reference
|
||||
title: Fleet Audit and Posture
|
||||
description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work.
|
||||
tags: [audit, security, prometheus, caddy, monitoring, git]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Fleet Audit and Posture
|
||||
|
||||
## Monitoring gap audit
|
||||
|
||||
Use after any reboot, or whenever Prometheus targets look wrong. Two failure
|
||||
modes, only the first announces itself.
|
||||
|
||||
**1. A service that should exist and doesn't is invisible.** A missing
|
||||
exporter shows up as nothing — no failing target if the job was configured
|
||||
for a host that no longer serves, no alert because the series is absent. So
|
||||
check **existence, not health**:
|
||||
|
||||
```bash
|
||||
podman ps -a --format '{{.Names}} {{.Status}}' # includes dead ones
|
||||
ls ~/.config/containers/systemd/*.container # what should be boot-persistent
|
||||
systemctl --user is-enabled <name>.service # expect: generated
|
||||
```
|
||||
|
||||
A container in `ps -a` that is `Exited(…)` **and has no unit** is the classic
|
||||
case: rootless podman does not restart bare containers after the stop that
|
||||
shutdown performs, so a compose-only service dies at the first reboot and
|
||||
stays dead. If the compose file still names it, the repo's own rule is being
|
||||
violated — fix by writing the quadlet (mirror the same service on the other
|
||||
host), deleting the compose entry (one owner per container name), and making
|
||||
the deploy path reach it: planner service-set + path rule, dispatcher
|
||||
allowlist `<target>:<service>`, handler installing from `"$TARGET/..."` never
|
||||
a hardcoded peer's path.
|
||||
|
||||
**2. Classify every down target — do not report a count.**
|
||||
`targets?state=active` gives health **and** last error; two down targets can
|
||||
mean opposite things:
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| `connection refused` | service not listening → real outage (podman ps, logs, exit codes) |
|
||||
| `404` on `/metrics` | exporter path wrong or metrics disabled → config defect, service fine |
|
||||
| `unsupported Content-Type …` | scraping an HTML *page* as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter |
|
||||
| target absent | §1 |
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
|
||||
import json,sys
|
||||
for t in json.load(sys.stdin)['data']['activeTargets']:
|
||||
if t['health'] != 'up':
|
||||
print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
|
||||
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics # hand-probe before believing the scraper
|
||||
```
|
||||
|
||||
"7 of 11 up" without the four reasons is how a real outage hid behind three
|
||||
config defects on this fleet — the crash-looping GPU service was one of four
|
||||
failures reported as a number.
|
||||
|
||||
**3. Exposure posture while you are in there.** An exporter bound to the
|
||||
wildcard publishes an unauthenticated API on every interface. Expect exactly
|
||||
two listeners per host-exporter — loopback for host-local probes, the host's
|
||||
own LAN address for the Prometheus container: `ss -ltn | grep <port>` must
|
||||
show two explicit addresses, never `0.0.0.0`.
|
||||
|
||||
**4. Leave decisions that are not yours.** Fix mechanical drift (missing
|
||||
unit, wrong path, wildcard bind). Do NOT disable a probe to green a
|
||||
dashboard, or "fix" a crash-looping service whose posture depends on owner
|
||||
intent — capture the evidence (restart count, exit code, last error line) and
|
||||
hand it over intact.
|
||||
|
||||
## vhost proxy health probes
|
||||
|
||||
After deploying or auditing a Caddy/Anubis front door, verify **from a
|
||||
client**, not from the backend's localhost — that is where transport
|
||||
mismatches, auth walls, and dead upstreams show up.
|
||||
|
||||
```
|
||||
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/ping
|
||||
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/
|
||||
```
|
||||
|
||||
A status code is a diagnosis, not a failure:
|
||||
|
||||
- `200` on `/ping` from *arrs = alive; **auth state unknown** (`/ping`
|
||||
bypasses auth by design) — probe `/` to decide.
|
||||
- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404
|
||||
there = healthy).
|
||||
- `502` = Caddy could not reach/negotiate with the upstream: dead backend
|
||||
**or transport mismatch**. Probe the backend port directly from the proxy's
|
||||
network (`ssh <proxy-host> curl http://<backend>:<port>/…`). Direct 200 +
|
||||
vhost 502 ⇒ config bug, not an outage.
|
||||
- Login page to anonymous (grep body for `password`) = auth ON; instant
|
||||
redirect to `/transferlist/`-style routes = auth OFF = open door.
|
||||
|
||||
**Transport-mismatch mechanism:** `transport http {versions h2c 1.1}` makes
|
||||
Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it
|
||||
→ 502 per request while the same port answers 200 on plain HTTP/1.1. Fix =
|
||||
drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain
|
||||
HTTP off-loopback and needs a TLS transport. Match the transport to what the
|
||||
backend speaks.
|
||||
|
||||
**Shell trap when gating commits on a test suite:** `grep -c "^not ok"` exits
|
||||
**1 when zero mismatches** — inside an `&&` chain it silently kills
|
||||
everything after it while still printing `0`. Wrap:
|
||||
`(./tests/run.sh | grep -cE '^not ok' || true) && git commit …`.
|
||||
|
||||
## Security posture audit (behavioral)
|
||||
|
||||
Answers "what can the internet touch, and is everything else actually
|
||||
gated?" — **all findings must be behavioral probes**, not config reads.
|
||||
|
||||
**1. Inventory (source of truth).** `dankclaw/Caddyfile`: classify every
|
||||
vhost — guarded (`reverse_proxy anubis:8923`), exempt (direct upstream), or
|
||||
redirect-only. Public DNS: `dig +short +time=3 +tries=1 <name>.franksdank.cloud
|
||||
@1.1.1.1` per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by
|
||||
DNS absence.
|
||||
|
||||
**2. Guard chain (expect the challenge, not the app).** Per guarded vhost:
|
||||
`curl -sk https://<vhost>/` and grep the body for
|
||||
`anubis|challenge|proof-of-work` (~8 markers on a live challenge page). A 200
|
||||
with NO markers means the app is served directly — the guard is bypassed.
|
||||
|
||||
**3. Exempt vhosts — per-app gating probes.** OpenWebUI `/api/v1/users/me` →
|
||||
401. Gitea `/api/v1/user` → 401; repo visibility `/api/v1/repos/dank/<repo>`
|
||||
→ 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex `/web/`
|
||||
→ 302 (login is JS-rendered). Calibre-Web `/` → 302 to `/login`. Jellyfin
|
||||
`/Users` → 401. Prowlarr/Sonarr/Radarr `/api/v3/system/status`; Lidarr/Readarr
|
||||
`/api/v1/system/status` → 401 = gated, 200 = fully open. qBittorrent
|
||||
`/api/v2/torrents/info` → 403 without SID. Hermes/hermesadmin `/` → 302 to
|
||||
`/login`. IdP (`auth.`): `/` → 302 → `/if/flow/default-authentication-flow/`
|
||||
→ 200 (flow HTML is React; grep "password" finds nothing — the 200 on the
|
||||
flow URL is the proof).
|
||||
|
||||
**4. WAN port scan — from an EXTERNAL vantage.** LAN→WAN hairpin is
|
||||
unreliable; scan from gluetun's ProtonVPN egress:
|
||||
|
||||
```bash
|
||||
ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \
|
||||
8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \
|
||||
5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \
|
||||
>/dev/null 2>&1 && echo OPEN: \$p; done"'
|
||||
```
|
||||
|
||||
Confirm the egress is truly external first (`wget -qO- https://api.ipify.org`
|
||||
→ ProtonVPN IP, not Comcast). **Trap: plain-HTTP probes against TLS ports
|
||||
(443, 32400) are false negatives** — port open, protocol mismatch; re-probe
|
||||
with `wget -qO- https://IP[:port]/`. busybox wget has no `--secure` flag; it
|
||||
errors instantly and the `&&` chain silently skips the echo (a sub-second
|
||||
wall time on a "timeout" scan means the flag failed, not that ports closed).
|
||||
|
||||
**5. Residuals.** Grafana basic-auth residual: `curl -u admin:admin
|
||||
http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live
|
||||
(loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds:
|
||||
`sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no
|
||||
auth (OpenAI API, no token). Stale-record check: current WAN IP =
|
||||
`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress);
|
||||
compare against every public A record — a record at a different Comcast IP
|
||||
is stale.
|
||||
|
||||
**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`,
|
||||
`search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all
|
||||
verified) + `auth` (IdP, by design); everything else LAN-only. Open findings:
|
||||
LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback
|
||||
residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP);
|
||||
`plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin
|
||||
192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener
|
||||
(dead config, no exposure).
|
||||
|
||||
## Stash supersession audit before drop
|
||||
|
||||
Use when a repo (especially a live-deployment checkout where the working tree
|
||||
IS the deployment) has an old stash whose content may have been superseded by
|
||||
later commits, and dropping blindly risks losing unique facts.
|
||||
|
||||
1. **Inventory**: `git stash show --stat stash@{0}`.
|
||||
2. **Probe for unique content per file**: `git show "stash@{0}:<path>"` and
|
||||
diff against HEAD; count lines present only in the stash version. A file
|
||||
with zero unique-vs-HEAD lines is fully superseded — safe to drop.
|
||||
3. **Classify each unique-content hit**: *stale fact* (superseded by a later
|
||||
refactor, e.g. old model naming) → drop is correct; *live durable fact*
|
||||
(an operational invariant never re-recorded) → extract it into live
|
||||
docs/memory or flag to the user before dropping.
|
||||
4. **Archive before dropping** — zero loss: tag the stash commit, then drop:
|
||||
|
||||
```bash
|
||||
git tag archive/stash-<YYYY-MM-DD>-<base-branch>-wip "stash@{0}"
|
||||
git stash drop "stash@{0}"
|
||||
```
|
||||
|
||||
The tag pins the full stash object including its index parent; recoverable
|
||||
via `git show archive/stash-…` or `git checkout archive/stash-…^3 -- <path>`
|
||||
for the untracked-files third parent. Stashes are invisible in normal history
|
||||
and expire from reflogs (~90 days); tagging converts "drop" into a
|
||||
reversible archive at zero workflow cost.
|
||||
|
||||
## Whole-fleet audit (git/test/live/docs cross-check)
|
||||
|
||||
Deep-audit the homelab fleet — git/test state, live service health, and
|
||||
docs-vs-live cross-check. Run from the dev checkout
|
||||
(`/home/blank/Projects/serverAppConfig`).
|
||||
|
||||
**1. Probe the live fleet — self-identifying, ONE ssh per host.** Do NOT
|
||||
trust the display order of parallel SSH outputs: they can render crossed and
|
||||
you will attribute host A's output to host B (2026-09-21: a false host↔IP
|
||||
"conflict" dissolved by a single self-contained per-host probe). Put
|
||||
`hostname` + a distinctive marker + the LLM identity in the SAME command:
|
||||
|
||||
```
|
||||
for ip in 192.168.0.143 192.168.0.235; do
|
||||
ssh -o BatchMode=yes dank@$ip \
|
||||
'echo "host=$(hostname)"; \
|
||||
echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \
|
||||
echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo'
|
||||
done
|
||||
```
|
||||
|
||||
Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
|
||||
= AI stack + `frank-large` :7770 + `frank-small` :7771 (RTX 3090 + Strix
|
||||
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary`
|
||||
:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
|
||||
hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
|
||||
it is unaffected by IP drift.
|
||||
|
||||
**2. LLM endpoints — health + identity.** `curl /health` per endpoint, then
|
||||
`/v1/models` (first 120 chars) — a 200 with the right model `id`/`name` is
|
||||
the pass condition.
|
||||
|
||||
**3. Tests — the deploy completeness gate.** `./tests/run.sh` (bats) must
|
||||
end `ALL TESTS PASSED`; `tests/e2e/run.sh` runs Playwright against the LIVE
|
||||
fleet. **A single-test flake is normal**, not a regression: the plex crawl
|
||||
intermittently lands on `app.plex.tv`, the grafana SSO test occasionally
|
||||
times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just
|
||||
that spec (`npx playwright test --project=standard specs/<n>-<name>.spec.ts`)
|
||||
to classify before alarming. A `deploy.sh` e2e gate can fail on a flake
|
||||
while the change itself is fine — verify the actual change live, then re-run
|
||||
the suite.
|
||||
|
||||
**4. Repo state.** `git status --short --branch; git log --oneline -20;
|
||||
git stash list; git worktree list` (prune stale worktrees), and
|
||||
`du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a
|
||||
**shared checkout with parallel instances**: uncommitted peer WIP and
|
||||
stash-list churn are expected. Never commit or drop a peer's files. To
|
||||
deploy with a dirty tree: `git stash push -u -m "..." -- <peer paths>`,
|
||||
deploy, `git stash pop`. `deploy.sh` refuses a dirty tree.
|
||||
|
||||
**5. Prometheus — expected-down targets (not outages).** Query
|
||||
`/api/v1?query=up` from inside the container; the structurally red-by-
|
||||
construction jobs (`comfyui` HTML scrape, `services` up/down-as-scrape,
|
||||
`gitea` exporter off) are documented in [[Observability]] — anything else
|
||||
down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a
|
||||
same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is
|
||||
a *host bind*, not a published port, so the bridge-networked Prometheus
|
||||
container cannot reach it at the host's LAN IP — netavark DNATs published
|
||||
ports only. Scrape same-host services via
|
||||
`host.containers.internal:<port>`; verify the fix from inside the container
|
||||
before editing: `podman exec prometheus wget -qO-
|
||||
http://host.containers.internal:<port>/metrics`.
|
||||
|
||||
**6. Cross-check docs vs live.** Confirm the AGENTS.md host↔IP↔hardware↔
|
||||
stack mapping against step 1. The LLM endpoint table and `models.yml` are
|
||||
probed live and stay current; the prose host labels are the part that goes
|
||||
stale after an IP move. Report discrepancies; do not "fix" prose from a
|
||||
single crossed probe.
|
||||
|
||||
## Bulk-redact an identity from a docs + config tree
|
||||
|
||||
Removing a retired host / machine / project name from a repo that holds both
|
||||
documentation and live config. Evidence-first, surgical, reversible.
|
||||
|
||||
**1. Inventory before touching anything.** Count hits **per file** (the
|
||||
count decides whether this is a 5-minute job or a user-input decision),
|
||||
excluding logs/`*.db`/runtime state, and capture `git status --short` first
|
||||
— you must know which files carry in-flight peer WIP so you never stage
|
||||
them.
|
||||
|
||||
**2. Classify every hit:**
|
||||
|
||||
| Bucket | Test | Action |
|
||||
|---|---|---|
|
||||
| **Narrative** | prose describing the thing | Reword to the class ("the workstation", "a client box") |
|
||||
| **Load-bearing** | the identifier IS a lookup key, hostname match, or config value | **Keep.** Prove it: read the code that consumes it |
|
||||
| **Evidence** | quoted logs, transcripts, reports | Keep timestamps/lines; de-identify only the host field |
|
||||
|
||||
Load-bearing examples that look like prose but are not: `install.sh` doing
|
||||
`PKGS=("common.pkgs" "$HOST.pkgs")` with `HOST="$(hostname)"` (the manifest
|
||||
*filename* is the key); a `--host <name>` case list; a Wi-Fi SSID. Hardware
|
||||
model strings, driver versions, PCI addresses are subject matter, not
|
||||
identity. State the boundary in the commit message.
|
||||
|
||||
**3. Subdirectories: distill, don't rename or keep.** For a per-host doc
|
||||
tree (`docs/system/<host>/`), ask before acting — rename/trash/distill are
|
||||
genuinely different. When distilling: read every page, sort into
|
||||
still-true-and-machine-independent (fold into a general page) vs
|
||||
specific-to-the-retired-box (let go), then `mv` the directory to
|
||||
`.AGENTTRASH/YYYYMMDD-topic/` and repoint every index. Wikilinks resolve by
|
||||
filename anywhere in the tree, so deleting a page can break a link in a
|
||||
file you never grepped — run the link checker.
|
||||
|
||||
**4. Execute with count assertions, never blind sed.** Per-file mapping of
|
||||
`(path, old_exact_string, new_string, expected_count)`; abort the entry if
|
||||
the count differs; exact strings, not regex. **Do not hand this to a
|
||||
subagent** — observed: dropped list markers so bullets stopped rendering, a
|
||||
numbered list with two `3.` items, "corrected" paths to nonexistent
|
||||
directories, a deleted sentence where a reword was required. If a subagent
|
||||
already did it: `git checkout --` its touched paths (excluding the WIP
|
||||
files from step 1) and redo it.
|
||||
|
||||
**5. Audit:** `grep -rniE '<name>|<its-ip>' <scope>` (expect nothing), then
|
||||
`git diff -U0 -- <scope> | grep -E '^\+[^+]'` and **read every added line**
|
||||
— the diff is the step that catches damage; the grep only proves a string
|
||||
is gone. Then the repo's own linter.
|
||||
|
||||
**6. Commit scoped** — only your own paths, never `git add -A`. If the
|
||||
checkout IS the live deployment, committing on the current branch is usually
|
||||
correct; a branch switch can swap live config out from under a running
|
||||
process. Push/PR only when asked.
|
||||
|
||||
**7. Report the residue honestly** — what still carries the name and why:
|
||||
excluded scopes, load-bearing config, git history, and the cost of rewriting
|
||||
history (force-push + re-clones). "Purged from the working tree" is a true
|
||||
claim; "purged" alone is not.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — the monitored stack
|
||||
- [[Stale Signal Audit]] — the down-target decision table
|
||||
- [[Observability Provisioning]] — adding scrape targets and rules
|
||||
- [[Homepage Status Probes]] — the status page that renders probe states
|
||||
- [[Deploy State Triage]] — the deploy plumbing the audits exercise
|
||||
@@ -0,0 +1,272 @@
|
||||
---
|
||||
type: reference
|
||||
title: Fleet Ops Playbook
|
||||
description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host.
|
||||
tags: [operations, incidents, gpu, systemd, wol, forensics]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Fleet Ops Playbook
|
||||
|
||||
## Endpoint down, connection refused — GPU holder reclaim
|
||||
|
||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
||||
`frank-small` (dankclaw, root unit), `frank-large` (dankclaw, rootless
|
||||
Quadlet).
|
||||
|
||||
**1. Classify the stop before touching anything**
|
||||
|
||||
```bash
|
||||
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
|
||||
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
||||
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
|
||||
```
|
||||
|
||||
`inactive` + `enabled` + unmasked + `NRestarts=0` → **intentional stop**,
|
||||
never restored. No crash; do not look for segfaults. `NRestarts>0` /
|
||||
`activating` → different path; read the journal first.
|
||||
|
||||
**2. Are the GPUs actually free?** A check, never an assumption:
|
||||
|
||||
```bash
|
||||
rocm-smi --showmeminfo vram | grep Used # idle baseline ~60 MiB/card
|
||||
ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
|
||||
```
|
||||
|
||||
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
|
||||
prod unit → get the holder, and compare its bytes against the service's
|
||||
requirement before concluding a start would even fit (frank-medium 35:30
|
||||
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
|
||||
restart on allocation, not config).
|
||||
|
||||
**3. Identify the holder and prove it is abandoned**
|
||||
|
||||
```bash
|
||||
ps -o pid,ppid,user,lstart,etime,comm -p <pid>
|
||||
sudo readlink /proc/<pid>/cwd /proc/<pid>/exe
|
||||
tr '\0' ' ' < /proc/<pid>/cmdline # port, ctx, split
|
||||
sudo ls -l /proc/<pid>/fd # where stdout/stderr go
|
||||
sudo ss -tnp | grep <port>
|
||||
awk '{print $14+$15}' /proc/<pid>/stat; sleep 10; awk '{print $14+$15}' /proc/<pid>/stat
|
||||
```
|
||||
|
||||
Abandoned **only if all** hold: utime+stime **identical** across the 10 s
|
||||
sample (frozen CPU, not just low); **zero** established connections; bind
|
||||
scope `127.0.0.1` (loopback scratch, invisible to endpoint monitors); log
|
||||
file (fd 1/2 target) last written long ago, last lines showing success not a
|
||||
hang. Active CPU or live connections → someone's running session. Stop and
|
||||
ask.
|
||||
|
||||
**4. Reclaim**
|
||||
|
||||
```bash
|
||||
cp -a <logfile> <logfile>.preserved-YYYYMMDD # evidence outlives the kill
|
||||
kill <pid> # SIGTERM first; exits <1 s
|
||||
rocm-smi --showmeminfo vram | grep Used # expect ~60 MiB baseline
|
||||
```
|
||||
|
||||
Never `pkill -f` a pattern present in your own command line.
|
||||
|
||||
**5. Restore and verify — health 200 alone is NOT proof**
|
||||
|
||||
```bash
|
||||
sudo systemctl start <svc>
|
||||
for i in $(seq 1 60); do curl -s -m 5 -o /dev/null -w '%{http_code}\n' \
|
||||
http://127.0.0.1:<port>/health; sleep 10; done # 503 while loading -> 200
|
||||
sudo tr '\0' ' ' < /proc/<MainPID>/cmdline | grep -oE '\-\-(tensor-split|split-mode|ctx-size|port|spec-type|parallel|flash-attn)'
|
||||
```
|
||||
|
||||
Prove with a **real completion** from off-host — content, reasoning, spec
|
||||
counters:
|
||||
|
||||
```bash
|
||||
curl -sS -m 180 http://<host>:<port>/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"<alias>","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":64,"temperature":0}'
|
||||
# choices[0].message.content / .reasoning_content / timings.draft_n, draft_n_accepted
|
||||
```
|
||||
|
||||
Record the runtime fingerprint (`<bin> --version` → `b<build>, commit <sha>`)
|
||||
and confirm it is the pinned runtime. Append a dated entry to
|
||||
`serverAppConfigs-docs/log.md`, run `./tests/run.sh`, commit, push (origin is
|
||||
the local Gitea on Dankclaw — push allowed).
|
||||
|
||||
**Gotchas seen here:** dankserver's login shell is ZSH — never start an
|
||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||
timeout. First request after a halogen (`frank-large`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-large`:
|
||||
read logs **without** sudo (per-user storage). If the restore never happened,
|
||||
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||
|
||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||
|
||||
Any scratch/experimental vLLM container on a host that also runs a production
|
||||
LLM service (`frank-primary`, `frank-large`, `frank-small`) MUST stop the
|
||||
production unit first, and the window script must verify it. Two independent
|
||||
failure modes if skipped:
|
||||
|
||||
1. **Expert-offload planner dies on host RAM.**
|
||||
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
||||
plan time. A production rank worker holds tens of GiB of unevictable
|
||||
pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod
|
||||
boot sees ~86 GiB instead of ~239 and dies at plan time:
|
||||
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
||||
Looks like a planner bug; is pure RAM contention. Signature: container
|
||||
exits within ~60 s, after `Resolved architecture: …` but before worker
|
||||
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
|
||||
gun.
|
||||
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
|
||||
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
|
||||
only once the prod unit is down.
|
||||
|
||||
Window skeleton (dankserver):
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-primary.service
|
||||
sudo systemctl mask --runtime frank-primary.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
|
||||
sleep 3
|
||||
podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; }
|
||||
# ... launch scratch container, watch logs ...
|
||||
podman rm -f <scratch> # stuck vLLM ignores SIGTERM
|
||||
sudo systemctl unmask --runtime frank-primary.service
|
||||
sudo systemctl reset-failed frank-primary.service
|
||||
sudo systemctl start frank-primary.service
|
||||
# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink
|
||||
```
|
||||
|
||||
The deploy restores the unit after bench windows, so stop is sufficient
|
||||
there; `mask --runtime` is for scratch boots that must survive a stray
|
||||
restart. Always end a window by re-checking the conjunction before
|
||||
reporting — if a restore after failure doesn't happen automatically, prod
|
||||
stays DOWN. See the repo's benchmarking policy (AGENTS.md) and
|
||||
[[LLM Bench Windows and Measurement]].
|
||||
|
||||
## Iterating on a fleet service before commit
|
||||
|
||||
The repo's contract is git → `deploy.sh` → host, but live testing needs the
|
||||
new file on the host first, and the naive way (copy into the host checkout)
|
||||
blocks the very deploy that would install it. The loop:
|
||||
|
||||
1. Edit locally; run `./tests/run.sh`.
|
||||
2. Ship working files **for testing**:
|
||||
`cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>'`
|
||||
(then `chmod +x` if executed — `cat >`/scp drop the exec bit and a
|
||||
container entrypoint fails status 126, "found but not executable").
|
||||
3. Test against the live service; iterate.
|
||||
4. **Before committing**, prove the host copy is byte-identical to what you
|
||||
are about to commit (`sha256sum` both ends) — otherwise step 6 loses work.
|
||||
5. Commit and push.
|
||||
6. Clean the host checkout so the deploy can pull: untracked hand copies →
|
||||
`mv` to `/tmp/handcopy/`; modified tracked files → `git checkout -- <paths>`
|
||||
(safe only because of step 4); confirm `git status --short` prints nothing.
|
||||
7. `./scripts/deploy.sh <target> --service <svc>`.
|
||||
|
||||
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
|
||||
working tree files would be overwritten by merge" — exactly the file you
|
||||
shipped); `git checkout --` silently eats uncommitted work without step 4; a
|
||||
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
|
||||
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
|
||||
container restart with no deploy; root-owned files from a container write
|
||||
need `chown` back before git operations; and **a restart is not a
|
||||
verification** — verify the *specific* claim (new code path's own output,
|
||||
file checksum on the host, the API serving the resource).
|
||||
|
||||
Why not keep a scratch copy on the host? The deployed artifact must be the
|
||||
committed one. Probes may live anywhere; runtime state lives in git.
|
||||
|
||||
## Incident window reconstruction
|
||||
|
||||
Use when the question is **"what was running when it broke?"** and the answer
|
||||
must come from artifacts, not recollection. Three independent clocks,
|
||||
cross-checked:
|
||||
|
||||
1. **Per-run duration** — scripts wrap commands in `time` → `real 4m19.129s`.
|
||||
2. **Log mtime** — last write of the log.
|
||||
3. **Journal first-failure instant** — first error line from the affected
|
||||
service.
|
||||
|
||||
`start ≈ mtime − real` for the last run in a log; each run nests inside the
|
||||
previous one's window; the run whose `[start, end]` contains the journal's
|
||||
first-failure instant is the answer.
|
||||
|
||||
```bash
|
||||
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
|
||||
ssh <host> 'grep -ah "^real" <logdir>/*.log'
|
||||
ssh <host> 'grep -aE "^####|^\+ /path/to/binary" <logdir>/<name>.log | sed "s/\(.\{130\}\)/\1\n /g"'
|
||||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
|
||||
```
|
||||
|
||||
Traps (all hit for real): `/tmp` does not survive a reboot — launch
|
||||
experiment scripts from a persistent path (`/home/dank/models/logs/<tag>.log`
|
||||
survived while `/tmp/x4*.sh` died); a log's mtime is not proof a run finished
|
||||
(truncation with the fd still held keeps growing); long command lines are
|
||||
display-truncated — wrap before reading or you report parameters never used;
|
||||
off-by-one filenames (`x4b.log` vs `x4b-experiment.log`) silently skip
|
||||
evidence — list the directory and match exactly; **overlap is a hypothesis,
|
||||
not an observation** unless a sampler proves it; do not invent the cause —
|
||||
reconstruct the *window* and keep the *mechanism* separate (`journalctl -k |
|
||||
grep -i oom`); publish "trigger not established" when the proof is absent.
|
||||
|
||||
Fleet-specific: container healthchecks shell out to the **podman CLI**, so
|
||||
the first `invalid internal status … could not find any running process` in
|
||||
the user journal is an early, per-minute indicator of rootless-podman state
|
||||
death — it timestamps the incident to within one healthcheck interval (how
|
||||
the 18:58:30 Dankclaw break was pinned to a bench still loading).
|
||||
|
||||
Worked example (2026-09-16, Dankclaw): log mtimes x4e-run 18:58:29 /
|
||||
x4f-run 18:59:56; durations E1 `real 4m53.946s`, F `real 5m24.785s` (exit 143
|
||||
= killed); journal first failure 18:58:30 ⇒ **F** was in flight (launched
|
||||
18:54:31, inside its window).
|
||||
|
||||
## Wake-on-LAN for a sleeping fleet host
|
||||
|
||||
A fleet host is asleep (100% ping loss; `ip neigh show <ip>` shows STALE or
|
||||
FAILED) and you need it up — typically the FrankBook-Pro Mac mini
|
||||
(192.168.0.112, **MAC `de:6b:6d:c1:52:23`**, sleeps and breaks the e2e
|
||||
`11-admin` spec).
|
||||
|
||||
- **Find the MAC**: `ip neigh show <ip>` — even a STALE entry carries it;
|
||||
fallback Pi-hole/DHCP leases or fleet docs.
|
||||
- **Send the packet**: `6×0xFF + 16×MAC` = 102 bytes, UDP to port **9** (and
|
||||
7), broadcast. The dev workstation has **two interfaces on 192.168.0.0/24**
|
||||
(Wi-Fi + wired USB-Ethernet, default route metric 100) — a plain
|
||||
`sendto("255.255.255.255", 9)` leaves only the default-route interface.
|
||||
Bind explicitly to each interface's local IP and target the subnet
|
||||
broadcast `192.168.0.255` as well as `255.255.255.255`; burst ~15 packets
|
||||
per interface ~0.2 s apart. `wakeonlan` is not installed; use a raw UDP
|
||||
socket with `SO_BROADCAST=1`:
|
||||
|
||||
```python
|
||||
import socket, time
|
||||
MAC = "de:6b:6d:c1:52:23"
|
||||
mac = bytes(int(x,16) for x in MAC.split(":"))
|
||||
pkt = b"\xff"*6 + mac*16 # 102 bytes
|
||||
for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"),
|
||||
("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]:
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
|
||||
s.bind((local, 0))
|
||||
for _ in range(15):
|
||||
s.sendto(pkt,(dest,9)); s.sendto(pkt,(dest,7)); time.sleep(0.2)
|
||||
s.close()
|
||||
```
|
||||
|
||||
- **Poll for the wake**: a Mac mini takes **30–90 s**; `ping -c 1 -W 1` every
|
||||
5 s for ~2–3 min before concluding failure.
|
||||
- **If it does not wake** (the common case): macOS honors WoL only when BOTH
|
||||
are enabled on the mini — **System Settings → Battery → Options → "Wake
|
||||
for network access"** = ON, and the active port's **"Enable Wake for LAN"**
|
||||
(Network → [interface] → Details). If either is off, no packet from any
|
||||
source wakes it, and you cannot change it while asleep. A full power-off
|
||||
(smart plug cycled) ignores WoL entirely. Ask the user to toggle the
|
||||
setting, then re-fire.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the bench windows this playbook arms
|
||||
- [[Remote Job Runner]] — supervising the long jobs that create these windows
|
||||
- [[Container Host Reachability]] — the pasta port gate at container create
|
||||
- [[Quadlet Lifecycle]] — the unit layer behind stop/mask/start
|
||||
- [[Model Swap and Rollback]] — swap-time incidents and rollbacks
|
||||
@@ -0,0 +1,204 @@
|
||||
---
|
||||
type: reference
|
||||
title: Payload and Disk Reclaim
|
||||
description: Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index, prove nothing live references the payload, stage to trash with fingerprints, purge only on approval; plus the purge-and-refresh protocol with fetch verification.
|
||||
tags: [operations, disk, podman, payloads, huggingface, trash]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Payload and Disk Reclaim
|
||||
|
||||
Standing procedure for "clean up old models" / "disk is full" on a
|
||||
serverAppConfigs host. Order matters: **measure, prove nothing live uses it,
|
||||
stage, ask.** Staging frees nothing until the user approves the purge — say so
|
||||
up front or the request reads as unfulfilled.
|
||||
|
||||
## 1. Measure the whole filesystem, not the category asked about
|
||||
|
||||
```sh
|
||||
du -sh /home/dank/models /home/dank/.local /home/dank/.cache \
|
||||
/home/dank/comfyui-data /home/dank/GitPackages 2>/dev/null
|
||||
df -h /home/dank | tail -1
|
||||
podman system df # reclaimable image space
|
||||
du -sh ~/.local/share/containers # where rootless storage physically is
|
||||
```
|
||||
|
||||
The answer is usually NOT the category the user named. On Dankclaw
|
||||
(2026-09-28) retired models were 510G, but podman images were **279G
|
||||
reclaimable (95%)** and uv+pip caches **73G** — 352G of regenerable garbage,
|
||||
bigger than the models. Lead with that finding; it changes the decision.
|
||||
|
||||
Check for sparse files before claiming a saving:
|
||||
|
||||
```sh
|
||||
stat -c 'apparent=%s blocks=%b' <file>; du -sh --apparent-size <file>; du -sh <file>
|
||||
```
|
||||
|
||||
A 3.3G "log" (`logs/x4e-run.log`) was **4K real** (8 blocks × 512) — deleting
|
||||
it frees nothing. `du` vs `du --apparent-size` finds these instantly.
|
||||
|
||||
## 2. Classify against the runbook index
|
||||
|
||||
Every payload directory maps to a runtime in the AGENTS.md retired-runbook
|
||||
table. Three buckets:
|
||||
|
||||
- **live** — referenced by a running service's env/unit.
|
||||
- **rollback** — named as a rollback target in a runbook. ASK before
|
||||
touching: they are the difference between a one-minute flip and a 30G
|
||||
re-download.
|
||||
- **retired** — maps to a retired runtime. Safe candidates.
|
||||
|
||||
Trap: a directory can hold **both** a dead target and a live drafter
|
||||
(frank-large's DFlash2 drafter lives beside the previous target). Move the
|
||||
single file, never the directory, and say so in the manifest.
|
||||
|
||||
## 3. The live-reference check — before moving anything
|
||||
|
||||
```sh
|
||||
for d in <candidate dirs>; do
|
||||
grep -rl "$d" --include="*.sh" --include="*.env" --include="*.service" \
|
||||
--include="*.container" --include="*.yaml" . | grep -v '^./old/'
|
||||
done
|
||||
```
|
||||
|
||||
Then **read every hit**; do not count them. Most are benign (comments,
|
||||
retired fetch-script destinations, `retire_*_quadlet` helpers that move a
|
||||
quadlet not a model) — but on Dankclaw this check found a live bug:
|
||||
`serve-frank-small.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
|
||||
payload retired a week earlier. The env file always overrode it, so it was
|
||||
invisible; a missing env file would have produced a "missing asset" naming a
|
||||
model that no longer exists.
|
||||
|
||||
**Durable lesson:** `:`-defaults in a serve script are never exercised in
|
||||
production, so they rot silently. When reclaiming a payload, grep serve
|
||||
scripts too — the fallback is a live reference in all but name.
|
||||
|
||||
Also check service **env files** (`scripts/*.env`) for `MODEL_DIR`/`MODEL`
|
||||
paths, not just unit `Volume=` lines: a payload can be mounted by path and
|
||||
named by env. `models/logs` and small tool dirs are not cache — logs are
|
||||
evidence, keep them.
|
||||
|
||||
## 4. Stage, never delete
|
||||
|
||||
```sh
|
||||
STAGE=/home/dank/.AGENTTRASH/$(date +%Y%m%d)-<topic>
|
||||
mkdir -p "$STAGE"
|
||||
mv <path> "$STAGE"/
|
||||
```
|
||||
|
||||
Same filesystem ⇒ instant and recoverable — **and reclaims no space**; say so
|
||||
rather than implying space was freed. Write `MANIFEST.md` into the stage dir:
|
||||
per item — path, size, owning runtime, when retired, the `mv` command to
|
||||
restore. Record what you **deliberately kept** and why; that is what a future
|
||||
reader needs. Always check for pre-existing staged trash
|
||||
(`du -sh ~/.AGENTTRASH/*`): 118G from a 2026-09-21 cleanup sat un-purged;
|
||||
report the running total, not just the new batch. Root-owned leftovers need
|
||||
`sudo`, and check `swapon --show` first (a trashed swapfile must not be
|
||||
active).
|
||||
|
||||
## 5. Verify, then report — and do not purge
|
||||
|
||||
```sh
|
||||
systemctl is-active frank-large frank-small
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
|
||||
```
|
||||
|
||||
Report: staged items and sizes, what was kept and why, what is **still not
|
||||
freed**, remaining candidates untouched. Purge only on explicit approval.
|
||||
|
||||
## Repo-flow gotcha: do NOT push before you deploy
|
||||
|
||||
`scripts/deploy.sh` sets `BASE_REVISION=$(git rev-parse origin/main)`. If you
|
||||
pushed separately, the diff is empty and the planner reports "changed
|
||||
services: (none)" — looks like a planner bug, is not one. The tool is
|
||||
designed for the **unpushed** state: edit → test → `deploy.sh` (it pushes,
|
||||
then dispatches). If you already pushed, pull on the host instead, and reason
|
||||
about whether a restart is actually owed. Any serve-script/env change needs
|
||||
`./tests/run.sh` before commit; a `:`-default change is usually **inert**
|
||||
(env overrides it) and owes no restart — but say so explicitly.
|
||||
|
||||
# Purge and refresh protocol
|
||||
|
||||
When the task is clearing/reclaiming/**refreshing** model caches on a serving
|
||||
host, add these steps.
|
||||
|
||||
## Fingerprint before deleting
|
||||
|
||||
`ls -l --block-size=1 <dir>`; for LLM payloads compare each file against the
|
||||
Hub's declared size. Fingerprints are the only way to answer "did the refresh
|
||||
change anything" afterwards — **a refresh is also an update check**: if
|
||||
upstream never changed, the bytes come back identical, which is itself the
|
||||
finding. Sidecars are versioned **in place**: equal main-checkpoint sizes do
|
||||
not prove the repo is unchanged, and a stale sidecar is invisible from its
|
||||
name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560 while the main
|
||||
checkpoint returned byte-identical). If a file is irreplaceable upstream
|
||||
(delisted revision, gated repo), say so before deleting.
|
||||
|
||||
## Confirm re-fetchability first
|
||||
|
||||
Query the Hub anonymously; a gated repo 403s even with the host's token:
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
|
||||
| python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
|
||||
```
|
||||
|
||||
An error/empty result means it is NOT safe to delete. Filter by **quant
|
||||
name**, never `endswith('.gguf')` — mradermacher ships large quants as
|
||||
`.partNofM` fragments that filter hides.
|
||||
|
||||
## Refresh per engine — know the recovery path BEFORE deleting
|
||||
|
||||
- **frank-large (halogen)**: the engine fetches its own payload via
|
||||
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
|
||||
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
|
||||
podman refuses a mount whose source is missing
|
||||
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
|
||||
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB
|
||||
at 130–240 MB/s ≈ 10–20 min, then load.
|
||||
- **llama.cpp services**: no self-fetch; the serve script refuses to start
|
||||
without weights. Pull yourself and verify:
|
||||
|
||||
```bash
|
||||
curl -fL -C - --retry 10 -o "$DEST/<file>" \
|
||||
"https://huggingface.co/<repo>/resolve/main/<file>" # -f is mandatory
|
||||
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to LFS oid + GGUF magic
|
||||
```
|
||||
|
||||
`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — a
|
||||
gated repo once produced three 153-byte shards with identical sha256s and a
|
||||
cheerful DONE line for 104 GiB. Always `-fL -C -`, compare byte counts to the
|
||||
Hub's declared sizes, compare sha256 to the LFS oid. Assemble `.partNofM`
|
||||
fragments with `cat part1 part2 part3 > name.gguf` in numeric order (per-part
|
||||
hashes verify the whole — concatenation is deterministic). Public repos need
|
||||
no token: omit `-H "Authorization: Bearer …"` from argv entirely (it is
|
||||
world-readable in `/proc/<pid>/cmdline`).
|
||||
|
||||
## Restore, then verify behaviour — not health alone
|
||||
|
||||
Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`),
|
||||
`reset-failed`, `start` (root units need `sudo`). Watch the engine's own
|
||||
fetch/load lines, then `/health`, `/v1/models`, and a **real completion** on
|
||||
every restored endpoint. Read the engine's memory ledger (kv pool / weights /
|
||||
blocks left) rather than assuming the config still fits. Sweep the vhosts too
|
||||
(`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki` via
|
||||
`curl -sk --resolve <host>:443:192.168.0.143`) — a purge touches state the
|
||||
whole host shares.
|
||||
|
||||
## Things that bite
|
||||
|
||||
- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark** —
|
||||
the A* run commits ~116 GiB and leaves ~10 GiB available; cache churn risks
|
||||
the OOM/state failures this fleet has already seen.
|
||||
- **Reboot is the repair of last resort** if rootless podman's pause process
|
||||
dies mid-work (`invalid internal status … could not find any running
|
||||
process`); `podman system migrate` panics in 5.7.0.
|
||||
- A same-filesystem trash reclaims no space; root-owned leftovers need
|
||||
`sudo`; never `rm` without explicit permission.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Model Swap and Rollback]] — the runbook index this classification reads
|
||||
- [[Quadlet Lifecycle]] — restoring the unit after a payload refresh
|
||||
- [[Remote Job Runner]] — supervising long re-fetches
|
||||
- [[Deploy State Triage]] — the push-before-deploy trap in context
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
type: reference
|
||||
title: Remote Job Runner
|
||||
description: Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally, the script, its invariants, verification, and reporting discipline.
|
||||
tags: [operations, ssh, jobs, sentinel, detached]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Remote Job Runner
|
||||
|
||||
Use for any long-running job on a remote host (benchmarks, model loads,
|
||||
builds, sweeps) whose result must survive the session. Also whenever you catch
|
||||
yourself writing `ssh host 'tail -f log | sed …'` as a watcher.
|
||||
|
||||
## Why the obvious approach fails structurally
|
||||
|
||||
`ssh host 'tail -f log | grep/sed …'` as a background watcher fails in ways
|
||||
that all LOOK like "the job died":
|
||||
|
||||
- the session broker **aborts long-lived ssh** — the watcher vanishes and the
|
||||
result is lost;
|
||||
- a `ready` regex can match the log's own **header** on the first line — the
|
||||
supervisor reports success instantly while the job is still running;
|
||||
- a `sed -n '/PAT/,$p'` filter **exits at the first match**, ending the pipe;
|
||||
- output goes into the ssh stream, so an aborted session loses everything.
|
||||
|
||||
## The mechanism: detach on the HOST, poll a sentinel file
|
||||
|
||||
Completion is a **`.done` sentinel touched last**, not a log pattern. Status
|
||||
is then one fast independent command with no long-lived process to lose.
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# job.sh {start|status|wait|out|tail|watch} ... Jobs live in $JOBS.
|
||||
set -uo pipefail
|
||||
JOBS=/home/dank/jobs
|
||||
mkdir -p "$JOBS"
|
||||
|
||||
case "${1:-}" in
|
||||
start)
|
||||
NAME="$2"; shift 2
|
||||
[ "${1:-}" = "--" ] && shift
|
||||
rm -f "$JOBS/$NAME.done" "$JOBS/$NAME.rc"
|
||||
# setsid detaches from this session's process group: the critical bit.
|
||||
setsid nohup bash -c '
|
||||
NAME="$1"; shift
|
||||
"$@" > "'"$JOBS"'/$NAME.out" 2>&1
|
||||
echo $? > "'"$JOBS"'/$NAME.rc"
|
||||
touch "'"$JOBS"'/$NAME.done"
|
||||
' _ "$NAME" "$@" < /dev/null > /dev/null 2>&1 &
|
||||
echo "started $NAME"
|
||||
;;
|
||||
status)
|
||||
# Glob *.out (not *.done) or running jobs are invisible.
|
||||
names="${2:-$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)}"
|
||||
for n in $names; do
|
||||
if [ -f "$JOBS/$n.done" ]; then
|
||||
printf ' %-18s DONE rc=%s %s lines\n' "$n" \
|
||||
"$(cat "$JOBS/$n.rc" 2>/dev/null)" \
|
||||
"$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
|
||||
elif [ -f "$JOBS/$n.out" ]; then
|
||||
printf ' %-18s RUN %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
|
||||
else
|
||||
printf ' %-18s PENDING\n' "$n"
|
||||
fi
|
||||
done
|
||||
;;
|
||||
wait)
|
||||
NAME="$2"; LIMIT="${3:-3600}"; waited=0
|
||||
while [ ! -f "$JOBS/$NAME.done" ] && [ "$waited" -lt "$LIMIT" ]; do
|
||||
sleep 5; waited=$((waited+5))
|
||||
done
|
||||
[ -f "$JOBS/$NAME.done" ] \
|
||||
&& echo "DONE $NAME rc=$(cat "$JOBS/$NAME.rc" 2>/dev/null)" \
|
||||
|| echo "TIMEOUT $NAME after ${waited}s"
|
||||
tail -"${4:-25}" "$JOBS/$NAME.out" 2>/dev/null
|
||||
;;
|
||||
out) NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;;
|
||||
tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;;
|
||||
watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;;
|
||||
*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;;
|
||||
esac
|
||||
```
|
||||
|
||||
```bash
|
||||
job.sh start NAME -- <command> # launch detached
|
||||
job.sh status [NAME] # one-shot state
|
||||
job.sh wait NAME 900 25 # block, then print the tail
|
||||
job.sh out NAME 60 # retrieve any past result
|
||||
job.sh tail [NAME] # live follow
|
||||
job.sh watch # refreshing status table
|
||||
```
|
||||
|
||||
## Invariants
|
||||
|
||||
- **`.done` is touched LAST**, after the exit code is written — its presence
|
||||
alone means the output file is final.
|
||||
- **Glob `*.out` for status**, never `*.done` — globbing `.done` hides running
|
||||
jobs entirely, the exact blindness this fixes.
|
||||
- **Capture the exit code** (`rc=7` proved a self-test worked; without it a
|
||||
job can "complete" having failed).
|
||||
- **Never put the payload in a line you grep for** — the sentinel is the only
|
||||
completion signal.
|
||||
- **Never `pkill -f <pattern>`** where the pattern also matches your own ssh
|
||||
command line — it kills the session. Distinctive names or `pkill -x`.
|
||||
- **Write helper scripts to a file and `scp` them.** Triple-nested
|
||||
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
||||
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line);
|
||||
every failure disappeared once the script was a file.
|
||||
|
||||
## Verify before trusting it (3am law)
|
||||
|
||||
```bash
|
||||
job.sh start selftest -- bash -c 'echo one; sleep 2; echo two; exit 7'
|
||||
job.sh start slowtest -- bash -c 'echo start; sleep 30; echo end'
|
||||
sleep 4; job.sh status # selftest DONE rc=7; slowtest RUN
|
||||
```
|
||||
|
||||
Both paths, including the failure case, and the rc round-trip — a runner that
|
||||
only proves the happy path is not trustworthy.
|
||||
|
||||
## Reporting discipline
|
||||
|
||||
- Give the user a real terminal when the run is long:
|
||||
`alacritty --title "bench - dankserver" -e bash -c 'ssh -o
|
||||
ServerAliveInterval=30 -t dank@192.168.0.235 "bash /home/dank/job.sh tail"' &`
|
||||
- State the MECHANISM, not vibes: the job is durable on the host;
|
||||
`job.sh out NAME` retrieves it later. Do not narrate "still loading"
|
||||
repeatedly — check once, act, report the resolved result.
|
||||
- When the user asks for status, one fast command, not a narration.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the sweeps this runner executes
|
||||
- [[Deploy State Triage]] — the two-process deploy this lesson generalizes
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
type: reference
|
||||
title: SSH and Remote Access
|
||||
description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session.
|
||||
tags: [ssh, remote, fish, gui, diagnosis]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# SSH and Remote Access
|
||||
|
||||
## Passwordless SSH bootstrap without sshpass/expect
|
||||
|
||||
Use when you must install your pubkey on remote hosts but `sshpass` and
|
||||
`expect` are not installed and you only have a password. Modern OpenSSH
|
||||
(≥ 8.4) can drive the password prompt from a helper program, so the password
|
||||
flows through the **process env only** — never a file, never argv.
|
||||
|
||||
1. **Generate a key** (if none):
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
|
||||
```
|
||||
2. **Create an askpass helper** that echoes the password from an env var (the
|
||||
helper itself stores NO secret):
|
||||
```bash
|
||||
printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
|
||||
chmod 700 /tmp/askpass.sh
|
||||
```
|
||||
3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY:
|
||||
```bash
|
||||
env DEPLOY_PW='<password>' \
|
||||
SSH_ASKPASS=/tmp/askpass.sh \
|
||||
SSH_ASKPASS_REQUIRE=force \
|
||||
setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
|
||||
```
|
||||
Password via the transient `env` argument only.
|
||||
4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names
|
||||
(`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`).
|
||||
5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` —
|
||||
BatchMode fails fast if a password would be required; a clean success
|
||||
proves the key works.
|
||||
|
||||
Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
|
||||
TTY is present and prompts interactively (hangs in a non-interactive tool);
|
||||
`setsid` detaches the controlling terminal — required on older OpenSSH even
|
||||
with `force`; the helper must be executable (`chmod 700`) and print ONLY the
|
||||
password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
|
||||
be rejected first ("Too many authentication failures"). Passwordless **sudo**
|
||||
on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`).
|
||||
|
||||
## Remote execution on a fish login shell
|
||||
|
||||
Running remote commands or shipping files over SSH to a host whose login
|
||||
shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN
|
||||
`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the
|
||||
workstation).
|
||||
|
||||
1. **Auth**: `SSHPASS='<pw>' sshpass -e ssh -o
|
||||
PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'`
|
||||
— password via env, never argv (ps-visible).
|
||||
2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login
|
||||
shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at
|
||||
*parse* time on fish (`Missing end to balance this for loop`). Always wrap
|
||||
compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b ||
|
||||
echo MISSING; done"'` (escape inner `$`).
|
||||
3. **File shipping**: never combine `cat > /remote/path <<EOF` with a
|
||||
trailing `bash -s` in the same ssh command — `cat` drains stdin, so the
|
||||
interpreter silently runs on empty stdin and prints nothing (a silent
|
||||
no-op that *looks* like success). Ship real bytes via stdin redirect:
|
||||
`ssh host 'cat > /remote/path' < localfile`, then verify byte-truth with
|
||||
`md5sum` on both ends.
|
||||
4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish
|
||||
(`$status` there) — wrap in bash or drop the status echo; ssh's own exit
|
||||
code propagates to the caller anyway.
|
||||
|
||||
Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first
|
||||
probe died on fish loop parsing; a `cat >` ship silently swallowed the
|
||||
heredoc meant for `bash -s`, deploying wrong bytes until md5-compared.
|
||||
|
||||
Related: dankserver's login shell is ZSH — never start an unquoted word with
|
||||
`=` in an ssh one-liner (`echo ===` aborts the whole chain).
|
||||
|
||||
## GUI hang — remote diagnosis over SSH
|
||||
|
||||
Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||||
|
||||
1. **Find the smoking gun** — the kernel hung-task reporter already logged
|
||||
it:
|
||||
```bash
|
||||
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
|
||||
```
|
||||
Look for `INFO: task X blocked for more than N seconds` and `blocked on a
|
||||
semaphore likely last held by task Y:PID` (modern kernels name the lock
|
||||
holder outright).
|
||||
2. **Full stacks** for victim and holder:
|
||||
```bash
|
||||
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
|
||||
cat /proc/<pid>/status | head -5 # State: R (spinning) vs D (blocked)
|
||||
```
|
||||
Victim `D` + holder `R` = livelock/deadlock with the lock held;
|
||||
`voluntary_ctxt_switches` in the millions on an `R` kthread = spinning,
|
||||
not progressing.
|
||||
3. **Hardware or software?** GPU/CPU still alive, no Xid errors?
|
||||
```bash
|
||||
nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
|
||||
journalctl -b -k --no-pager | grep -i xid
|
||||
```
|
||||
Instant answers with no Xid ⇒ driver lock issue, not hardware.
|
||||
4. **Trigger window**: block start = first hung message timestamp − reported
|
||||
seconds; then inspect user-session events around it
|
||||
(`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
|
||||
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`),
|
||||
USB workers, etc.
|
||||
|
||||
**Recoverability verdict:** compositor in `D` state is unkillable and a
|
||||
spinning kernel thread holding the lock cannot be poked — **the only exit is
|
||||
reboot**: `sync && systemctl reboot --force --force` (double-force skips
|
||||
stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on
|
||||
something that can complete, e.g. storage), recovery may be possible by
|
||||
clearing that dependency first.
|
||||
|
||||
After diagnosing: map stack symbols to driver internals to name the exact
|
||||
subsystem, check upstream for the exact symbols + driver version, and propose
|
||||
targeted off-switches (`NVreg_RegistryDwords`, module params from
|
||||
`/proc/driver/nvidia/params`) rather than blanket driver swaps.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Fleet Ops Playbook]] — incident procedures that run over these sessions
|
||||
- [[Remote Job Runner]] — long jobs that must survive session loss
|
||||
- [[Deploy State Triage]] — the deploy flow these credentials serve
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
type: index
|
||||
description: Operational procedures for the fleet — job runners, disk and payload reclaim, container reachability, audits, SSH craft, and incident playbooks.
|
||||
tags: [operations, index]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Operations
|
||||
|
||||
Procedures for operating the fleet. The LLM-specific pages live under
|
||||
[LLM](../LLM/index.md); infrastructure and architecture under the
|
||||
`../Infrastructure/` and `../Architecture/` folders.
|
||||
|
||||
- [[Remote Job Runner]] — detached long-job execution with sentinel files;
|
||||
why ssh tail-watchers fail
|
||||
- [[Payload and Disk Reclaim]] — measure-classify-prove-stage-ask disk
|
||||
procedure, plus the purge-and-refresh protocol
|
||||
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
|
||||
the pasta port gate
|
||||
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
|
||||
stop-first, pre-commit iteration, incident-window reconstruction, WoL
|
||||
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
|
||||
posture, whole-fleet audit, bulk redaction, stash audits
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
|
||||
login shells, GUI-hang diagnosis over ssh
|
||||
|
||||
## Related
|
||||
- [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)
|
||||
— monitoring config as code
|
||||
- [Deploy State Triage](../Architecture/Deploy%20State%20Triage.md) — the
|
||||
deploy plumbing
|
||||
- [Quadlet Lifecycle](../Architecture/Quadlet%20Lifecycle.md) — the unit layer
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
type: reference
|
||||
title: PICO-8 Asset Standards
|
||||
description: Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction, detail flow and hierarchy, sheet organization, and the cohesion checklist.
|
||||
tags: [pico8, game-dev, pixel-art, standards]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# PICO-8 Asset Standards
|
||||
|
||||
Ensure every PICO-8 asset (tileset, spritesheet, character sheet, or asset
|
||||
sheet) is cohesive, production-grade, and ready to drop into a game without
|
||||
rework.
|
||||
|
||||
## Palette discipline
|
||||
|
||||
- Use the PICO-8 16-color palette exclusively.
|
||||
- Establish a dominant color family (2–3 related hues) for the main subject.
|
||||
- Reserve accent colors for highlights, danger zones, and focal points only.
|
||||
- Maintain consistent color roles across all assets in a sheet (if shadow is
|
||||
dark blue on one tile, it is dark blue everywhere).
|
||||
|
||||
## Perspective & lighting
|
||||
|
||||
- Choose ONE perspective per sheet (top-down, side-view, or isometric) and
|
||||
never mix.
|
||||
- Apply light from a single consistent direction across all tiles and
|
||||
sprites.
|
||||
- Shadows follow the same shape logic on every asset (e.g. all cast shadows
|
||||
point bottom-right).
|
||||
|
||||
## Detail flow & hierarchy
|
||||
|
||||
- Start tiles/props from the palette's dominant colors, then add edge
|
||||
highlights and shading.
|
||||
- Keep tile density uniform — avoid one hyper-detailed tile next to empty
|
||||
flat tiles.
|
||||
- Props and decorations should read clearly at 8×8 or 16×16 without zooming.
|
||||
|
||||
## Sheet organization
|
||||
|
||||
- Group related assets together (terrain together, props together,
|
||||
characters together).
|
||||
- Leave consistent spacing between assets for clean cropping.
|
||||
- Label sections visually through grouping, not text.
|
||||
|
||||
## Cohesion checks (before finishing a sheet)
|
||||
|
||||
- All colors appear in the PICO-8 palette.
|
||||
- Perspective does not flip mid-sheet.
|
||||
- Lighting direction is consistent.
|
||||
- Edge treatment (dither, outline, anti-aliasing style) matches across
|
||||
assets.
|
||||
- No orphan tiles that don't connect to any neighbor.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
- Mixing 8-bit and 16-bit detail levels.
|
||||
- Using outline style on some tiles but not others.
|
||||
- Forgetting the palette constraint — colors outside the 16 PICO-8 colors.
|
||||
- Inconsistent tile edge connections (a grass tile that only connects to one
|
||||
dirt variant).
|
||||
@@ -115,6 +115,64 @@ macOS mini) as quadlets inside its podman machine — not on Dankserver.
|
||||
- [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack
|
||||
- [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices
|
||||
|
||||
## Skill Distillation (2026-10-05)
|
||||
|
||||
The managed skill library distilled into the vault, one reference page per
|
||||
skill-cluster. See [log.md](log.md) for the wave plan and results.
|
||||
|
||||
**Operations:**
|
||||
|
||||
- [[Remote Job Runner]] — detached long-job execution with sentinel files
|
||||
- [[Payload and Disk Reclaim]] — disk reclaim: measure, classify, prove,
|
||||
stage, ask; plus purge-and-refresh
|
||||
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
|
||||
the pasta port gate
|
||||
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
|
||||
stop-first, pre-commit iteration, incident-window reconstruction, WoL
|
||||
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
|
||||
posture, whole-fleet audit, bulk redaction
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
|
||||
GUI-hang diagnosis over ssh
|
||||
|
||||
**Architecture:**
|
||||
|
||||
- [[Stale Signal Audit]] — classifying green-but-dead and down-but-fine
|
||||
signals
|
||||
- [[Quadlet Lifecycle]] — quadlet units, drift, crash-loops, compose
|
||||
cutover
|
||||
- [[Deploy State Triage]] — revision markers, pending lockouts, deploy
|
||||
interruptions, shared-checkout hygiene
|
||||
|
||||
**Infrastructure:**
|
||||
|
||||
- [[Observability Provisioning]] — Prometheus/Grafana config as code
|
||||
- [[Homepage Status Probes]] — the status page's checker, vantage rules,
|
||||
and triage
|
||||
|
||||
**LLM cross-cutting (full index: [LLM/index.md](LLM/index.md)):**
|
||||
|
||||
- [[GGUF Compatibility]] · [[Llama Cpp Serving Sizing]] ·
|
||||
[[Llama Cpp Multi-Device Splits]] · [[PCIe Link and Atomics]] ·
|
||||
[[RDNA4 Container Serving]] · [[RDNA4 MoE Expert Offload VRAM Budget]] ·
|
||||
[[vLLM Boot Forensics and Config Interrogation]] ·
|
||||
[[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] ·
|
||||
[[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] ·
|
||||
[[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] ·
|
||||
[[FreeToken Builds]]
|
||||
|
||||
**Agent:**
|
||||
|
||||
- [[OMP Harness]] — compaction arithmetic, per-model samplers, preamble
|
||||
audits
|
||||
- [[Agent Docs and Edits]] — doc-structure audits, hashline edit recovery
|
||||
- [[Command Center Dotfiles]] — the ~/.omp repo-as-deployment
|
||||
|
||||
**Desktop / Research:**
|
||||
|
||||
- [[Noctalia and Niri]] — the desktop shell, palette recolor, template
|
||||
rendering, rotation pinning, iCX3 RGB
|
||||
- [[PICO-8 Asset Standards]] — cohesive PICO-8 asset sheets
|
||||
|
||||
## LLM Fleet
|
||||
|
||||
The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
# Bundle Change Log
|
||||
|
||||
|
||||
## 2026-10-05
|
||||
- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
|
||||
dankserver.** Built FreeToken at PR head `77a7201` on the host into the
|
||||
@@ -26,7 +27,45 @@
|
||||
Dankclaw Prometheus llm-fleet scrape of :7770 reads down until
|
||||
repointed. Rollback: tag `frank-primary-vllm-shapeG-20261003`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
## 2026-10-05
|
||||
- **Skill library distilled into the vault (waves 1–7).** All ~173 managed
|
||||
skills read in full and merged, one polished reference page per
|
||||
skill-cluster:
|
||||
- **LLM cross-cutting (16 pages, `LLM/`):** GGUF Compatibility,
|
||||
Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and
|
||||
Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget,
|
||||
vLLM Boot Forensics and Config Interrogation, vLLM Porting and Compile
|
||||
Triage, LLM Bench Windows and Measurement, LLM Performance Diagnosis,
|
||||
Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split
|
||||
Triage, FreeToken Builds, plus deltas folded into the retired runbooks
|
||||
(GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline,
|
||||
FLM pack registration, ik hybrid envelope, frank-medium split
|
||||
rebalance, frank-primary llama.cpp GGUF preflight, batch-ubatch split
|
||||
section, bench-window D-state/stale-read notes, second-endpoint
|
||||
bring-up in Model Swap).
|
||||
- **Architecture (3):** Stale Signal Audit, Quadlet Lifecycle, Deploy
|
||||
State Triage.
|
||||
- **Operations (6 + index):** Remote Job Runner, Payload and Disk
|
||||
Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit
|
||||
and Posture, SSH and Remote Access.
|
||||
- **Infrastructure (2 + updates):** Observability Provisioning, Homepage
|
||||
Status Probes; updates to Authentik (guard-chain flip recipe, `ak
|
||||
shell` mechanics, user-provisioning playbook, outpost netns probe),
|
||||
Anubis (guard-chain layer diagnosis), LAN-DNS (per-client resolver
|
||||
drop-in + browser DoH bypass, Porkbun zone ops), Swap Configuration
|
||||
(filesystem/rename caveats), OpenWebUI (introspection, verification
|
||||
battery, ComfyUI imagegen), Hermes (config-home drift, live config
|
||||
edits, auxiliary routing, password rotation — the pre-existing page was
|
||||
preserved and extended).
|
||||
- **Host pages:** FrankBook-Pro ARR-Stack filled with the API-version
|
||||
split, schema-first probing, auth-posture method, auth-config trap, and
|
||||
the qBittorrent E2E verification procedure added to [[FrankBook-Pro/qbittorrent]].
|
||||
- **Desktop/Agent/Research:** Noctalia and Niri (config layers, remote
|
||||
IPC, palette recolor, offline template render, rotation pinning, iCX3
|
||||
RGB), Agent/ (OMP Harness, Agent Docs and Edits, Command Center
|
||||
Dotfiles, index), PICO-8 Asset Standards.
|
||||
- Index updated: `LLM/index.md` gained a Cross-Cutting References
|
||||
section; Operations/ and Agent/ got index pages. AGENTS.md now points
|
||||
at the new sections. Link checker + full test suite green.
|
||||
- **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright
|
||||
project.** The `11-admin` walk included the six `*arr`/qBittorrent admin
|
||||
links that terminate on 192.168.0.112, so every deploy gate died there when
|
||||
|
||||
Reference in New Issue
Block a user