diff --git a/AGENTS.md b/AGENTS.md index 60314c0..6647568 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,7 +40,11 @@ the [runbook index](#runbook-index). The full swap history lives in - `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve scripts, systemd units, env tunables, bench harnesses - `old/` — retired LLM scripts (reference only) -- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format) +- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format). + The managed skill library is distilled there one page per skill-cluster: + `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`, + `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service + pages (`serverAppConfigs-docs/log.md` carries the wave plan and results) - `tests/` — bats suite + Playwright e2e ## Git Workflow diff --git a/serverAppConfigs-docs/Agent/Agent Docs and Edits.md b/serverAppConfigs-docs/Agent/Agent Docs and Edits.md new file mode 100644 index 0000000..fdc9484 --- /dev/null +++ b/serverAppConfigs-docs/Agent/Agent Docs and Edits.md @@ -0,0 +1,111 @@ +--- +type: reference +title: Agent Docs and Edits +description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each. +tags: [agents, docs, markdown, hashline, edit-recovery] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Agent Docs and Edits + +## Doc-structure audit (after parallel agent edits) + +Use after one or more subagents have edited markdown docs (OKF/Obsidian +vault with `---` frontmatter), before committing. Link checkers and +completeness seams validate *references*, not *structure* — agents reliably +produce structurally broken markdown that still links cleanly. + +Damage catalogue: + +1. Banner/prose prepended **above** the opening `---` → the frontmatter is + no longer frontmatter (title/tags/description become body text). +2. A rewrite **appended** instead of merged → duplicate headings and a stray + second `---` mid-file. +3. Literal model artifacts in the body: `<|channel>}...`, + `+- `, `- - `. +4. Wrapped bullets re-emitted with a `- ` prefix on every physical line. +5. Line-range replacement that dropped a sentence tail → dangling fragment, + often mid-parenthesis. + +Checks (fast, run all): + +```bash +head -3 # must be `---` if the page has frontmatter +grep -c '^---$' # exactly 2 (opener + closer) +grep -n '^## ' | sort -t: -k2 | uniq -f1 -d # duplicate sections +grep -rnE '^\+- |^- - |<\|' # artifacts +python3 scripts/check-vault-links.py +./tests/run.sh # docs completeness seam etc. +``` + +Repair order (matters): + +1. If the file was destroyed rather than edited: stash the agent version to + `.AGENTTRASH//`, then `git checkout HEAD -- ` and + re-apply only the intended changes by hand. +2. If it merely duplicated: **salvage unique content first** — cut the + one-of-a-kind section, paste it into the copy you keep, THEN delete the + duplicate ranges. Never delete first. +3. Re-add missing sentence tails (insert before the orphaned fragment), + keeping list continuation lines indented. +4. Re-run all checks above. + +Gotchas: agents can write AFTER they appear finished (job settles, file +changes) — re-verify structure after the last background job reports, not +before. Editing an agent's already-damaged file with fresh line numbers is +safe, but its snapshot tag changes on every edit — re-read before each +subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so +nothing moved there reaches the repo — that is intended (temporary hold, not +archive). + +## Hashline edit recovery + +Failure mode: multi-op edits resolve every `PUT N.=M:` / `CUT N.=M:` index +against the **original** file numbering, and the post-apply diff view +**elides lines** (it shows an anchor window, not the whole file). Two traps +follow: + +1. A second edit call built from a diff-view snapshot uses line numbers from + a *partial* view. Lines outside the window are invisible but still + counted. +2. If the first apply was itself wrong, the file shifted. The diff view + tells you the damage, but not the true offsets. + +Result: the repair pastes fresh text **beside** stale text instead of over +it. + +Procedure: + +1. **Never carry line numbers across calls.** Re-read the target range for a + fresh tag before every subsequent edit. A read is cheap; a duplicated + paragraph costs a commit and a revert. +2. **One op per call when the previous apply was imperfect.** Fixed offsets + are only trustworthy when every preceding apply landed exactly as + specified. +3. **Re-read after every apply**, and verify the whole affected region — the + diff view is a sample, not proof. Check the *boundaries*: the line before + the edit and the line after it. Damage shows up as a repeated sentence + straddling the seam. +4. **Repair by CUT, not by re-PUT.** Once you know the exact stale lines + from a fresh read, delete them. Do not try to PUT a corrected superset + over a range whose extent you are guessing at. +5. Verify with a content assertion, not a diff review: `grep -c` for the + duplicated sentence, plus a syntax/compile check on any touched script. + +Same rule for any imperative `edit` tool with an old/new match: the matched +text must be the **entire** text being replaced. Matching a bare fragment +inside a paragraph splices the replacement into the middle of the sentence +and silently corrupts the file. Re-read the file afterwards to confirm. + +### Anchored in-page links + +Renaming a Markdown heading breaks `[[#Old Heading]]` anchors. When a +section is retitled, grep for the old heading text and fix every anchor in +the same pass — doc link-checkers frequently do not validate `#fragment` +targets, so a green suite will not catch a dangling one. + +## Related + +- [[OMP Harness]] — the session context these repairs protect +- [[Command Center Dotfiles]] — the repo these edits commit to +- [[Testing Policy]] — the seams that catch broken vault structure diff --git a/serverAppConfigs-docs/Agent/Command Center Dotfiles.md b/serverAppConfigs-docs/Agent/Command Center Dotfiles.md new file mode 100644 index 0000000..de95761 --- /dev/null +++ b/serverAppConfigs-docs/Agent/Command Center Dotfiles.md @@ -0,0 +1,87 @@ +--- +type: reference +title: Command Center Dotfiles +description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas. +tags: [dotfiles, omp, deploy, rsync, blankpad] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Command Center Dotfiles + +Repo `~/.omp` is itself the deployment: packages are home-relative trees +under `dotfiles//`; `scripts/deploy.sh` symlinks live files into the +repo checkout. `git pull` + `deploy.sh` is the whole propagation story. + +## Transport decision (agreed with Frank, executed 2026-09-09) + +- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to + `@:/home//.omp/`, NOT clone-from-Gitea — `main` can be + ahead of `origin/main`; push is a user-only trigger, so clone would ship + stale state. +- Deploy runs on-target as the target user: `bash + /home//.omp/scripts/deploy.sh` (resolves the repo via `git -C + $(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be + present at the target path — full-replica transport guarantees this). + +## Pre-overwrite audit (MANDATORY before first sync on a non-blank host) + +Diff every file that `deploy.sh --check` reports as DRIFT against the repo +copy BEFORE letting quarantine-and-replace run. Host-specific divergence +belongs in an override package, not in quarantine: + +1. Pull divergent live files from target + (`ssh ... 'cat ' > local-staged-copy`). +2. Secret-audit staged copies (grep token/secret/password/key shapes) + before anything enters git. +3. Compose `dotfiles/zz-/` on the source machine from those live + copies; commit to repo first. +4. Ship the package dir via rsync, then run deploy sync — override deploys + LAST and win over base packages. + +## deploy.sh mechanics (verified on blankpad) + +- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it + first, always. +- Default mode = sync: adopt identicals silently, link new, + quarantine-and-replace divergents into `.quarantine///` + (gitignored; never rm). +- **Override-target bug (fixed cc2c652):** `zz-` packages nest + base-package dirs internally (`zz-blankpad/niri/.config/...`); target + derivation must strip the nested package segment, else overrides deploy to + phantom `$HOME/niri/.config/...` while real targets keep pointing at + canonical copies (visible symptom: check shows 0 actions but + `readlink ~/.zshrc` points at base `zsh/.zshrc`). +- systemd units auto-enable only when ExecStart binary resolves on that + host; dconf packages load via `dconf load /`, never symlinks. +- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once + post-transfer even before first sync. + +## ssh/sshpass transport gotchas (all bit live) + +- Resolve the target via mDNS FQDN (`avahi-resolve -4 -n .local`); a + bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or + firewall default-drop, not auth failure — needs a local/console fix on + target. +- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; an env passed + via the tool's env parameter did not propagate to the sshpass child + process. +- The remote login shell may be **fish**: never send bash syntax + (`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash -c + "..."` or ship a script. See [[SSH and Remote Access]]. +- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A + here-doc attached to the ssh command is consumed by `cat` — the intended + script body becomes the file content, silently. Verify shipped bytes with + an md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat` + drained it (false DIVERGED). + +## Verification of a completed deploy + +Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped += idempotent steady state. Spot-check `readlink` on override files points +into `dotfiles/zz-/`, not base packages. + +## Related + +- [[SSH and Remote Access]] — the transport mechanics this deploy rides +- [[Noctalia and Niri]] — the desktop packages this repo ships +- [[Deploy State Triage]] — the serverAppConfig deploy flow (different repo) diff --git a/serverAppConfigs-docs/Agent/OMP Harness.md b/serverAppConfigs-docs/Agent/OMP Harness.md new file mode 100644 index 0000000..2486790 --- /dev/null +++ b/serverAppConfigs-docs/Agent/OMP Harness.md @@ -0,0 +1,262 @@ +--- +type: reference +title: OMP Harness +description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean. +tags: [omp, harness, compaction, models-yml, context, samplers] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# OMP Harness + +The harness reaches the fleet through `~/.omp/agent/models.yml` (model +registry; single source of truth — there is no copy on the servers) and +`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in +this file is repo-managed; edits are harness-side only. + +## Compaction chain arithmetic (verify before ANY window change) + +When a fleet endpoint's context grows past its harness-declared window, OMP +compacts by sending the conversation to a designated target model. Before +changing any `contextWindow`/`maxTokens`, verify the chain arithmetically — +a target whose window is smaller than the source's trigger point silently +breaks compaction. + +Where the contract lives: + +- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL), + `methodOrder`, `thresholdPercent` (70), idle settings +- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`, + `compactionModel` references +- Implementation: omp plugin `dist/core/compaction/compaction.js` + +The arithmetic (the coupled contract every endpoint's maxTokens comment +encodes): + +1. **Source trigger** = `thresholdPercent × source contextWindow` + (0.70 × 200,704 = 140,492 for frank-primary on the current shape) +2. **Output budget** = capped at the target model's `maxTokens` + (32,768 for frank-small; 36,864 frank-primary; 57,344 frank-large — the + largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it) +3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot + window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤ + 200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700 + (frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-small). + +Procedure (recompute from the LIVE files every time — windows rotate with +every swap; the numbers above are the 2026-10-05 state): + +1. Read the `config.yml` compaction block + `models.yml` entries for source + and target. Every endpoint's `contextWindow` comment carries its own + arithmetic — read them. +2. Read the target checkpoint's `max_position_embeddings` (native window) — + the served window may be a concurrency contract, not the native limit + (`--parallel N` DIVIDES the pool; the per-slot number is the contract). +3. If native ≥ trigger + budget, reslice the pinned pool: + `MAX_RUNNING × CONTEXT_LENGTH ≤ MAX_TOTAL_TOKENS` (the pin exists because + the runtime's autofit leaves too little for deferred loads + prefill + workspace; check `nvidia-smi` headroom — expect ~0.8 GB workspace margin). +4. Changes: host env (CONTEXT_LENGTH-style env tunables) via git → + `./tests/run.sh` → deploy; harness `models.yml` directly (no repo copy). +5. Verify: `/health` 200, boot log's pool line, then a long-prompt probe at + ≥ trigger height (the compaction-fit guarantee), then concurrent requests. + +Known reconciliation trap: comments in models.yml have described compaction +as running "on the dankserver endpoint" while `remoteEndpoint` pointed at +dankclaw — the comment/config disagreed. The comments now name the real +target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is +truth; fix comments in the same change.** `maxTokens` never changes for a +reslice — it caps output; `contextWindow` caps input. Boot-shape OOM +laddering is a different problem ([[Llama Cpp Serving Sizing]]). + +## Per-model sampler config (`compat.extraBody`) + +`models.yml` has **no per-model sampler keys**. Sampling is global +(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`, +`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is +`compat.extraBody`: + +**Why it works:** `applyOpenAIExtraBody` runs **last** in the +openai-completions builder — after `applyChatCompletionsCompatPolicy` — via +`Object.assign(params, extraBody)`. So extraBody overrides global settings +and compat-policy output. + +```yaml +compat: + reasoningEffortMap: + max: xhigh + extraBody: + temperature: 1.0 + top_p: 0.95 + top_k: 20 + min_p: 0.0 + presence_penalty: 0.0 + repetition_penalty: 1.0 + preserve_thinking: true +``` + +Use **wire names**, not OMP camelCase. Two traps: + +1. **`preserve_thinking` is not user-settable as a compat key.** + `qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the + *auto-detected* `thinkingFormat` + local baseUrl, **before** + `applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in + models.yml does **not** enable it. Custom-named models (`frank-large`) + match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`. +2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy + also sets it — `Object.assign` is shallow and replaces the whole object, + silently dropping the `reasoning_effort` kwarg the compat policy set. + Top-level `preserve_thinking` is honoured by halogen. (frank-primary's + FreeToken entry currently carries a `chat_template_kwargs` extraBody with + a comment asking to verify-and-prune — the exact trap to check at that + endpoint's next swap.) + +**Gotcha: silent greedy decode.** halogen has **no server-side sampler +defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the +endpoint decodes greedy while looking configured. This is why frank-large's +entry carries the recipe on the wire and a comment saying "do not drop it". + +**Reasoning-effort folding.** Qwen3.8's template accepts only +`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything +else ("Unexpected reasoning effort high"). Every endpoint's +`reasoningEffortMap` folds all harness levels onto that set; a role whose +level is unmapped fails on its FIRST request, and the failure names Jinja, +not the model. + +### Verify the exact wire body (echo-capture) + +Don't trust the YAML — capture what OMP actually emits: + +1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and + returns a canned `chat.completion` (include `reasoning_content` so + thinking paths exercise). +2. Copy the **real** models.yml into a throwaway profile with only the + baseUrl swapped: + ```bash + sed 's|http://:7770/v1|http://127.0.0.1:18099/v1|' \ + ~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml + ``` +3. Run and inspect: + ```bash + omp --profile wiretest --model "/" -p "hi" \ + --no-tools --no-session --no-extensions --no-skills --no-rules + jq 'del(.messages, .tools)' last-body.json + ``` + Confirm every intended field is present with the right value. + +**Verify a field is *honoured*, not just accepted** — unknown fields often +return 200 anyway. Prove behavior with a **prompt-token delta**: replay an +assistant turn carrying a long `reasoning_content`, then compare +`usage.prompt_tokens` with the flag true vs false. A large delta means the +field changed template rendering; zero delta means it was ignored. + +Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`), +never `rm`. + +## Static-preamble audit ("the context feels bloated") + +What loads statically every session (measure each, don't guess): + +1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k + tokens — historically the largest block). +2. Skill index — only names+descriptions load; bodies are on-demand. Sum + description bytes across all SKILL.md frontmatter: + ```bash + python3 - <<'EOF' + import glob, re, os + for root in [os.path.expanduser("~/.omp/agent/managed-skills"), + os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]: + n=tot=0 + for sk in glob.glob(root+"/*/SKILL.md"): + fm=re.search(r'^---.*?---', open(sk, encoding="utf-8", errors="replace").read(), re.S) + d=re.search(r'description:\s*(.*(?:\n(?:\s{2,}.*)*)?)', fm.group(0) if fm else "") + n+=1; tot+=len(d.group(1)) if d else 0 + print(root, n, tot) + EOF + ``` + Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user + skills. +3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name + 'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest + MEMORY.md. +4. Harness/system prompt + tool schemas + CLI persona: fixed cost, ~20 KB; + not prunable. + +Known rot patterns: + +- Near-duplicate auto-captures of the same lesson (three "authentik ORM" + entries, the freetoken-build pair, the three job-runner skills) — dedupe + via `manage_skill delete`/`update`. +- Corrupted captures containing stray CJK characters (`一团ly`, `等方法`) + mid-sentence — a known defect of the auto-capture path; delete the corrupt + copy, keep the clean one. +- Stale host/service names in consolidated memory (pre-rename + `frank-default-large`) — the repo's AGENTS.md topology table is ground + truth. + +Fix strategy: AGENTS.md prose (measured runbook narratives) → move to +`serverAppConfigs-docs/LLM/` runbooks, keep laws + tables in AGENTS.md (the +repo's own docs-tree rule mandates this). Managed skills live in +`~/.omp/agent/managed-skills` (safe to edit); user-authored skills in +`~/.omp/agent/skills` are NEVER touched by tooling. Memory store edits: +stage removed captures to trash first (`~/.AGENTTRASH/YYYYMMDD-topic`). + +Baseline to beat (2026-10-05): ~35–40k tokens of preamble — AGENTS.md ~18k, +skill index ~10k, lessons ~3.5k, harness ~6k. Proposed reductions: −12k +(AGENTS.md slimming), −3k (skill dedupe). + +## Managed-skill library hygiene + +Use when auditing, pruning, or consolidating `~/.omp/agent/managed-skills` +(the auto-captured skill store; the full index is injected into EVERY +session's context, so dead or duplicate entries are a permanent tax). + +The static-context budget (measured 2026-10-05): + +| Block | Size | Notes | +|---|---|---| +| `AGENTS.md` (repo) | 72,179 B ≈ 18k tok | biggest single item | +| Skill index (~173 skills) | ~40 KB ≈ 10k tok | names+descriptions only | +| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures | +| Harness system prompt + tool schemas | ~23 KB | fixed | + +Audit procedure: + +1. Build an inventory: per skill, name, mtime, body word count, description, + first 12 headings (one pass, out to a file; read in pages). +2. Group into: retired-runtime (host/service gone; its runbook already lives + in `serverAppConfigs-docs/LLM/`), verbatim-duplicate pairs/triples (same + lesson captured twice — often the SAME DAY), same-lesson families + (N skills = one procedure), keep. +3. Read full bodies of every merge candidate before proposing a merge; + description alone proves overlap only for exact dupes. +4. Present A/B/C/D verdicts for user approval. NEVER execute before the + discussion concludes — the user gates execution explicitly. +5. On approval: stage everything to trash first + (`~/.AGENTTRASH/YYYYMMDD-/`), never rm. After merging: recreate + the surviving skill via `manage_skill` update with body = superset, + delete absorbed ones. + +Known rot classes (this library, 2026-10-05): + +- Same-day duplicate captures (freetoken-build-dankclaw vs + freetoken-dankclaw-build; the three job-runner skills). +- CUDA-skill vs its self-declared HIP twin (pcie-link-capability-probe vs + pcie-link-probe-hip-rdna4) → merge. +- Era ghosts: skills naming retired runtimes (freetoken, lucebox, r9v/ + radiance, old halogen-flash-server image, ik_llama, GLM-5.3 experiment). +- Skills contradicting user standing orders (live-bench-dashboard vs the + 2026-09-29 delete-dashboards order) — flag for user, don't assume. +- Memory store: corrupted captures with CJK noise from broken ssh commands; + stale host/service renames. + +Sizing verdicts that worked: bench family → 2 skills (harness vs +discipline), not one mega-skill; keep profiler/kernel-time attribution +standalone; fold go/no-go arithmetic skills into their probe skill +(heterogeneous-accelerator-split-triage → PCIe probe). + +## Related + +- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely +- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides +- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit +- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes diff --git a/serverAppConfigs-docs/Agent/index.md b/serverAppConfigs-docs/Agent/index.md new file mode 100644 index 0000000..d042b3c --- /dev/null +++ b/serverAppConfigs-docs/Agent/index.md @@ -0,0 +1,23 @@ +--- +type: index +description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home. +tags: [agent, index] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Agent + +Pages about the agent's own tooling and context — how omp is configured, how +docs get edited safely, and how the agent's home is deployed. + +- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via + `compat.extraBody`, preamble and skill-library audits +- [[Agent Docs and Edits]] — doc-structure audits after parallel edits; + hashline edit recovery +- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync + transport, override packages + +## Related + +- [[Testing Policy]] — the seams these pages' checks ride +- [LLM](../LLM/index.md) — the fleet these agents operate diff --git a/serverAppConfigs-docs/Architecture/Deploy State Triage.md b/serverAppConfigs-docs/Architecture/Deploy State Triage.md new file mode 100644 index 0000000..46a479c --- /dev/null +++ b/serverAppConfigs-docs/Architecture/Deploy State Triage.md @@ -0,0 +1,151 @@ +--- +type: reference +title: Deploy State Triage +description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints. +tags: [deployment, deploy-state, systemd, podman, triage] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Deploy State Triage + +The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/` +on each target (rootless, user `dank`, format exactly two lines +`revision=` / `status=deployed|pending`, written atomically via tmp file ++ `mv`). It is written ONLY by a `deploy.sh` dispatch. + +## 1. Stale marker vs live container + +**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or +a restart after the checkout ff-pulled to a newer revision — starts the +container from the CURRENT `scripts/.env` but does NOT advance the +marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT +"behavior pending". + +**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service +whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of +a 100+ GiB payload — and frank-primary (dankserver:7770) is also the +harness's own model backend, so a blind deploy is a self-inflicted outage for +possibly zero behavioral change. + +**Procedure (read-only first):** + +1. Committed state: `git log --oneline main..` empty ⇒ merged; the + operational delta is usually one `scripts/.env`. +2. Probe the target (ssh as `dank`, passwordless sudo present): + - `cat ~/.local/state/serverAppConfigs/deploy-state/` + - `git -C ~/serverAppConfigs rev-parse HEAD` + - `systemctl is-active/is-enabled `; + `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:/health` + - the on-host env file already carries the new keys (it landed via the + ff-pull). +3. **Decisive — read the LIVE container, not the marker.** Use `podman` as + `dank` (rootless store; bare `docker` is NOT the shim → "No such object"; + `sudo podman` inspects an empty store): + - `podman inspect --format '{{.State.StartedAt}}'` — booted AFTER + the pull ⇒ it read the new env + - `podman inspect --format '{{.Args}}' | tr ' ' '\n' | grep -i ` + - engine env: `podman exec sh -c 'for f in /proc/[0-9]*/environ; do + tr "\0" "\n" <$f 2>/dev/null; done | grep -i '` + - confirm the launcher maps each tunable to its flag + (`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 → + --enable-prefix-caching`. + +**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is +live — **do NOT restart**; reconcile the marker (below). Only if the live +container lacks the flag/env (or predates the pull) is a real +`deploy.sh --service ` warranted. + +**Reconcile (no restart)** — mirror the deployer's atomic idiom: + +```bash +S=$HOME/.local/state/serverAppConfigs/deploy-state +H=$(git -C ~/serverAppConfigs rev-parse HEAD) +t=$(mktemp "$S/..XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/" +``` + +Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs +`tests/e2e/.env.local`; expect the documented `11-admin` red if the +FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` + +a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression). +Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy +no-op). + +**Anti-patterns:** trusting a commit message ("X deployed") or the marker +alone — both are bookkeeping; the container's argv/env is ground truth. +Reading container state with `sudo podman` / bare `docker`. Restarting a +healthy heavy endpoint to satisfy a stale marker. + +## 2. Pending lockout + +Symptom: `deploy.sh ` fails or silently stops with +`service is not deployed on ` and the `[3/3]` e2e gate never +appears. + +1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail` + reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then + `echo $?`. +2. Inspect the state store (no sudo): + `ssh 'cat ~/.local/state/serverAppConfigs/deploy-state/'`. + A `status=pending` entry for a service absent from `service_is_allowed` + in `scripts/deploy-remote.sh` = retired-service lockout. +3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes + not-allowed pending entries to + `~/.AGENTTRASH/-retired-deploy-state/` and the run + continues. Verify the WARN line names the service and the state file is + gone. +4. On a host without the fix: move the stale entry to + `~/.AGENTTRASH/-retired-deploy-state/` manually (never `rm`), re-run. +5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes; + `deploy-state/` no longer holds the retired name. + +Related contracts: the superseded seam +(`deployment_scoping.bats` "remote runner recognizes a pending service at +the same revision") now asserts prune-and-continue (exit 0), not exit 2; +`deployment_flow.bats` extracts `add_pending_services` via sed and +HOME-overrides the trash path. **When retiring a service going forward:** +delete or reconcile its deploy-state entry as part of the retirement commit — +the prune is a safety net, not the intended cleanup path. + +## 3. Interrupted or raced deploys + +The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh` +(runs on the target over ssh). **Killing the dev-side job does NOT kill the +host-side one** — two `deploy-remote.sh` processes can end up managing the +same unit, one deploying a revision you already superseded. Check: +`pgrep -af "deploy-remote"` on the target, read the revision argument on +each, kill stale ones by explicit PID. + +A killed deploy leaves behind: + +- the unit **stopped/failed** (`Restart=always` does NOT fire after an + explicit stop — start it yourself); +- the state marker **pending** (pre-deploy or in-flight revision); +- the e2e gate un-run (the deployer runs it last). + +Reconcile the marker ONLY after the service itself is verified healthy — a +marker that runs ahead of reality is worse than one that lags (the planner +uses it to decide what is pending). Then close the loop: + +1. Verify against MEASURED, config-specific numbers (KV capacity line, arena + slots, argv flags, `NRestarts`), not just health 200. +2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout + that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree` + does NOT have it; copy it in first). +3. If the killed revision was restoring a known-good shape, confirm the + numbers match the last measured-good boot of that shape — equality is the + proof, not health alone. + +Boot failures must be read from the journal, not `podman logs`: serve scripts +`podman rm -f` the container on every start (previous attempt lost), and +`podman logs` on a container still *Stopping* returns the OLD configuration's +numbers — which looks exactly like your change having no effect. Key on +container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the +cause from `journalctl --since "N min ago" | grep -a "\["`. +See [[vLLM Boot Forensics and Config Interrogation]] for the full capture +procedure. + +## Related + +- [[Deployment Flow]] — the deploy pipeline itself +- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis +- [[vLLM Boot Forensics and Config Interrogation]] — journal capture diff --git a/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md b/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md new file mode 100644 index 0000000..c5f1b24 --- /dev/null +++ b/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md @@ -0,0 +1,209 @@ +--- +type: reference +title: Quadlet Lifecycle +description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth. +tags: [podman, quadlet, systemd, boot-persistence, compose, drift] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Quadlet Lifecycle + +Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed, +working container MUST have a self-healing, boot-persistent Quadlet unit — +`.container` file in the repo (`//quadlet/`), installed by the +deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on. +A container that runs only from compose dies silently at reboot. This page +is the operating manual for that lifecycle. + +## 1. Ownership audit — find the containers that will not survive a reboot + +```bash +for c in $(podman ps --format '{{.Names}}'); do + printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')" +done +``` + +- **label present** → the unit owns it; persistence = `[Install]` + `loginctl + show-user -p Linger` = yes. +- **label absent, compose service exists** → the classic gap: dies at reboot. + Check the publish address while there — compose forms often bind the + wildcard (an unauthenticated UI/API on every interface). +- **label present AND a compose service with the same name** → two owners; a + stray `podman compose up` can create a duplicate that fights the unit for + the name and the port. Remove the compose service, leave an explanatory + comment (repo convention). + +## 2. Cutover: compose service → Quadlet unit + +Checklist (precedents to copy: +`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`, +`dankclaw/observability/quadlet/`): + +1. Unit file at `//quadlet/.container`, mirroring compose + line for line: `Image=` (`localhost/` prefix for locally built images + + `Pull=never`), `ContainerName=` (Caddy and scrape configs address + containers by name), `Network=`, `User=0:0` when the image's own user + cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved), + `Exec=` for compose `command:`, `PublishPort=`, health keys, + `[Service] Restart=always` + `RestartSec=10`, `[Install] + WantedBy=default.target`. **Never `systemctl --user enable` the generated + unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated` + is healthy; explicit enable fails with "transient or generated"). +2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs + it (Caddy reaches containers by name over the shared network; the wildcard + publish is a guard-chain bypass). +3. Handler in `scripts/deploy-remote.sh`: `install_quadlet ` → + `systemctl --user restart .service` → `wait_for_health`; retire any + legacy hand-written unit into `~/.AGENTTRASH/` (a leftover + `~/.config/systemd/user/.service` that shells out to + `podman compose up` collides BY NAME with the generated unit); + `podman rm -f ` so the unit's container can take the name. +4. Planner: map `//**` → service in `scripts/deploy-plan.py` + AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies + ordered. Add a `test-changed` case above the generic + `*/quadlet/*.container` fallback. +5. Retire the compose entry in the SAME change — two owners is how a stale + duplicate comes back. +6. Seam test `tests/_quadlet.bats`: boot persistence, restart policy, no + wildcard publish, container name matches Caddy/scrape references, health + command, deployer installs+restarts+waits, never `rm`, never `enable`. + Migrate any older assertion that read the compose file for devices/flags. +7. Deploy `--service `; prove: `is-active`/`is-enabled` = + active/generated, container `healthy`, endpoint answers, + `list-dependencies default.target` shows the unit, off-host probes on + previously published ports now refuse, and the container no longer carries + `io.podman.compose.project`. + +**Traps that cost real time:** + +- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has + no shell → the check fails forever. `--health-cmd` won't take exec form + either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build + with `--format docker` — the default OCI format has no HEALTHCHECK field + and drops it with only a warning. +- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a + gluetun env carrying `HOME=/root`; systemd sets HOME for the service and + the unit dies with `stat /root/.config: permission denied`. Use a + per-service env under `/home/dank/Containers//env`, seeded by the + handler with a single-variable filter, validated so a missing secret fails + loudly. +- `Network=host` exporters scraped by containerized Prometheus must bind the + host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1` + for the homepage probe — two `--web.listen-address` flags, never the + wildcard. +- alloy 1.x refuses the `ui` config block; the unit must pass + `--server.http.listen-addr=0.0.0.0:12345` before its port can be published. +- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry + no supplementary groups). +- When the last service leaves a compose file, the compose-validity seam + fails (requires ≥1 service): keep the file as comments and relax the seam + deliberately, or delete the file and fix the planner's compose-diff path — + decide with the owner, the deploy tooling reads it. + +## 3. Drift / crash-loop diagnosis (5 minutes, in order) + +Symptom: crash-looping or perpetually unhealthy container whose DEFINITION +looks right; or a service quietly down for days because a restart policy kept +something running. Cause: the running container was created from an OLDER +definition — a compose/unit edit does not touch an existing container, and +the restart policy faithfully resurrects it forever. + +1. **Who started it:** `podman inspect --format 'unit={{index + .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} + created={{.Created}}'`; compare `created` with when the definition changed + (`git log -p --follow `). Created-before-the-edit is the signature. +2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth. + `{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even + for containers started with `--device`/`--group-add` — a false negative + that sends you hunting the wrong root cause. +3. **What the process says:** `podman logs | grep -iE 'error|cannot|Runtime'` + (a capability error like `No HIP GPUs are available` beside a missing + device flag confirms drift). Count the damage: `{{.RestartCount}}` — one + instance reached 9,146 restart attempts on stale config. + +Quickest unblock: `podman compose up -d --force-recreate `. Durable fix += the §2 cutover. **Verify the capability, not the status:** `podman exec + python3 -c "import torch; print(torch.cuda.is_available())"`; +`ss -ltn | grep ` for the actual bind addresses. ComfyUI incident: +2026-09-16, stale device-less container, 9,146 restarts. History repeats: +2026-08-26 Caddy (no unit), 2026-09-11 five observability services. + +## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28) + +- **Hard-fail keys** — the whole unit silently fails to generate + (`systemctl --user cat ` shows "could not be found"): any + dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key … + in group 'Container'`); `Command=` is unsupported, use `Exec=`. +- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on + spaces into the healthcheck argv; a quoted probe + `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` → + command-not-found → container reads **unhealthy while the DB is up**. + HealthCmd must contain only space-free tokens + (`HealthCmd=pg_isready -d authentik -U authentik`). If health reads + unhealthy but logs look fine, compare the baked + `{{.Config.Healthcheck.Test}}` array against the intended command before + believing the status. +- **Probe a new key before shipping:** drop a throwaway `probe.container`, + `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i + "unsupported key"`, `systemctl --user cat probe.service` (shows the + generated ExecStart verbatim), then remove + reload. + +## 5. Rootless Quadlet log retrieval + +Step 0 — establish ownership: `systemctl --user status .service` +(works ⇒ rootless user unit); `loginctl show-user -p Linger`. If +`--user` knows the unit, EVERY command runs as that user with NO sudo. + +| Need | Command | +|---|---| +| Current instance tail/follow | `podman logs --tail 200 / -f ` | +| Across restarts/recreates | `journalctl --user -u .service -n 200 --no-pager` | +| Time-bounded / crash-only | `--since "14:30"` / `-p warning` | +| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` | + +Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on +recreation** — the journal is the only durable record. Traps: +`sudo podman logs` → "no such container" (root's store does not contain +rootless containers; same for `sudo podman ps`); a system-level +`journalctl -u ` returns nothing for a user unit; during first-start +weight downloads health reports unhealthy BY DESIGN (check the payload file +is growing). Verbose vendor switches are a config change — through git and +deploy.sh, never a live env poke. + +## 6. Compose bring-up with ground truth (rootless hosts) + +1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user — + check every namespace (`sudo podman ps -a`) and native units + (`systemctl list-units`) before declaring a service absent. Compare + compose mount sources against the real filesystem — compose files name + phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at + `/mnt/entropy`). Fix compose through git first, deploy second. +2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`) + or creation fails `statfs …: no such file or directory`. +3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll + `{{.State.Health.Status}}`, then the rest. Fresh containers answer + 000/refused for the first seconds — poll. +4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token + request for nonexistent/private repos ("manifest unknown"/token len 0 = + repo or tag does not exist, not auth). Query the registry API + (`ghcr.io/token?scope=repository:/:pull` → + `/v2///tags/list`). +5. **Read the interface from inside the image** before guessing flags/envs: + `podman exec cat /z/cpp.sh` etc. HTTP 500 with a styled error page + means RUNNING-but-misconfigured, not down. Copyparty ac ground truth: + entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV + flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none + configured it serves cwd read/write for everyone (an unauthenticated + world-writable share). +6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a + mangled pattern is not absence — re-verify the target exists. +7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns + (host 1000 == container 0): drop the user line, run as container-root. + +## Related + +- [[Deployment Flow]] — the deployer that installs these units +- [[Stale Signal Audit]] — the audit that catches drift's victims +- [[Container Network Namespaces]] — reachability between containers +- [[Dankclaw/ComfyUI]] — the 9,146-restart incident +- [[Infrastructure/Observability]] — same class, twice diff --git a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md new file mode 100644 index 0000000..7347336 --- /dev/null +++ b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md @@ -0,0 +1,144 @@ +--- +type: reference +title: Stale Signal Audit +description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time. +tags: [testing, observability, audit, signals, gates] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Stale Signal Audit + +Run after any change and before trusting a green gate. The productive question +is **which signal here cannot possibly be correct?** Every pass on this repo +(4 of 4) found real defects — including one in the rule file every future +agent reads first. + +## The surfaces, and what each vantage point can reach + +| Surface | Producer's vantage point | Can reach | +|---|---|---| +| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses | +| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only | +| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN | + +Anything else — another host's loopback, a container-internal port never +published — is unreachable no matter how healthy the service is. The failure +mode is always the same shape: a signal that **cannot** succeed by +construction, which teaches the operator to ignore the surface it lives on. + +## 1. Permanently-red probes (reachability) + +For every monitor/probe/alert target ask: *can this checker physically reach +that endpoint?* Seen classes: a card probing another host's container-internal +port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a +`/metrics` that is disabled (Gitea 404). + +- A scrape cannot express "is this HTTP service answering?" — the right probe + is the **blackbox exporter** in probe mode (which also gives guarded vhosts + a real uptime signal, since Prometheus cannot solve an Anubis challenge). +- **Fix in the direction of honesty, never of silence:** unreachable target → + mark the homepage card `wip: true` with the reason in `description` (the + checker skips `wip`; `wip` is the existing convention for "real service, + not probeable"), or repoint the job at something reachable. NEVER open a + database port on the LAN to make a badge green, and never delete a scrape + job — an absent scrape and a healthy one are indistinguishable. +- Alert rules must not depend on always-red signals — scope them to services + that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for + exactly that reason). +- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from + `connection refused` to `unsupported Content-Type "text/html"` means the + service came up and only the scrape is wrong. Read the error, not just + up/down. + +## 2. Vacuous assertions (highest severity — the gate is already dead) + +A seam that greps a file passes *by absence* once the file is retired: +`grep` exits 2 on a missing path and `!` inverts it. + +```bash +grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50 +[ -f "$path" ] || echo "MISSING: a seam may be grepping this" +``` + +Fix shape — one place asserts the invariant, each local site keeps a live +check: + +```bats +@test "no compose file competes with the quadlet units" { + local cf="$REPO_ROOT//compose.yaml" + [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; } +} +# at the per-service site: still fails if the entry returns +[ ! -f "$cf" ] || ! grep -qE '^\s+:' "$cf" +``` + +**Prove teeth:** recreate the offending file +(`printf 'services:\n :\n image: x\n' > `), run the seams, +confirm RED, then move the file to trash. A re-armed check you have never +seen fail is still unverified. + +## 3. Gates that never run + +Playwright specs in `tests/e2e/` are matched by **number prefix** in +`playwright.config.ts`; an unregistered spec silently never runs and the +suite count stays the same. Count tests before/after; +`tests/e2e_scaffold.bats` fails on an unregistered spec by design. + +## 4. Validate the checker before acting on it + +A checker's output is a hypothesis about another tool's semantics. Modelling +Obsidian with naive text matching produced 45 "broken links" of which 3 were +real — the rest were case-sensitivity (a `qBittorrent` link to +`qbittorrent.md`), path-qualified resolution (a path-qualified +`folder/Note` link resolves by path SUFFIX, not +stem), and `%20` URL-encoding in markdown links. Fix the checker first, +re-derive the report, only then edit documents. The result lives in +`scripts/check-vault-links.py` with those three rules in its docstring, +gated by the `no unresolved links in the docs vault` seam. Same class: a +`find … | head -1` selector that reads the wrong file; a truncated listing +that makes you "fix" a file that exists. + +## 5. Prose invalidated by your own change + +After retiring a file or adding a feature, grep for EVERY reference, not just +code: + +```bash +grep -rn "" --include="*" . | grep -v '^./.git/' +grep -rn "no \|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" . +``` + +This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long +after it was restored with GPU access; a unit said "there are no alerting +rules" hours after five were added; a docs page documented three scrape jobs +as working when they could never report up. **Clean cutover includes prose** — +docs, unit docstrings, and the repo's rule file (the most expensive place to +be stale). + +## Order and verification + +1. Enumerate the surfaces (status board, metrics targets, alert rules, test + gates, docs/rule files, dashboard panels). +2. For each, ask what the signal CAN express, not what it says. +3. Validate any checker before editing the things it flags. +4. Fix at the root; never weaken an assertion to make it pass. +5. Gate the class so it cannot return (a seam), then prove the seam fails on + the defect. + +Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each +Prometheus panel expression against the API (0 series on a panel that should +show data is a defect; 0 `ALERTS` when nothing fires is not); read the +deployed `/srv/homepage/status.json` and triage every `state != up` entry +against reality; query `/api/v1/targets?state=active` and print +`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`. +Report per finding: the surface, why the signal could not have been correct, +the fix, and the red/green evidence — flag any regression you introduced +yourself in the same breath (one was found this way: a tightened bind killed +a reverse-proxied vhost while every gate stayed green, because the spec +covering that hostname did not exist). + +## Related + +- [[Testing Policy]] — the seam inventory this audit walks +- [[Observability]] — the scrape/probe surfaces it audits +- [[Quadlet Lifecycle]] — the ownership labels the probes read diff --git a/serverAppConfigs-docs/Dankclaw/Hermes.md b/serverAppConfigs-docs/Dankclaw/Hermes.md index aa70d4d..26de54c 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes.md @@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto The npx package cache lives in the container layer, so a recreation (not just a restart) pays one first-call npm download; subsequent calls are cached. +## Config-home drift ("cannot connect" to models that answer curl) + +`podman exec printenv HERMES_HOME` for each hermes container. If +the webui reads the image default (`/opt/data`) while the gateway reads +`/data`, the webui runs the image-default template config (`provider: auto` → +openrouter.ai with no key = connect failure). + +The recreation trap is the same one the volumes table shows: image `VOLUME` +points get a fresh anonymous volume on every container recreation +(`podman run --replace --rm`) — edits inside such a mount are lost on +restart/redeploy. Compare volume ids before/after a restart to confirm. + +Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state +root), deploy service-scoped. Verify with the app's own code, not guesses: + +```python +from api import config as C; C.resolve_model_provider("") +# -> (model_id, provider, base_url); bare ids map to the owning provider by +# scanning the providers: dict models allow-list; @provider:model forces one. +hermes_cli.providers.resolve_user_provider(name, cfg["providers"]) +# -> ProviderDef with real base_url; accepts api/url/base_url keys. +``` + +Named providers (`providers:` dict) are a catalog + connection source; +`custom:` connections resolve only via `custom_providers[]` entries. If +neither holds the provider name, the request has no endpoint and fails to +connect. Probe the endpoint with the **exact served model id** — the retired +FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200 +with `{"error":"Failed to load model!"}` for unknown tags; status-code +checks miss this class entirely. + +## Live config edits + +Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------` +uid 109999; all reads/edits need `sudo`). The repo file +`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag +live — always diff the live file first; never treat the example as truth**. + +1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD` + (in-dir `.AGENTTRASH/` also exists for removals). +2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in + the live file are tool-generated — preserve them. +3. **Sweep stale references** across the whole data dir: + `sudo grep -rln "" /home/dank/Containers/hermes/data + --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are + historical — leave them. +4. **Derived caches:** `webui/models_cache.json` is keyed by a config + fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it + is stale. Let it regenerate lazily or trash it — never hand-edit. +5. **Restart the model-consuming units** (they parse config at startup): + `systemctl --user restart hermes-webui.service hermes.service`. + `hermes-dashboard.service` is unaffected by routing changes. +6. **Verify behaviorally**: one tiny completion per endpoint with the exact + served id — POST `/v1/chat/completions` and read the body, never just the + status code (endpoint names rotate; see the LLM knowledge base). +7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the + live shape, redacting secrets (`password_hash`, phone account stay + live-only), then `./tests/run.sh` before commit. + +`provider: custom` + `base_url` entries route by base_url and send the model +id verbatim; named providers resolve via their `api:` URL. **Renaming a +served alias anywhere (serve scripts, env ALIAS) MUST propagate to this +config in the same change** or Hermes requests 404 on nonexistent models — +this exact gap caused the 2026-09-05 outage. + +## Auxiliary routing against local backends + +When a Hermes feature that calls an auxiliary task (e.g. title generation) +fails against a local endpoint, the usual cause is a **request shape the +local runtime does not implement**, not a routing mistake. + +Diagnose with the real call path — run the actual function inside the +container with the app's own constants (the probe cannot drift from +production): + +```bash +base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \ + 'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \ + podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \ + podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py' +``` + +Copy the file AFTER any restart (`--replace --rm` discards the container +layer). The CLI at `/opt/hermes/hermes` uses system python and dies +`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`. + +```python +import sys; sys.path.insert(0, "/opt/hermes") +from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text +from agent.auxiliary_client import call_llm +resp = call_llm(task="title_generation", messages=[...], max_tokens=64, + temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT}) +``` + +What actually breaks, and the fix: + +- **`response_format` (json_object / json_schema) is a hard 400 on every + local backend here** — neither FreeToken nor halogen implements constrained + decoding. Config cannot override it: `call_llm` merges the task's + `extra_body` first and then `update()`s the caller's, so the hardcoded one + wins. Only temperature and max_tokens have reactive retry paths. +- **Fix: move that one task to the Responses transport**, which never + forwards `response_format`: + + ```yaml + auxiliary: + title_generation: + provider: dankserver-small # names track the live config; diff first + model: frank-small + api_mode: codex_responses # real per-task key + extra_body: + reasoning: + effort: none # mandatory, see below + ``` + +- **Disable thinking for any cap-limited task.** The adapter omits reasoning + only when config says `reasoning.enabled: false`. The + `reasoning_effort: none` *shorthand* produces exactly that and leaves the + server default (thinking ON) — the 64-token title budget gets spent on the + think: empty content, `finish_reason: length`. The explicit + `extra_body.reasoning.effort: none` instead sends + `{"effort":"none","summary":"auto"}`, which the local `/v1/responses` + honors. +- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no + proxy needed for the alternate transport. + +Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs — +it must pass from config alone (that is the production path); keep the live +file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes +config change is docs+example only, no service deploy needed. + +Gotchas: `auxiliary..enabled: false` is checked BEFORE +`apply_instant_title`, so disabling a broken title task also removes the +derived title — it does not degrade gracefully the way a failure does. A +local call can also fail at the model level: a 200 with an error-text body, +or a chatty non-title answer. Always read the raw content, not the status. + +## Webui password rotation ("Hermex") + +Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON +`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL) +on match, 401 on miss. No username field. `/health` answers 200 +unauthenticated and `/` answers 302 — **never verify auth via those**; probe +login with a deliberately wrong password (expect 401) and the real one +(expect 200). Details on the webui surface: [[Hermes-WebUI]]. + +1. Run the whole probe as a heredoc'd bash script + (`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles + inline `${…}` even inside single-quoted ssh args; never interpolate curl + `-w %{http_code}` inline. +2. Update BOTH env files with + `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=/"`: + - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile) + - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the + quadlet's `[Container] EnvironmentFile=` is what the container reads at + create; rotating only the root file leaves the running container serving + the old secret. +3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads + env; no deploy needed if the unit file itself didn't change). +4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN + client at `http://192.168.0.143:8787` (webui publishes on the LAN + interface since commit `4443897`; off-network ingress is SSH tunnel only — + user `tunnel`, port **22**, key-only auth, forward remote end + localhost:8787). + +Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not +see rootless containers; run `podman exec hermes-webui printenv +HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode +600). If rotating because a secret leaked into transcripts/logs, rotate in +place via a script that fetches the live value from the running container +(`printenv`) instead of transcribing it. + ## See Also - [[Hermes-Dashboard]] — the LAN browser management UI diff --git a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md index 85cffb0..2f1e0f3 100644 --- a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md +++ b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md @@ -64,6 +64,49 @@ The Ollama connection is disabled (`ollama.enable=false`). > repository. Model swaps, context sizing, and spec tuning happen there. See > the [LLM knowledge base](../LLM/index.md). +## Container introspection (DB and backend python) + +- Working dir for imports: `/app/backend`; package name `open_webui` + (underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set + BEFORE importing any `open_webui` module (import-time hard requirement); + the secret lives at `/app/backend/.webui_secret_key`. +- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3` + module (`podman exec open-webui python3 -c "import sqlite3; + conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config + keys use parameterized LIKE queries (avoids shell quoting). +- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` — + `Models.get_all_models()` is a coroutine (await it); model objects are + pydantic `ModelModel` instances (attribute access, not `.get()`), with NO + `status`/`enabled` attribute — use `is_active` from the DB. +- **API auth for verification:** `create_token` lives in + `open_webui.utils.auth` (NOT `open_webui.utils.utils`); + `get_verified_user` requires an `id` claim (not just `sub`); + `SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the + container: + + ``` + podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \ + python -c "from open_webui.utils.auth import create_token; \ + print(create_token({\"id\":\"\",\"sub\":\"\",\"email\":\"…\",\"role\":\"admin\"}))"' + ``` + + (admin UID from the `user` table). + +## Verification battery + +1. `GET /api/v1/models/list?limit=100` → returns EXACTLY the intended + models. Response key is `items`, not `data`. +2. `POST /openai/chat/completions` per model with a trivial prompt ("Reply + with exactly one word: pong") → each returns content `pong`. Reasoning + models need `max_tokens>=100` — the thinking block eats the budget. +3. A 500 "Server Connection Error" → `podman logs open-webui` for + `ClientConnectorError`: a network-reachability problem (host LAN IP vs + `host.containers.internal`), not a model problem. + +**Durability:** the model table + connections are DB-only (not git-managed); +a fresh data volume loses them. For reproducibility, add an idempotent +first-start seed script or document the setup in the service doc. + ## Web search Chat web search is served by [[SearXNG]] on this host, wired in the quadlet @@ -147,6 +190,23 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start `podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`. +## Image generation (ComfyUI) + +Image generation rides the local [[ComfyUI]] integration, not the chat +picker — ComfyUI's models never appear in the chat model picker (that is +chat LLMs only), so ComfyUI needs its own image-generation connection +configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and +serves `:8188`. + +Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the +route to `POST /api/v1/images/generations`; the `comfyui` engine is still +upstream) with `Authorization: Bearer ` and body +`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response +carries base64 image data or a URL. + +Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts +on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad. + ## Retired instances - The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind. diff --git a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md index 9d181cc..c542859 100644 --- a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md +++ b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md @@ -117,6 +117,18 @@ frank-large `health=200` throughout. the model is loaded. Trust the engine's `host memory left for everything else` line, not `free`. +- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles + directly. btrfs needs `chattr +C` (NOCOW), no compression, and no + snapshots in the swapfile path — check `df -T` before planning, or + `mkswap` will reject the file. +- **Renaming an *active* swapfile works (inode-based) but leaves a stale + path in `/proc/swaps`** — swap it off first, then rename, then `swapon` + (which is exactly what the step-3 consolidation does). Migration is + observable while it runs: in the 2026-09-12 run `swapoff` of 4 GiB took + 65 s and used-RAM rose only ~3 GiB (pages touched during migration), the + rest relocating on disk. If used-RAM climbs toward the whole swapped + amount, let it finish rather than aborting. + ## Trash The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at diff --git a/serverAppConfigs-docs/Desktop/Noctalia and Niri.md b/serverAppConfigs-docs/Desktop/Noctalia and Niri.md new file mode 100644 index 0000000..3fafdd4 --- /dev/null +++ b/serverAppConfigs-docs/Desktop/Noctalia and Niri.md @@ -0,0 +1,303 @@ +--- +type: reference +title: Noctalia and Niri +description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning. +tags: [desktop, noctalia, niri, blankpad, workstation, ghostty] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Noctalia and Niri + +Noctalia (`/usr/bin/noctalia`) is the desktop shell on niri, running on both +the workstation and blankpad. Since the 2026-09-10 unification both run the +**bare `noctalia` v5 binary** (niri `spawn-at-startup "noctalia"`); the old +Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is +retired on both hosts. + +## Config layers + +- Hand-edit declarative layer: `~/.config/noctalia/.toml` (e.g. + `config.toml`). Main config: `~/.config/noctalia/config.toml`; the niri + config includes `noctalia.kdl` for display settings. +- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and + **overrides declarative values for the same keys — never hand-edit it**; + UI/IPC writes land there. +- Main reference: `noctalia-reference` facts — IPC `noctalia msg ` + replaced the `qs -c noctalia-shell ipc call` wrapper; app templates live at + `/usr/share/noctalia/assets/templates/` (gtk, ghostty, alacritty, hyprland, + sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango, + umbriel, foot, emacs, kitty); palette generation `noctalia theme [image]`. + +## Remote IPC over ssh (the D-Bus trap) + +`noctalia msg ` resolves the running instance via the D-Bus session +bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain +ssh exec prints `error: noctalia is not running` even when the shell IS +running — do not conclude "shell dead" from this error. Working remote +procedure (runs in the systemd user-manager context where the bus address +exists): + +``` +systemd-run --user --pipe --wait noctalia msg [args] +``` + +Verified: `msg status` returns live JSON state; unknown panel ids return an +error listing all valid panel ids (clipboard, control-center, launcher, +polkit, session, setup-wizard, test, tray-drawer, wallpaper). + +Verified v5 msg verb grammar (keybinds-safe): `panel-toggle `, +`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`, +`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`, +`media next|previous|playPause` (graceful no-op, rc 0, with no active +player), `brightness-up [target] [step]`, `brightness-down`, +`settings-open [context]`, `settings-toggle [context]`, `config-reload`, +`wallpaper-get [connector]`, `wallpaper-set `, `wallpaper-next`, +`color-scheme-get|set`, `theme-mode-get`, `templates-apply`, +`screenshot-fullscreen`. + +**Wallpaper directory semantics:** empty/unset `[wallpaper] directory` +resolves to XDG Pictures via user-dirs.dirs; an explicit directory key +overrides. Fleet default pattern: point `directory` at the repo-tracked +wallpapers dir so git-tracked wallpapers serve the shell without copying +files. After writing config.toml over ssh the file watcher hot-reloads +automatically; `config-reload` via systemd-run forces it immediately. Note: +the *current* wallpaper stays the built-in default until the next rotation +or an explicit `wallpaper-next`; `wallpaper-get` then reporting a path +inside the configured directory is the confirmation the config took effect. + +**Lock chain (unified):** swayidle-lock.service ExecStart on both hosts: +`/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock' +before-sleep '…' after-resume '…'`. `msg` resolves via the session bus, so +no WAYLAND_DISPLAY pinning is needed in unit context. + +## Custom-palette recolor (without restarting the session) + +Why custom, never the community cache: +`~/.local/state/noctalia/community-palettes/*.json` is a **sync cache** — +`.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches +from `https://api.noctalia.dev/palettes`, so edits there get reverted. +Custom palettes live at `~/.config/noctalia/palettes/.json` +(`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never +synced. Missing/invalid file → silent fallback to the builtin palette. + +Palette JSON shape = the community file's shape: `dark`/`light` maps of +`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`, +`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block +(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`, +`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme +-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the +*generator* format, not this one. Copy an existing community JSON and mutate +with `jq`. + +Procedure: + +1. Read the live scheme first — never assume: `noctalia msg color-scheme-get` + (e.g. `community Gruber Darker`); `noctalia msg theme-mode-get`. +2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml + /tmp/noctalia-settings-backup-$(date +%s).toml`. +3. Build the palette from the current one (jq keeps structure): + + ```sh + mkdir -p ~/.config/noctalia/palettes + jq '.dark.mPrimary = "#f43841" + | .dark.terminal.normal.yellow = "#f43841" + | .dark.terminal.bright.yellow = "#ff4f58" + | .dark.terminal.foreground = "#f43841"' \ + "$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \ + > ~/.config/noctalia/palettes/"Gruber Red.json" + ``` + + Prefer a hex already present in the palette (its own `mError` red) so the + theme stays internally consistent. Set BOTH `normal` and `bright` + variants, and both `dark` and `light` if the user toggles modes. +4. Switch (persists `theme.source = custom` + `theme.custom_palette`): + `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with + `color-scheme-get`. +5. Re-render every configured template: `noctalia msg templates-apply`. + Regenerates `~/.config/ghostty/themes/noctalia`, + `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm, + starship, etc. Symlinked-into-dotfiles paths show up as modified files + worth committing. +6. Reload surfaces: compositor — `niri msg action load-config-file` + (`reload-config` is NOT a valid niri subcommand); ghostty — see the trap + below. + +Verification: `ghostty +show-config | grep -E '^(foreground|palette)'` +(proves the theme file resolves — note it prints only the delta from +defaults; cross-check with `ghostty +show-config --default` before +concluding a key is unset); `noctalia config validate` → `✓ Config is +valid`; visual via `noctalia msg screenshot-fullscreen` (saves to +`~/Pictures/screenshot_.png` and ALSO overwrites the clipboard with the +PNG — warn the user); `grep -c ~/.config/noctalia/palettes/*.json` +→ 0 for no leftovers. + +Track it in the dotfiles repo (`~/.omp`, the command-center — +`scripts/deploy.sh` is a stow-equivalent over `dotfiles//` trees): +mirror the shape for the new palette so `git pull` + `deploy.sh` is the +whole propagation story — move the JSON into +`~/.omp/dotfiles/noctalia/.config/noctalia/palettes/` and symlink it back +into `~/.config/noctalia/palettes/`; re-set the custom palette (re-reads +through the symlink). Do the link by hand for one file instead of running +`deploy.sh` — sync mode quarantines and replaces every divergent live file. + +Traps: + +- **NEVER `kill -USR1` ghostty to reload.** It does not reload config on + this fleet — it kills/respawns the terminal (took out Frank's window + 2026-09-13). Correct: the default keybind `ctrl+shift+,` = `reload_config` + (per `ghostty +show-config --default`). Existing windows keep old colors + until reloaded; new windows inherit immediately. +- Palette hex values are NOT a proxy for what the user sees: in + `Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change + the yellow" means the accent too, not just terminal index 3. +- Saturating terminal text makes ANSI 1/9 indistinguishable from default + text (grep highlights flatten) — flag it and offer a lighter red. +- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats + white-on-color here, ~3.9:1 — keep `mOnPrimary` dark). +- Revert = `noctalia msg color-scheme-set community ""`; the + community cache is untouched by the whole flow. +- Generated theme files symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0` + `noctalia.css`, ghostty theme) show up as tracked modifications after + `templates-apply`; commit them together with the new palette JSON so every + machine gets the same recolor. + +## Offline template render (when templates-apply never ran) + +Failure signature: niri `failed to read included config from +~/.config/niri/noctalia.kdl: No such file or directory` — the desktop runs +but with DEFAULT config (niri 26.04 falls back, does not exit): no custom +keybinds/rules, no noctalia autostart, default output placement. ghostty: +`theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`. +alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing. + +Root cause pattern: upstream dotfiles commits theme switches (niri include, +ghostty `theme = noctalia`, alacritty import) that depend on noctalia +TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from +the local palette) and are intentionally NOT in the repo; on any host where +`templates-apply` never ran, they don't exist. + +Fix (offline, works while a session runs) — built-in templates live at +`/usr/share/noctalia/assets/templates/` (declared in `builtin.toml`); render +each needed one with the live palette: + +```bash +# niri include (plain host-local file; do NOT symlink into repo — regenerated per host) +noctalia theme --theme-json ~/.config/noctalia/colors.json \ + --render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl" +mkdir -p ~/.config/ghostty/themes && \ +noctalia theme --theme-json ~/.config/noctalia/colors.json \ + --render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia" +mkdir -p ~/.config/alacritty/themes && \ +noctalia theme --theme-json ~/.config/noctalia/colors.json \ + --render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml" +``` + +All template tokens (primary, surface, error, primary_container, …) resolve +from the flat 16-key colors.json; container tokens are derived, never blank. + +Verify: `niri validate` → rc 0; `ghostty +validate-config` → rc 0; next +login journal shows `loaded config from config.kdl` with NO parse WARN, +custom output placement (`niri msg outputs` vs display.kdl), noctalia +autostarted (`pgrep noctalia`). + +Traps: **`noctalia msg templates-apply` on 5.0.1 returns "ok" but applies +NOTHING** (this build's opt-in config schema does not act on it) — do not +trust its exit code, use the explicit `--render` calls above. Niri config +errors at boot are NOT fatal — the compositor keeps running on defaults, +which reads as "some settings silently stopped working", not a crash. +Regenerate after any palette change; the rendered files are static +snapshots. + +## Niri output rotation and pinning + +Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel). + +**Ground truth first:** `niri msg outputs` (connector names, current mode, +scale, transform, logical pos/size) and `niri --version`. Output **names are +connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings — +the EDID strings printed before them are decoration. On blankpad the live +config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`, +and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the +repo path (that is what gets committed); check with `ls -ld`. + +**Transform direction is counter-clockwise** (`niri msg output +transform --help`): `90` = content rotated 90° CCW ⇒ the panel's **top +bezel sits on the LEFT**; `270` = top bezel RIGHT; `180` = upside down; +`flipped-90`/`flipped-270` = mirrored variants. `90` vs `270` is a +physical-stand fact no tool can read (portable monitors expose no +orientation in EDID/MCCS) — pick one, apply, and hand the user the one-liner +to flip; don't stall on the question. + +Two traps when editing the config: + +1. **An `output "X" { … }` block resets every property you don't restate to + its default.** Pinning position on `eDP-1` without repeating `scale 1.25` + silently drops the laptop panel to scale 1.0. Restate `scale` (and + `mode`) in every block you touch. +2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning + only the externals shoved `eDP-1` from `0,0` to `3936,0`. Pin every + output's position to keep the left-to-right order. Positions are logical + pixels at each output's own scale; x-offsets chain (eDP-1 logical width + 1536 → DP-2 at 1536 → +1200 portrait width → HDMI-A-1 at 2736): + +```kdl +output "DP-2" { scale 1 transform "90" position x=1536 y=0 } +output "HDMI-A-1" { scale 1 transform "90" position x=2736 y=0 } +output "eDP-1" { scale 1.25 position x=0 y=0 } // MUST restate scale +``` + +Apply and verify: + +```bash +niri validate -c ~/.config/niri/config.kdl # exit 0 +niri msg action load-config-file # live reload; NOT reload-config +sleep 1; niri msg outputs | grep -E '^Output|Transform|Logical|Scale' +``` + +Success looks like `Logical size: 1200x1920` (swapped) plus +`Transform: 90° counter-clockwise` on each rotated panel, positions matching +the intended chain. One-off rotation without touching config (temporary): +`niri msg output DP-2 transform 90`. Commit via the dotfiles repo +(`~/.omp`, `main`; push to the local Gitea permitted): `git pull --ff-only` +→ add the display.kdl → commit → push. Pitfalls: `reload-config` is not a +niri subcommand; never SIGUSR1 a terminal to "reload" (niri config reloads +do not touch ghostty themes); no man page for niri on this host — `niri + --help` and `/usr/share/doc/niri/default-config.kdl` are the docs. + +## GPU RGB over I2C (EVGA iCX3) + +Turn RGB on/off (or read it back) on an EVGA FTW3 / iCX3-controller GPU +(the 3090 on dankclaw) using the GPU's own nvidia i2c bus. No display +engine, no X, no daemon — the mode is EEPROM-saved and survives reboot. + +- **Find the controller:** the nvidia i2c driver exposes the card's lighting + ports as `/dev/i2c-N`; on the 3090 FTW3 that is ports 1,5,6,7,8 → + i2c-1..i2c-5, and only i2c-1 carries a live slave (NVIDIA port 1). Probe + each bus. The iCX3 MCU answers at address **0x2D**. `/dev/i2c-*` are + `root:i2c 660` — run with `sudo` (dank has passwordless sudo). +- **Protocol (raw I2C_RDWR, NOT the SMBus ioctls)** — write the register + byte first, then the payload: + - enable `0xB2` write `04 C6 EB EA 15` (required once before mode writes) + - mode `0xC0` write `09 m0 m1 m2 m3 .. .. .. .. SYNC`; m0..m3 are the four + zone modes, `MODE_OFF = 0x00`; OFF = `09 00 00 00 00 FF FF FF FF 00` + - save `0x90` write `04 9E EB 00 90` (commit to EEPROM — the + reboot-persistence step) + - fw `0xB1` block read (firmware version) + - Read the mode back: write reg `0xC0`, then read 16 bytes; bytes 1..4 are + the zone modes (0x00 = OFF), bytes 6..9 the per-zone LED counts. +- **Verify persistence:** after a reboot, read the mode — all four zone + modes `0x00` means the EEPROM save held (confirmed 2026-09-19 post-reboot). +- **Tool:** a ~90-line I2C_RDWR reader/writer lived at `/tmp/evga_raw.c` + (`evga_raw mode|off|save|enable|fw`) — `/tmp` is wiped by reboot, so + recreate it if needed; OpenRGB's `EVGAGPUv3Controller.cpp` carries the + same protocol plus the full colour API. +- Gotchas: raw I2C_RDWR only (the iCX3 answers the raw form, not the SMBus + ioctls); the `enable` write must precede the first mode write in a + session; without the `save` write the mode is RAM-only and resets on power + cycle. + +## Related + +- [[Command Center Dotfiles]] — the stow-equivalent deploy this page links into +- [[Hermes]] — the agent that drove the CDP daemon alongside this desktop +- [[GGUF Compatibility]] — unrelated but shares the "read the file, not the name" lesson diff --git a/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md b/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md index 2b76699..d4147bc 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md +++ b/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md @@ -5,3 +5,115 @@ description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Reada tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver] timestamp: 2026-09-04T00:00:00Z --- + +# *arr Stack + +The *arr automation stack runs on **FrankBook-Pro** (the Mac mini) as quadlets +inside its podman machine; all egress rides the gluetun WireGuard netns. The +per-service pages ([[Prowlarr]], [[Sonarr]], [[Radarr]], [[Lidarr]], +[[Readarr]], [[qBittorrent]]) carry deployment facts; this page holds the +cross-cutting API and auth-posture procedures. + +## API version split (memorize before any *arr HTTP call) + +- Sonarr/Radarr: `/api/v3/…`; Lidarr/Readarr/Prowlarr: `/api/v1/…`. +- Prowlarr is port **9696** — 8090 is Jackett. +- **Empty body = endpoint absent**, not a failure: probe endpoints by status + code first; v1 has no `/api/v1/profile` endpoint at all. +- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object + (GET it, patch `settings`, PUT the whole object back). An array body fails + with `Path: $ | BytePositionInLine: 1`. +- API keys live in each container's config: + `podman exec grep -o "[^<]*" /config/config.xml | cut -d">" -f2`. + They are local-only secrets — they stop being secret the moment a vhost + serves the API unauthenticated. + +## Schema-first probing (validators are free documentation) + +Send the FULL schema template object with fields filled; v1 validators reject +partials, and each error names the next required field — peel validators one +live call at a time instead of guessing payload shapes: + +- `'Fields' must not be empty` → send a full schema copy, not `fields:[]`. +- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is + required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys. +- `DefaultMetadataProfileId must be > 0` → fetch real ids from + `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`); + `GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠ + profiles. +- `targetType null` → configContract not registered in that build (this + Prowlarr build has no Prowlarr-plugin indexer; integration is + Torznab-family only). +- `testingPackageId` in a PUT body downgrades validation failures to + warnings. +- `syncLevel` is enum `ApplicationSyncLevel` — omit it or use the int; a + string is rejected. + +**Verification rules:** `POST /downloadclient` returns the COLLECTION, not +the created object — verify via GET. Test endpoints return verdicts in the +BODY at HTTP 200 (`{}` = pass, a list = fail) — judge from bodies, not +status codes. Command POSTs (`MoviesSearch`, `ApplicationSync`) run +server-side and respond only when the work finishes: curl `-m` timeouts +(HTTP 000) during a search are NOT failures — poll queue/history afterward +instead of re-firing; warm small commands (`RefreshMonitoredDownloads`) +answer in ~20 ms, so a hang means live work in progress, not a dead API. + +**DB as arbiter:** when endpoint shapes are opaque, stop the container and +edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly, +then start. The indexers table columns are +Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the +enable flag column is `Enable` (not `EnableRss`). Prowlarr's real application +settings values live in `prowlarr.db` → `Applications.Settings` JSON (the +`/api/v1/applications` GET responses carry `fields: []`). + +## Auth posture probe (ground truth is behavior, not config flags) + +`AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in +`config.xml` is ambiguous on paper; on the wire it means a login screen with +**zero users** — first-account-wins. + +1. **Confirm which host runs the service before probing** (`podman ps` on + the target, not the box you ssh'd into first — the arrs live on + FrankBook-Pro). +2. Probe an API endpoint that bypasses the SPA shell, locally and through + any vhost: `curl -s -o /dev/null -w "%{http_code}" + http://:/api/v3/system/status`. **SPA roots (`/`) always + serve a 200 shell — never conclude "open" or "gated" from `/`.** + 401 = gated; 200 + JSON = API fully open (anyone on the net can drive it + with curl). +3. For config truth, dump the whole file + (`podman exec cat /config/config.xml`), not a piped + `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the + pipeline hides a miss and you conclude "empty field" when the grep itself + failed. +4. Decision framing: when posture = gated-with-zero-users, present it as a + decision menu (None behind the front-door wall vs External/authentikarr), + not a silent change — first-account-wins means whoever logs in first owns + the instance. + +## The `None` + `Enabled` auth-config trap (v1 .NET apps) + +`None` + +`Enabled` is +self-contradictory: the UI Save is permanently disabled, and any save throws +`DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every +page after restart. The valid no-auth representation is **absence of the +`` key**, not the literal string `Disabled`. Fix = +delete the key from `/config/config.xml` +(`sed -i "//d"`), restart the container. Tell-tale: +one v1 app boots clean while siblings 500-sprawl → diff their config.xml +auth blocks; the healthy one shows the correct shape. (Prowlarr runs +`Forms` + `Enabled` healthily — the crash combo is `None`+`Enabled` only.) + +General rules that saved the session these came from: container +`State=running` ≠ app serving — read `podman logs --tail 12 ` before +theorizing; when a UI gate blocks saving, go under it (these apps keep +plaintext XML config on disk — read/edit via `podman exec`). + +## See Also + +- [[Prowlarr]] — indexer feeds and the v2 sync model +- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] — the managers +- [[qBittorrent]] — download client, 4.6.7 pin, E2E verification +- [[FrankBook-Pro]] — the host and its podman machine +- [[Homepage Status Probes]] — how the e2e gate treats this sleeping host diff --git a/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md b/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md index 78eefd5..7cbf6f2 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md +++ b/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md @@ -111,6 +111,90 @@ Restart=always probe one ssh-context out returns the residential IP — a false leak alarm. - Ensure umask is set to `022` so downloaded files are readable by arr apps. +## E2E verification after any pipeline change + +Run after ANY change to the torrent pipeline — WireGuard credential rotation, +qB version change, gluetun reconfig, category re-seeding. Proves the full +grab path (tunnel egress → qB download → category save path → clean +removal). All commands from a dev box via `ssh frankbook-pro`; inside the +Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for +`podman machine ssh`. + +**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112 +(NOT core — `core` is the podman-machine VM user, reached only via +`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped +2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a +low MaxAuthTries: when password-bootstrapping use `-o +PreferredAuthentications=password -o PubkeyAuthentication=no +-o IdentitiesOnly=yes` or the key offers exhaust the attempts before the +password is tried. + +**1. Egress gate — MUST run inside gluetun:** + +```bash +podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json' +``` + +Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config); judge by org, +never country alone. NEVER run this one ssh-context out (bare `wget` on the +Mac host): it silently returns the residential Comcast IP and reads as a +hard-gate leak that does not exist. A residential reading ⇒ re-run inside +the container before diagnosing. + +**2. Login — generic cookie parse.** The QPW fetch is flaky (~1 in N returns +empty); assert length before use — an empty-password run once produced a +Forbidden loop that cost hours: + +```bash +QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2) +[ ${#QPW} -eq 20 ] || exit 1 +SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \ + http://127.0.0.1:8080/api/v2/auth/login \ + | grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1) +curl -b "$SID" … # pass the WHOLE match +``` + +WebUI from the LAN: `http://192.168.0.112:8080` (published on the gluetun +unit). Username `admin`; password = the machine-local `QBIT_WEBUI_PASSWORD` +(20 chars, generated once at provisioning by the deploy handler — never in +the repo). Parse-bug 403s never reach qB as failed logins — no ban accrues; +server-side truth is `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log` +(`WebAPI login success/failure` + source IP). Repeated REAL failures accrue +toward qB's 5-failures/3600 s IP ban. + +**3. Add a legal test torrent.** Blender open-movie torrents (~300 MB, e.g. +Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file +(`-F "file=@…"`) and `category=Finished Torrents`. Never leave a real-looking +grab behind. + +**4. Poll and verify.** Poll `GET /torrents/info` (~20 s): state +`downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading` +at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents` +(qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT +inherit category paths — one more reason for the 4.6.7 pin). Confirm on +disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is +normal for the tunnel. + +**5. Cleanup — delete only your artifact:** + +```bash +curl -s -o /dev/null -w "%{http_code}" -b "$SID" \ + --data-urlencode "hashes=" \ + --data-urlencode "deleteFiles=true" \ + http://127.0.0.1:8080/api/v2/torrents/delete # expect 200 +``` + +Target the known hash literally — the user's real grabs share the directory +and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir +gone, all other entries intact. + +**Version-coupling constraints:** qB is pinned to 4.6.7 (Readarr's EOL +client rejects qB ≥ 5's `204` login at runtime — see the pin rationale +above). Do not unpin while Readarr exists. After ANY qB version flip, +regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k +iterations) — a 5.x-written hash line makes 4.6.7 reject every login. +Rewrite while qB is stopped. + ## See Also - [[ARR-Stack]] — full *arr stack documentation diff --git a/serverAppConfigs-docs/Infrastructure/Anubis.md b/serverAppConfigs-docs/Infrastructure/Anubis.md index 72baabb..0986af2 100644 --- a/serverAppConfigs-docs/Infrastructure/Anubis.md +++ b/serverAppConfigs-docs/Infrastructure/Anubis.md @@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges — distinguish challenge from deny at this release; grep the body for `Access Denied` / `Making sure you` instead. +## Guard-chain layer diagnosis (a user reports a guarded page "broken") + +Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as +"broken / links gone / not loading" and the fault must be localized to +server-side or client-side/session. Chain topology: +`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost +→ Caddy :8017 (@home/@data handlers) → app/static`. + +Layer-by-layer checks (run from the host via ssh): + +1. **Caddy unit + guard containers**: `systemctl --user is-active + caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE + 'anubis|authentik|caddy'` — expect `Up … (healthy)`. +2. **Anubis challenge page validity**: `curl -sk + https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143` + — a healthy challenge contains `"anubis_challenge"` JSON with + `difficulty` and the main JS. +3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep + -iE 'error|exception|traceback|500|provider'` — routine dramatiq task + logs are normal. +4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in + its netns — `curl http://127.0.0.1:8017` from the HOST returns `000` + connection refused, which is EXPECTED, not a fault): + `podman exec caddy sh -c 'wget -q -S -T5 --header "Host: + www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for + `/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in + this container has NO `-H` flag — use `--header "Host: …"`.) +5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json` + parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`). + +Tool/behavior notes: the anubis container is distroless — `podman exec anubis +sh` fails (`sh: executable not found`); inspect via its logs or the served +challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's +podman IP) for ALL clients — rootless-podman published-port SNAT; +clients are indistinguishable by IP and clearance is cookie-based. Do NOT +mistake that for the bug. Homepage render: `app.js` does +`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch +failure the catch sets `#summary` to `status unavailable: ` and the +grid stays EMPTY — which a user reads as "links gone / not loading." A +30-service status.json with a 9-service services.json renders fine +(stateByName lookup). + +**Decision:** if every layer is healthy (challenge valid, outpost clean, +handlers 200, files render), the fault is client-side/session. STOP +thrashing the live system — repeated failed browser challenges share the +masked 10.89.1.22 identity. Get the authenticated view: ask for the exact +symptom (challenge spinning? login loop? empty grid? "status unavailable" +text?) or the authentik login to drive the page in a real browser. + ## Verify live ``` diff --git a/serverAppConfigs-docs/Infrastructure/Authentik.md b/serverAppConfigs-docs/Infrastructure/Authentik.md index df38b5d..d4348bf 100644 --- a/serverAppConfigs-docs/Infrastructure/Authentik.md +++ b/serverAppConfigs-docs/Infrastructure/Authentik.md @@ -342,6 +342,168 @@ The silent re-auth path was proven end-to-end: after deleting `qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage state and passed with no login prompt (2 tests, 2.9 s). +## Guard-chain flip recipe (adding a guarded vhost) + +Move a public vhost from `reverse_proxy :` onto the chain: +client → TLS edge → Anubis :8923 → outpost decision → internal +`http://:8017` listener → Host-matched handler → real upstream. + +**Order (never reorder):** + +1. **Provider first.** Create the outpost ProxyProvider before touching + routes, or authed traffic hits the outpost error page instead of the app. + Fleet convention for `external_host` is the exact browser-facing origin + (the inventory table carries both bare origins and trailing slashes; + routing matches on hostname, verified 2026-09-02). + `internal_host = http://caddy:8017`, all three flows set, and an + Application attached — the two provisioning rules above. +2. **Route flip in ONE commit/deploy**: the vhost becomes + `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and + inside the `http://:8017` listener add `@ host ` + + `handle @ { reverse_proxy }`. Anubis guard set, Caddy TLS + edge, `:8017` routing table, and provider list change together. +3. Deploy via the repo deploy flow (commit first; deploy pushes main itself). + +**Verification (all three):** + +- Unauthenticated probe from a client: `curl -s https:///` → HTTP 200 + challenge page; grep the body for `challenge|not a bot|anubis` (~8 + markers on a live challenge). A 502 here instead of the challenge = chain + mis-wired. +- Post-auth routing proof: grep the LIVE config inside the container — + `podman exec caddy sh -c "grep -n -A3 /etc/caddy/Caddyfile"` — + after the deploy prints its reload line; confirm the reload actually ran, + don't trust exit 0. +- Outpost reachability: `podman exec authentik-server getent hosts caddy` + resolves the compose DNS alias (10.89.1.x) that `internal_host` dials. + +**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP +000 (connection refused) — the post-auth listener lives in the caddy +container's netns; probe inside it or grep the config. Non-browser clients +cannot solve the JS challenge — before flipping, audit service-to-service +traffic: ingestion/push paths must target internal addresses (`loki:3100`), +never a public hostname of a guarded service. Exempt-by-construction +categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP +endpoints, services with their own complete auth — document the exemption +in the vhost comment. + +## Driving the ORM (`ak shell`) + +All authentik object work happens through the server container's ORM shell — +no repo change, no deploy (these providers are NOT repo-managed; capture new +provider+application pairs as a blueprint or setup script to avoid drift). + +```bash +ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p' +from authentik.core.models import User, Group +... +EOF +``` + +- The heredoc IS the program: `ak shell` reads all of stdin and execs it + (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven + one). +- Bootstrap prints JSON log lines, then + `### authentik shell (2026.8.3)` and `229 objects imported automatically…` + — pipe through `sed -n '/229 objects/,$p'` to drop the noise. +- 229 models are pre-imported. Import paths (each wrong guess costs a failed + exec): `User`, `Group`, `Application`, `ApplicationEntitlement`, + `PropertyMapping` → `authentik.core.models`; `ProxyProvider` → + `authentik.providers.proxy.models` (NOT `authentik.outposts.models`); + `LDAPProvider` → `authentik.providers.ldap.models`; + `authentik.core.models.application` does NOT exist as an import path; there + is no `authentik.policies.group` module in this build. +- For live DB truth prefer the postgres container + (`podman exec authentik-postgres psql -U authentik -d authentik -At -c + "select * from …"`) — **query with `select *` and stop guessing column + names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises + FieldError — `type` moved to subclass tables; the outpost `enabled` column + may not exist under that name; provider pk is `oauth2provider_ptr_id`). + +## User provisioning playbook (user-level account) + +There is no `User.create_user` classmethod, and `is_superuser` is NOT a +constructor kwarg (it is a property derived from group membership) — passing +it raises `TypeError: User() got unexpected keyword arguments`: + +```python +from authentik.core.models import User, Group +u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True) +u.set_password("...") # password arrives on stdin only, never argv +u.save() +for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"): + u.groups.add(Group.objects.get(name=g)) +``` + +`path` defaults to `users` (regular). The access map (verified live +2026-10-04): there are NO per-app policy bindings on the proxy providers +(all use the implicit-consent "Authorize Application" flow), so guarded-vhost +access is decided entirely by GROUP MEMBERSHIP consumed downstream — + +- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups` + regex gate) and the homepage admin panel; regular users get only + `category: user` panels from `services.json`. +- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements + "Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the + "Open WebUI roles" property mapping → `roles` claim. Verify live with + `u.app_entitlements(app)`. +- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost + (filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`). +- `Non-Admin Users` = the plain-user marker group (precedents: alex, + qa-e2e). +- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`, + `openwebui_admins`, `authentik Admins`, `authentik Read-only`. +- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty, + Plex, Seerr, Marreta) need no per-user group: any authenticated user + passes the outpost. + +Verify login behaviorally — `authenticate()` is the exact PasswordStage code +path: + +```python +from django.contrib.auth import authenticate +assert authenticate(request=None, username="ridley", password="...") # -> User +assert authenticate(request=None, username="ridley", password="nope") is None +``` + +Look flows up by slug, not name: +`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is +"Welcome to authentik!" — `.get(name=…)` fails). Stage order: +Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no +enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks. + +More field gotchas (each cost a failed exec): `ApplicationEntitlement.application` +can be null (guard with `getattr`); `LDAPProvider` has +`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/ +`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an +`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls` +(use `type(p).__name__`); `User` has no `.status`. + +Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates +every outpost session fleet-wide; purge the stale `proxysession` rows +instead (section above). + +## Probing the outpost from inside the netns + +Rootless podman gives each container its own netns; **host `ss -ltn` cannot +see listeners inside other containers' namespaces** — a missing listener on +the host is an observation artifact, not a fault. Distroless images have no +shell (`podman exec authentik-server sh` fails), so probe by joining the +namespace: + +``` +podman run --rm --network container: --entrypoint /bin/busybox \ + docker.io/library/busybox:latest \ + wget -q -T 4 -t 1 -O /dev/null -S "" 2>&1 | head -8 +``` + +- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings + triggers globbing (`no matches found`). +- Boot transience: immediately after a restart the same probe can return 503 + (configure task pending); poll until the answer changes before concluding. +- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman + version — the field is `.HostConfig.NetworkMode`. + ## Failure modes | Symptom | Cause | Fix | diff --git a/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md b/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md new file mode 100644 index 0000000..ac73c83 --- /dev/null +++ b/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md @@ -0,0 +1,158 @@ +--- +type: reference +title: Homepage Status Probes +description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix). +tags: [homepage, status, probes, e2e, homepage-status] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Homepage Status Probes + +The homepage is a static site — +`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}` +— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in +`scripts/deploy-remote.sh` refuses other targets). A checker +(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes +each card's target and writes `/srv/homepage/status.json` (mode **0644** so +Caddy can serve it — losing the mode 403s the page). + +## The checker's vantage and its three false-down shapes + +The checker probes TCP/HTTP **from the host netns**. It only has three states +— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can +never render honestly and must be dropped or re-pointed, not "fixed". + +1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS + only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` → + card says down while PMS answers `200` over HTTPS. Fix: + `"method":"tcp"` (connect is the honest liveness signal; the public URL + stays HTTP because Caddy re-keys). +2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the + compose→quadlet move moved the dashboard to `9119` and the webui to + `8787`; nothing listened on 8090, so a healthy stack read `down`. +3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net + `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`) + and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are + netns-scoped, so from the host these are unreachable *by construction* — + no probe value makes them honest. Drop those cards. + +A host-vantage probe can only reach a listener **published to the host** +(rootlessport/pasta); unpublished ports are invisible from the host netns. +See [[Container Host Reachability]] for the netns rules. + +## Triage procedure + +1. Read the card's probe target in `shared/homepage/services.json`; note + method (`tcp` vs `http`). +2. Does that port have a **host-visible listener**? + `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container + `podman inspect --format '{{json .NetworkSettings.Networks}} + {{.HostConfig.NetworkMode}}'` and `podman port ` (empty output = + published nothing → structurally unprobeable from the host). +3. Probe from the **correct vantage**: from a bridge-net container the host + is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on + `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`), + which belongs to the default `podman` net (`10.88.0.0/16`). Probe with + bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so + they expand **inside** `podman exec bash -c`. +4. Classify: real outage (container exited — read `State.ExitCode`: 137 = + SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM + killer) vs false down (running + unreachable target). +5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose + only exposure is an internal netns; never invent a probe target. +6. Verify by re-running the checker path, not by eyeballing: after deploy, + re-read `/srv/homepage/status.json` and diff the `down` list + before/after. + +Removing a card's `probe` key without a code change crashes the checker +(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code +fix. The `description` field is consumed only by `app.js` `descHTML()`, so +adding a `details` object (role/spec/inputs/use) is schema-safe: +`tests/homepage_config.bats` checks only name/category/section/probe. + +## Local render check (before every deploy) + +The page fetches `services.json` + `status.json` at load; `file://` is +CORS-blocked, so serve over HTTP: + +```bash +cd shared/homepage +python3 -c "import json; s=json.load(open('services.json')); \ + json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)" +python3 -m http.server 8899 --bind 127.0.0.1 +``` + +Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s** +(panels render after the async fetch; checking immediately returns 0 panels), +extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/ +`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a +plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for +visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy +time) — harmless. + +The scratch `status.json` is untracked and NOT git-ignored — move it to +`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`). + +CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim, +`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as +`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]` +reset so `?bare=1` strips it to black. + +## Deploy + +``` +./scripts/deploy.sh dankclaw --service homepage --skip-e2e +``` + +Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit +datetime), restarts the `homepage-status` timer. Verify live: +`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`. + +**`--skip-e2e` is justified for homepage-only changes when the Mac mini is +asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks +Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE, +no remote wake) times out that test — a pre-existing environmental failure, +not a regression. All non-Mac-mini e2e tests pass. + +## e2e raw-IP *arr failures — probe from the browser's box, not a peer + +When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the +Mac is asleep". Sometimes true; not always. Playwright runs on the dev box, +so a healthy probe from dankclaw proves nothing about the **browser's** path: + +```bash +# 1. From the box that RUNS the browser: +curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/ +# 2. From a LAN peer: +ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \ + --max-time 6 http://192.168.0.112:$p/; done' +# 3. Liveness (ICMP, not TCP): +ping -c 2 -W 3 192.168.0.112 +``` + +| peer probe | dev-box probe | ping | meaning | +|---|---|---|---| +| 200 | 200 | ok | transient — re-run the spec | +| timeout | timeout | fails | Mac asleep; wake, ports return in seconds | +| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help | + +Third row: confirm the Mac is innocent through the dankclaw jump +(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback +`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw +--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is +between the dev box and the Mac. + +Don't hold a deploy hostage to a known-environmental red gate: verify the +deployed service **directly** (its own ledger/health/real completion), state +the gate was skipped or partially red and why (`--skip-e2e` prints the +warning, so the skip is recorded), and report the residual gap plainly +("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer +probe returns 200; deploy verified by other means"). Do not weaken or skip +the spec — it guards a real SSO-plus-RBAC contract. + +## Related + +- [[Container Host Reachability]] — the netns rules behind the vantage +- [[Observability Provisioning]] — the monitoring stack reading the same fleet +- [[Deploy State Triage]] — service-scoped deploys and the e2e gate +- [[Testing Policy]] — the seam/e2e contract this page's checks live in diff --git a/serverAppConfigs-docs/Infrastructure/LAN-DNS.md b/serverAppConfigs-docs/Infrastructure/LAN-DNS.md index 2bb5029..bcb668d 100644 --- a/serverAppConfigs-docs/Infrastructure/LAN-DNS.md +++ b/serverAppConfigs-docs/Infrastructure/LAN-DNS.md @@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the | HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) | | Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint | +## Per-client fix: resolver drop-in + browser DoH bypass + +Symptom: a LAN machine gets `connection timeout` (curl rc=28) on +`https://*.franksdank.cloud` while the server is healthy. Root cause: the +public DNS hands out the WAN IP `71.227.227.172`, which has **no 443 +port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN +answer → SYN to a filtered port → timeout. There are TWO resolver layers to +check, in order: + +**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in +sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route +servers: + +```bash +dig +short www.franksdank.cloud @192.168.0.143 # expect 192.168.0.143 +dig +short www.franksdank.cloud @1.1.1.1 # expect 71.227.227.172 (the trap) +getent hosts www.franksdank.cloud # what the OS actually resolves +curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \ + https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n' # 200 = server fine + +sudo mkdir -p /etc/systemd/resolved.conf.d +sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF' +[Resolve] +DNS=192.168.0.143 +Domains=~franksdank.cloud ~franksdank.casa +EOF +sudo systemctl restart systemd-resolved && resolvectl flush-caches +``` + +The `~` (route-only) prefix is load-bearing: it routes only those domains to +Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`; +`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`. + +**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If +Layer 1 is verified but the BROWSER still times out, the browser is doing +its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR +provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it: +`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead +end). Diagnosis details: + +- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT + `~/.mozilla/firefox/` — find it via the running process if the default + path is empty. +- `grep network.trr. /prefs.js` — **no entries = shipped default, + and since Firefox 113 the shipped default is DoH ON.** + +Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy +& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** → +`franksdank.cloud` (and `franksdank.casa`). The exception list = domains +that do NOT use DoH; they fall back to the system resolver, which has the +route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome: +Settings → Privacy and security → Security → "Use secure DNS" → off, or +"Use the same settings as your system". Do NOT hand-edit `prefs.js` while +the browser is running — it holds prefs in memory and rewrites the file on +exit, clobbering the edit. + +This is a per-client fix: every LAN client that must reach the homelab by +name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not +bypass it with DoH. The server side needs no change. Verified on blankpad +2026-09-21: the drop-in was present since 09-12, curl returned 200, and +Firefox 155 with default (ON) DoH was the remaining hole. + +## Porkbun zone ops (franksdank.cloud) + +Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so +the API key/secret sourced from `/home/dank/serverAppConfigs/.env` +(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe +scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' < +/tmp/script.sh`. + +Endpoints (all POST, JSON body): + +- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud` + — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong + here). Response shape is **top-level `.records[]`**, NOT + `.content.records[]` as some docs claim — parse with + `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`. +- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey). +- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`, + `type`, `content`, `ttl`, `prio`). + +Quirks that bite (verified live 2026-08-31): + +1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud` + yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare + label (`ai`) and the API appends the zone itself. Verify after every + rename with a retrieve before declaring done. +2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of + rapid mutations makes some calls return `...` where jq + then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do + not assume the call succeeded — sleep ~8 s and retry the failed ids; + verify by re-listing the zone at the end. +3. Deletes/renames are per-record-id: always retrieve first, map name→id, + then mutate by id. + +Rename procedure (vhost rename = DNS + Caddy together): + +1. `retrieve` → collect ids for the renamed names. +2. `edit` each id with the new **relative** label, preserving + type/content/ttl. +3. Re-retrieve and confirm `.franksdank.cloud A `. +4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the + Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test + fails the suite if any Caddyfile vhost lacks an addn-hosts A record on + 192.168.0.143. +5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed + paths). Capture the suite's true exit with + `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit + code behind `tail`'s own status and has shipped untested commits here. +6. Verify live: `dig +short @192.168.0.143` then + `curl -o /dev/null -w %{http_code} https://` (expect 200/302; + first hit triggers dns-01 issuance, which works without inbound port 80). + ## See Also - [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI) diff --git a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md new file mode 100644 index 0000000..2291595 --- /dev/null +++ b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md @@ -0,0 +1,95 @@ +--- +type: reference +title: Observability Provisioning +description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification. +tags: [observability, prometheus, grafana, provisioning, deployment] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Observability Provisioning + +The monitoring stack is **provisioned as code**: dashboards, datasources, +scrape jobs and alert rules live in the repo and ride the normal deploy flow. +No hand-edits inside containers — they are wiped on recreate and never +committed. + +## The config triad + +| Repo path | What it holds | +|---|---| +| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) | +| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules | +| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) | + +Targets and datasources reference **container names** (`loki:3100`, +`alloy:12345`), never loopback-published ports; the LLM endpoints are reached +by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they +are host-netns services, not containers. Prometheus 3 refuses HTML responses +as metrics — HTML-scraped "probes" are structurally inert and need a blackbox +exporter (see [[Stale Signal Audit]]). + +## The deploy path + +`scripts/deploy-remote.sh` carries two copier functions: + +- `copy_prometheus_config` → `/opt/prometheus/config/` on the target +- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` + +`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and +`shared/grafana/**` to the `prometheus`/`grafana` services, so +`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change +touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files +exist, the copier functions exist, and the provisioned paths match the units' +mounts. + +## Entry points + +**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the +vantage first: container name if the exporter is a container on the same +netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module +if it is an HTTP probe, not a metrics endpoint. + +**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`. +Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet. +Rules reload with the prometheus service. + +**Add a datasource** — drop a YAML file into +`shared/grafana/provisioning/datasources/`. Point it at the container name of +the backend (`http://prometheus:9090`), not loopback. + +**Add a dashboard** — drop the JSON into +`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in +unified storage, not (only) the SQLite table — grep the API, not the DB, when +verifying (see the verify section of [[Fleet Audit and Posture]]). + +## Verify through the APIs, not the filesystem + +```bash +# prometheus: rules and targets actually loaded +curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head +curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \ + python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]" +# grafana: datasource + dashboard provisioned +curl -s -u admin: http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"' +``` + +Health answers from a stale process and provisioned resources can hide in +unified storage — a file existing on disk is not proof the running server +loaded it. See [[Stale Signal Audit]] for the never-green-target decision +table (refused = outage, 404/HTML = config defect). + +## Current delivered state (2026-09-17) + +Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`, +loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user +unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach +via `host.containers.internal`) → signal-cli. Some targets read down for +structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is +the blackbox exporter. See [[Observability]]. + +## Related + +- [[Observability]] — stack topology and the alert delivery chain +- [[Stale Signal Audit]] — classifying down targets before "fixing" them +- [[Quadlet Lifecycle]] — the units that host these services +- [[Deploy State Triage]] — how the planner selects these services diff --git a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md index 6cca651..cd8e3f9 100644 --- a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md +++ b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md @@ -271,6 +271,83 @@ graph TD 6. Step up in size: 9B, then gpt-oss-20b. 7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load. +## Registering packs absent from the catalog (NPU2) + +Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model) +to the FLM registry. Registry facts: user registry +`/home/dank/.config/flm/model_list.json`, shape +`{"model_path": "models", "models": {"": {"": {name, url, +file_url, size, flm_min_version, files[], vlm, default_context_length, +max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve +resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the +env var set lists custom tags (`⏬` = not downloaded). + +1. Back up the registry JSON first (`cp -a`). +2. Insert the entry via a python json edit + atomic `os.replace`. Set + `files` to the FULL HF file list — **including `vision_weight.q4nx` even + for text-only packs**: the qwen3.5-family loader opens it unconditionally + and a missing file makes the first request HANG silently (no error log). +3. **Do not use `flm pull ` for custom entries** — it throws + `json.exception out_of_range key ':' not found` while building + the download list. Place files manually into + `/home/dank/.config/flm/models//` with + `curl -sfL "https://huggingface.co//resolve/main/?download=true"` + — plain `raw/main/` returns 135-byte LFS pointer stubs, not + weights; verify byte counts against the HF tree API before trusting a + download. +4. Kernel packs: copy from the same-shape family pack (shape-match + `config.json` on hidden_size/num_hidden_layers/head_dim first) into + `/opt/fastflowlm/share/flm/xclbins//` via `sudo cp -a` + (root-owned dir). +5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so + the new process loads with all files present. +6. Verify with a REAL completion request (`model: ":"`) — + `/v1/models` alone proves registration, not loadability. + +Hard constraints: only ONE `flm serve` process can hold the NPU hardware +context — a second concurrent serve process fails every load with +`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status +0x2000003); never run scratch/verify instances while production holds the +NPU. Tag switching happens INSIDE one serve process; the first request +after a model switch pays ~6–11 s cold-load cost (visible in +`usage.load_duration`). Over ssh, never `pkill -f ""` with a +pattern present in your own command line — use bracket-guarded patterns +(`[q]wen3.5:4b`) or exact PIDs. + +## Tag load-failure diagnosis (200 with an error body) + +`POST /v1/chat/completions` answers **HTTP 200** with a flat +`{"error":"Failed to load model!"}` and no `choices`; `/v1/completions` +returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP +200. A status-code-only client reads this as a successful empty completion — +**always check for the `error` key**. Two distinct causes, distinguished via +`journalctl -u --full`: + +1. **Unknown tag** (not in the registry): immediate error, nothing loaded. + Correct behavior, not a bug. +2. **Registered tag whose kernel pack is missing**: FLM resolves the tag, + downloads missing weight files on the fly (first request hangs ~90–110 s), + then fails `Failed to load model: No such file + '/home/dank/.config/flm/xclbins//.xclbin'`; subsequent + attempts fail fast (~2 s). Root cause: the serve script exports + `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every + kernel lookup resolves under the user dir, which only contains symlinks + someone linked. **A tag listed in `/v1/models` is NOT proof it is + loadable on this host.** Fix: one symlink per missing pack + (`ln -s /opt/fastflowlm/share/flm/xclbins/ + /home/dank/.config/flm/xclbins/`) — verified live: after the + symlink the first request loaded and answered in 7.4 s wall. + +Switch-cost behavior: the `model` field routes per-request (no restart +needed); a request whose model differs from the resident one triggers a full +reload (7–11 s wall for an 8-token completion on ~4B-class packs); +alternating tags pays the swap each time. Single-lane NPU: concurrent +requests queue FIFO behind the current lock holder (`NPU Locked!` → +`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep +the journal for `Failed to load model` (the quoted xclbin path names the +missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks = +installed) against `/opt/fastflowlm/share/flm/xclbins/` (available). + ## Incident: qwen3.6-moe unreachable (2026-08-16) Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window. diff --git a/serverAppConfigs-docs/LLM/FreeToken Builds.md b/serverAppConfigs-docs/LLM/FreeToken Builds.md new file mode 100644 index 0000000..cf83a1d --- /dev/null +++ b/serverAppConfigs-docs/LLM/FreeToken Builds.md @@ -0,0 +1,97 @@ +--- +type: reference +title: FreeToken Builds +description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit. +tags: [freetoken, build, dankclaw, vllm, mtp] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# FreeToken Builds + +FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken +vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with +different dependency traps. Both were verified end-to-end 2026-09-11/12. + +## Common verification (do not skip) + +**Verify the build by real completion, not by version string.** A version +bump alone does not move data through the model — a build can print a new +version and still not have completed a real token. Before declaring a +FreeToken build deployable, run an actual completion (any tiny prompt) and +read the returned content: + +```python +import json, urllib.request +req = urllib.request.Request("http://:/v1/chat/completions", + json.dumps({"model": "frank-small", "messages": [{"role": "user", + "content": "Hi"}]}).encode(), {"Content-Type": "application/json"}) +print(json.load(urllib.request.urlopen(req, timeout=600))) +``` + +A 200 with a real `choices[0].message.content` is the only build green. + +## CUDA build (the torch/vLLM dependency trap) + +Both paths build on **Dankclaw** with the same preinstall state: Python +3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo +download.pytorch.org/whl/nightly/cu130, installed with +`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev+cu130" +--index-url https://download.pytorch.org/whl/nightly/cu130` — use +`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev +version). Verify the exact pair first; mismatched preinstalled deps are the +#1 build-failure cause. + +Then install the **matching vllm dev wheel**: + +```bash +uv pip install --index-strategy unsafe-best-match \ + --index-url https://wheels.vllm.ai//vllm-0.6.6.dev0+g.cu130-cp312-cp312-linux_x86_64.whl \ + --extra-index-url https://wheels.vllm.ai/nightly/cu130 +``` + +The wheel URL pattern is fixed: +`https://wheels.vllm.ai//vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl` +— read it back from vLLM's CI-run page (`run details → +build_and_install_package` job, "install the vllm wheel" step). **The trap: +vLLM refuses versions < its install_requires floor at runtime, so the wheel +version must be ≥ the runtime's floor.** A wheel built at a lower base +(0.5.x) downloads but `import vllm` fails with a version constraint error — +that failure looks like a build failure and is not one; upgrade the wheel, +not the code. + +If `uv pip install -e . --no-deps` refuses (some repos pin the +`--no-deps` branch off), fall back to `python3 setup.py` inside the venv, +after `python -m venv --copies .venv && source .venv/bin/activate` and +`uv pip install --no-build-isolation "torch==…"` — the venv must NOT +re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI). + +## ROCm build (gfx1151 via TheRock nightly) + +Same venv discipline, ROCm wheels from TheRock's nightly index: + +```bash +uv pip install --index-strategy unsafe-best-match \ + --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \ + "torch==2.10.0.dev+rocm" "triton==3.6.0" "pytorch-triton==3.6.0" +``` + +Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`, +repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two +prebuilt extension wheels into the tree (the repo's `setup.py` skips +compiling them when found). The ROCm build also needs +`install-fastrak-gfx1151.sh` from the repo's scripts and +`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's +x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub +`franksdank/freetoken-ops-gfx1151` — fetch it once and point +`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless +the extension API changed. + +`--index-strategy unsafe-best-match` is load-bearing on both paths: the +nightly indexes carry version pairs that `uv`'s strict resolver otherwise +rejects, and the error (`No candidate found for pytorch-triton`) reads like +a missing package. + +## Related + +- [[Model Swap and Rollback]] — the swap/deploy story these builds serve +- [[Llama Cpp Serving Sizing]] — the harness measurement discipline diff --git a/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md b/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md new file mode 100644 index 0000000..916ece2 --- /dev/null +++ b/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md @@ -0,0 +1,159 @@ +--- +type: reference +title: Heterogeneous Split Triage +description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking. +tags: [heterogeneous, multi-device, llama-cpp, triage, pcie] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Heterogeneous Split Triage + +Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across +the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X +across device A and device B" with arithmetic instead of enthusiasm. + +## 1. Characterise the link (bytes vs latency) + +```bash +lspci -vv -s | grep -E "LnkCap:|LnkSta:" # capability vs actual +nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv +``` + +Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and +upshifts under load. Measure while the device is busy, or you size the design +against the wrong number. Per-hop cost against the hidden state +(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at +Gen4 x4. + +- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token + for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost + never the constraint.** +- The real budget line is **hop COUNT**: every device boundary is a + host-side sync. A runtime that splits per *op* hits ~100 splits/token and + pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never + assume. +- **Design rule: partition by where the weights live, never by where the + activations flow.** Resident weights, activations crossing. + +## 2. Get the workload's real bottleneck (KV is usually it) + +``` +KV/token = kv_bytes_per_position × context +``` + +Attention reads the whole cached context every token, so KV/token scales +with context — compare it against expert/weight traffic per token and total +token time. At long context on a hybrid arch (few attention layers, many +positions), **KV dominates**. Consequences: + +- The highest-leverage placement is **KV cache on the fastest-memory device, + together with its attention compute** — never KV alone: separating them + puts KV/token over the link, which is fatal (GBs across 7.88 GB/s = + hundreds of ms/token). +- Check the fit: `kv_positions_per_slot × slots × bytes_per_position + + attention_trunk ≤ VRAM`. + +## 3. What placement never buys + +Prefill is compute-bound and expert weights are read once per batch, so +moving them to faster memory buys nothing while adding a serial stage. +Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 +(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the +GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A +`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that +decode gain to within ~2 points — usable for decode projections, and it +assumes sequential stages (true at batch 1; no pipelining). + +## 4. Is the second device even a separate pool? + +On a UMA APU, check before assuming additivity: + +```bash +cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small) +cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM +rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees +``` + +A large GTT window means the "GPU memory" is the *same* LPDDR under one +memory controller: adding a stage there adds a compute engine, not capacity +or bandwidth. Only the discrete card's VRAM is additive. Fencing more +(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM. + +## 5. Speculative decoding as the "cheap" split + +A drafter on the *other* device is bandwidth-free (token ids per round, tens +of bytes), so it is tempting. Before proposing it: + +- the drafter needs the target's **exact tokenizer** for a plain draft model + (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it + must be a head **trained for that target** (EAGLE/DFlash/MTP are + conditioned on the target's hidden states and are not portable between + models); +- a plain drafter costs **K sequential forwards per round**, so it must be + 10–50× faster than the target. Break-even with `r = t_draft/t_target`: + `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5` + (a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3; +- compare against the **incumbent**, not the no-draft baseline: a target + already running its own MTP head is a high bar; +- check the runtime accepts an external drafter at all — closed engines may + enumerate a fixed drafter set and 400 anything else (capability gates + keyed on the loaded checkpoint). That is a design boundary, not a config + gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd` + (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`. + +## 6. Expressing a split (llama.cpp, the usual vehicle) + +```bash +-dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest) +-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (KV co-location) +``` + +- buffer types match by name at load (`ggml_backend_buft_name`), so device + names work as `-ot` values; +- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`; +- several backends in one binary is fine (CUDA + Vulkan is the practical + pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing + build's CMakeCache tells you what needs adding; +- contiguous ranges are the WRONG split when KV bandwidth dominates: they + drag half the attention layers (and their KV) back to the slow pool. + +## 7. Hazards checklist + +- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they + are read from; moving them is pure loss. +- Keep `mmap` on for models with huge embedding tables (anonymous memory + gets the process cgroup-killed). +- CUDA graph capture can abort on unusual graph shapes (hyper-connections) + — check before assuming optimisations apply. +- A brand-new backend on the second device may not implement the arch's ops + at all: make the first measurement a **baseline of the runtime on that + device alone** — a split's arithmetic is worthless if the baseline is + worse than the incumbent engine. +- A split consumes the fast device entirely — check what else it was + serving before trading it away. +- Benchmark discipline on production hosts: stop and mask first + ([[Fleet Ops Playbook]]). + +## 8. Verdict shape + +Rank and report; prefer the boring answer when it wins: + +1. **Two independent endpoints** (zero shared traffic) — often the real + answer to "what's a useful combination", especially over a narrow link. +2. **One process, weights-resident, KV co-located with attention on the + fast pool** — promising and quantifiable (+% decode at long context); + needs a dual-backend build and a baseline measurement. +3. **Hot-expert cache / per-token weight streaming** — affordable only if + the miss set per token is small (a few experts; check + `misses × bytes × tok/s` against link bandwidth) and overlapped; injects + PCIe latency into the critical path; no mainstream runtime implements a + dynamic expert cache. +4. List explicitly what does NOT help: TP/collectives across unequal + devices, KV on the far device, model loading over the link, + prefill-motivated placements. + +## Related + +- [[PCIe Link and Atomics]] — the link and collective walls underneath +- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes +- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic diff --git a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md b/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md new file mode 100644 index 0000000..76e3d2c --- /dev/null +++ b/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md @@ -0,0 +1,271 @@ +--- +type: reference +title: LLM Bench Windows and Measurement +description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding. +tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# LLM Bench Windows and Measurement + +Fleet law first: **STOP the production service before any GPU experiment** +(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the +GPU and corrupts results; an unisolated GPU experiment has precedent for +wedging a host until power cycle. Reference harnesses: +`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`, +`scripts/autoresearch/frank-primary-arms.sh`. + +## 1. The window structure (one config per arm, a driver owns the window) + +1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND + no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts + EXITED containers and can never clear — `rm -f` the exited one first) AND + port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01: + `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s + then polls `/health` gets **200 from the still-running PRODUCTION engine** + — the arm "passes" with production's boot log, then `podman exec` dies + with `container state improper`. +2. **Boot the arm as the service, not a scratch port** (no scratch ports on + LLM hosts — one card, one server). The serve script takes its config via + env-file override (`FRANK_SMALL_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`) + pointing at a staged copy; the checkout is never mutated. +3. **Fast-fail the health wait**: poll `podman ps | grep -qx ` inside + the loop — a refused boot dies in one poll instead of 20 min. Capture the + serve script's FULL stdout on failure (a `tail -8` loses engine-init root + causes; raise happens above the tail). +4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots, + SpecDecoding lines) — they survive measure crashes. +5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills), + `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver + before the next boot; restore production from the EXIT trap and verify the + **conjunction oracle**: health 200 + `is-active` + `is-enabled` + + mask-symlink clean. A bare `systemctl start` is not a restore oracle. + (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is + enough: systemd never auto-restarts an explicitly stopped unit.) +6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for + mixed speeds. + +**Destructive tails (2026-10-01):** a scratch script whose tail does +`systemctl stop` + `podman rm -f ` fires later if a +probe loop survived the driver kill — it can destroy a just-deployed +production engine (signature: deploy exits green, then seconds-to-minutes +later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not +crashed). Before ANY deploy: `pgrep -af` every scratch script of the session +and kill strays by explicit PID; killing the dev-side job does NOT kill the +host-side `deploy-remote.sh` (two of them once managed the same unit). + +Two failure shapes inside the window that read as "the bench died": + +- **Stale reads from a stopping container.** `podman stop` answers while the + engine inside is still flushing; a `podman logs` taken in that window + returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or + `podman wait`) before reading logs, or you will write "no results" into + the table for a run that has them. +- **A D-state worker wedges the stop.** A bench child that wedged in the + kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and + then the container is `Stopping` forever. Kill by container first + (`podman kill ` → SIGKILL path), verify the process is gone from + `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the + window script proceed to a redeploy with a zombie bench worker holding + VRAM. + +## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth) + +**What llama-bench cannot express — decide BEFORE designing:** no +`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove +activation buffers fit beside the DEPLOYED KV — only the real unit answers +that), no `--parallel` (no slot shape, no concurrency), no `--spec-*` +(`tg` numbers are the drafterless baseline, useful as a control only), and a +warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per +cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at +~zero depth and meaningless for long conversations. + +**The separator trap:** `/` separates devices WITHIN one configuration, `,` +separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the +single value 36 → everything on one card. `--tensor-split 36,29` (server) → +`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`. + +**Harness shape:** one invocation per cell (an infeasible cell calls +`exit(1)` and kills every later cell of a cross-product invocation — and +infeasibility is usually the question); TSV row per cell as it completes + +`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi +--showmeminfo vram` VRAM trace per cell (a cell that allocated less than +intended looks fast); resume by skipping non-empty cell outputs; stop the +service in the harness, restore from `trap … EXIT`, verify afterwards. Match +the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`). + +**Run under a transient unit, not a shell:** + +```bash +sudo systemd-run --unit= --collect --no-block --property=Type=oneshot \ + --property=LimitMEMLOCK=infinity --property=User=dank \ + /home/dank/serverAppConfigs/scripts/.sh +``` + +Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has +8 MiB and fails the load for an unrelated reason. This also survives ssh +disconnects. **Boot rungs under the production environment, not a bench +shell** — a rootless shell probe booted 3×262144 bf16 that production refused +(systemd `User=` + memlock change what the pinned-expert path holds): +`sudo systemd-run --uid= --unit= -p LimitMEMLOCK=infinity -p +Restart=no env VARFILE=… /path/serve.sh`. + +**Verify before burning hours:** dry-run against a STUB binary with +`SERVICE=` (the trap/stop never reached), tiny grid, +including one forced-`exit 3` cell — prove the TSV row, failure capture, +resume logic. Validate the real flag set in ~1 s by pointing `-m` at a +nonexistent file. **Reading:** bracket the sweep (reference cell first AND +last); check per-cell `stddev_ts` (a difference smaller than spread is not a +result); single-sequence decode is usually flat across batch/ubatch (prefill +knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating +smaller-batch cells); report what the grid cannot prove (drafter, slots, real +context). + +## 3. HTTP benches of server-side parameters + +`llama-bench` benches the bench binary, not the running server — server-side +params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need +HTTP: + +- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure + `prompt_tokens / wall_time`. +- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and + spec acceptance all stabilize over thousands of tokens); measure + `predicted_per_second` from `timings` or `completion_tokens / wall_time`; + temperature 0. + +Known fleet harnesses: `scripts/bench-threads-batch-http.py`, +`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`. +sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3 +reasoning parser streams zero deltas — use `/v1/completions`. + +## 4. The measurement law (under speculative decoding this is absolute) + +**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token +mapping; chunk counting reported 8.3 tok/s for a config whose server counters +said 23–26 — a 3x false regression that nearly abandoned the change that +worked. Always: + +```python +decode_tok_s = usage.completion_tokens / (wall - ttft) +# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk +``` + +vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a +measurement (6.2 → 13.1 across adjacent windows on identical settings) — use +them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean +acceptance length`, per-position rates). Warm pass + ≥3 runs, median; +temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions +(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's +repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text +before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample +nearly shipped the wrong setting. + +## 5. Honest methodology + +- **Match the corpus to the workload.** Token soup under-reports speculative + decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft + settings; acceptance, not step time, moves throughput. Build the corpus + from the repo's own sources: `{"prompt": "\n\n"}`, + `--custom-output-len`. +- **Salt every run** with a unique per-arm token in the first bytes — it is + the prefix-cache key (and a proxy routing key). Without it arm 2 measures + arm 1's warm cache. +- **Two workload shapes need two clients:** distinct prompts (batch + behaviour, prefill parallelism) and sequential turns with growing history + (TTFT of the first streamed chunk per turn — the only measurement that + exposes prefix-cache behaviour). These pull in opposite directions: a cold + batch spreads prefill across replicas, a conversation's next turn only pays + if it lands where its prefix is cached. +- **Read the engine's ledger, not the card:** `Available KV cache memory` / + `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per + request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line + is a LIFETIME average across every prompt the replica ever saw — send the + same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold + → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not + full; it does NOT say a prefix survived. The KV budget is NOT monotonic in + chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure + three points before treating a knob as a lever. +- **Soak before believing:** 15–30 min of concurrent families, count failures + AND latency; report hit/miss split and medians with min/max — the median of + a bimodal distribution describes neither mode. +- **Verification that is not the benchmark:** run the client the fleet + actually uses; and a deployed unit is not proven until an unattended reboot + brings it back. +- **Record the "no" answers** with numbers and mechanism — placement balance, + concurrency admission and chunk-size tweaks all measured flat or worse + here; the wall was arithmetic (cached prefixes + live sequences share one + pool). + +## 6. Sweep hygiene (the arms that produced wrong verdicts) + +- **Prefetch uniformity:** every arm in a sweep carries the same + expert-prefetch setting. The 2026-09-29 campaign measured losing arms with + prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec + 12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed + verdict. +- **Knob-landed preflight:** for every `-e` knob, confirm + `podman exec printenv ` — a knob the fork silently ignores + produces a "parity" arm that is really the default (v1 of the gather sweep + lost all 3 arms this way; serve-script hooks are additive and can drop + vars). +- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was + deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers + compare against default-off, not "incumbent + one flag". +- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than + autofit (remainder) so pool numbers are reproducible; compute need = + tokens × measured B/token (hybrid models: KV dtype does NOT double mamba + states — measure, don't assume 2x). +- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly + deployed a 2x-prefill-tax config. Per-depth tables, never a single median — + one fork's gather win grows with depth (54 t/s @1k, 39.5 @short). +- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy + `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them. + +## 7. Bash laws that ate windows (each cost an arm) + +1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word → + `SPEC=3: command not found`, nothing runs. Pass literal assignments: + `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty. +2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside + an if/else it aborts ONLY that compound statement — the script CONTINUES, + every arm tears down clean with zero measurements. Detection law: `ls` + the per-arm `.measure.txt` files before trusting any summary; arm echoes + are not evidence. +3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u` + it aborts AFTER the harness stopped production. Two statements. +4. `sed -i` renames the file; already-running bash keeps the OLD inode — + mid-flight patches miss arms launched in the preceding minutes. Patch + early or run a make-up window. +5. Never heredoc python through ssh→zsh→bash — write to a file, ship it + (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting + fails silently. Same trap in `git commit -m`: backticks become command + substitution. +6. No comments INSIDE a backslash-continued command (the continuation joins + the comment); verify multi-line command construction with `bash -n` AND a + render test. + +## 8. Results location and recovery (frank-primary arms) + +| copy | path | +|---|---| +| per-arm evidence | `$ARMS_DIR` = `~/.jobs//` (`.rc`, `.done` beside it) | +| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `.out/.env/.overlay/.rc/.metrics` | +| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` | +| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) | +| fallback | `~/.omp/agent/sessions///.{shake,bash}.log` (permanent) | + +`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests — +constants 99.9/88.8 identical across arms by construction; **identical +numbers across arms == stub or bug, never a measurement.** After compaction, +re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session +logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per +window. + +## Related + +- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept +- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes +- [[LLM Performance Diagnosis]] — profiling and attribution methods +- [[Remote Job Runner]] — detached execution of long sweeps +- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md new file mode 100644 index 0000000..777c3fe --- /dev/null +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -0,0 +1,221 @@ +--- +type: reference +title: LLM Performance Diagnosis +description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing. +tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# LLM Performance Diagnosis + +When a served endpoint is slower than wanted and someone names a suspect +("PCIe is saturated", "the gather is the bound"), throughput arithmetic cannot +settle it — it is usually circular (it assumes the suspect consumes the step). +Get data in this order; the profiler window alone has retired multi-day +optimization plans. + +## 1. Is decode context-bound at all? (the flat-depth probe) + +Sweep decode across context depth. **Flat** ⇒ attention over history is not +the cost; the problem is per-token expert movement and KV/attention tuning is +ruled out entirely. A genuinely context-bound server shows a clear downward +slope. Record cold vs warm separately — a small gap means the cache is +working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304 +to 16,095 tokens. + +## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer) + +Symptom: smooth ~25–35 % utilization with 100 % peaks on an expert-offload +MoE. This is the EXPECTED signature, not a fault: + +``` +per decode step: compute burst ~100 ms (100 %) → expert-gather wait ~400 ms (~0 %) +resident fraction = arena slots / total experts (e.g. 6574/24576 ≈ 27 %) +PCIe bytes/token = miss_rate × lookups × slab_bytes (0.73 × 480 × 2.344 MiB ≈ 0.84 GB) +predicted duty ≈ compute_ms / (compute_ms + wait_ms) +``` + +Validate the terms against the deployed engine's boot plan +([[RDNA4 MoE Expert Offload VRAM Budget]]). If predicted ≈ observed, the low +average is structural. Note small-transfer reality: bulk bandwidth (91 % of +ceiling at 1 GiB) overstates what 2–3 MiB slabs get (~72 % plus ~20 % +per-transfer latency). Before blaming the link, measure it +([[PCIe Link and Atomics]]) and compare the engine's effective gather rate to +the measured slab-size bandwidth — a 40–50 % gap is the software read path, +not the wire. `rocm-smi --showuse` reads ~0 % during the whole multi-minute +weight load: show VRAM (`--showmeminfo vram`) as the load-progress signal. + +## 3. Kernel-time attribution (the profiler window) + +Run BEFORE building any replacement kernel. + +1. Stop + mask production; boot the shape under test with ONLY the profiler + config added. vLLM argv: + `--profiler-config '{"profiler":"torch","torch_profiler_dir":"/prof","torch_profiler_with_stack":false,"torch_profiler_record_shapes":false,"torch_profiler_with_memory":false,"torch_profiler_dump_cuda_time_total":false}'` + — `with_stack=false` is load-bearing (with stacks the stop hangs and the + trace buffer grows ~40 GB → kernel OOM-kill of the engine core, anon-rss + ~95 GB; the worker-side stop can hang 5 min on ROCm/kineto). Bind-mount + `/prof`. If a shell substitutes values into that JSON, verify the escapes + actually landed. +2. Window it: warm request → `POST /start_profile` → N generations of ≥128 + tokens (~15 s of traffic) → `POST /stop_profile` (client may time out at + 60 s; the trace still lands). `telemetry_report` has ONE caller: the + gpu_worker profiler-stop path. Overhead is small enough to keep decode + comparable (9.6 vs 9.9–10.0 tok/s measured). +3. **Parse ON THE SERVER** (traces are 100–150 MB gzipped, GB expanded). + Rank `ph=="X"` events with `cat in {kernel, gpu_memcpy, gpu_memset}` by + summed `dur` (µs): report % of device time, launch count AND µs/call + (22 µs × 708k launches is a different problem than 4 ms × 4k), plus the + first-to-last event span (fraction of wall the device was busy). Aggregate + by FULL mangled name (template args are the tuning signal) and by stripped + symbol. **Trap: `gpu_user_annotation` events masquerade as GPU time** (a + naive sum reported 63 % in one `execute_context_0(...)` entry; spans + overlap and can exceed the wall clock). +4. Divide by the window's token count → ms/token; if kernel time ≈ step + time, the table IS the optimization plan. + +Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s): + +``` +34.8 % 708,500 launches vectorized_elementwise_kernel <- the actual bound +10.0 % 42,000 MoE W4A16 GEMM + 8.9 % 83,160 router/top-k (sort-based) + 6.5 % 133,920 hyper-connection pre/post kernels + 1.05 % 20,664 +``` + +No amount of link arithmetic would have produced those two conclusions: the +suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk — +whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also +predicted a negative result in advance (spec decode cannot help a shape whose +per-token cost scales with the token count) which matched an unexplained flat +MTP measurement. + +## 4. Telemetry without benchmarking (live hosts) + +When benching is forbidden (production serving), mine the runtime's own +per-request telemetry — it measures the real workload and costs nothing. + +- llama.cpp (root unit, `-lv 3`): `sudo journalctl -u --since "-3h" + --no-pager` → `slot print_timing` lines (prompt eval, eval, total, `graphs + reused`, `draft acceptance … mean len`, `n_gen = N, tg = X t/s, tg_3s = …`). +- halogen (rootless quadlet): `journalctl --user -u ` (**never sudo** — + per-user storage) → `serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16 + rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok`. +- Read the envelope, never one sample: short-prompt `prompt eval` is + overhead-dominated (57 tok @ 168 t/s) — quote the longest available (1106 + tok @ 1013 t/s); decode and acceptance are content-dependent (0.46–0.76 + measured on one unchanged build); `graphs reused > 0` / `selected slot by + LCP similarity` prove prefix-cache reuse. +- **Attribute PCIe without touching the model:** background `nvidia-smi dmon + -s t -c 16 -o T`, ONE ordinary `max_tokens=200` request, `wait` at the end + (without it ssh closes and the client dies — the journal shows `W srv stop: + cancel task` and NO print_timing while curl still reports plausible + status). Measured: peak ~230 MB/s against a ~7,900 MB/s link = ~3 %, 0 at + idle — the sampled-logits stream, not the transport, is decode traffic on + that shape. +- One `max_tokens=200` request is production traffic and fine; anything that + pins the card is a bench → stop-and-mask first. Check WHICH instance you + are reading (`systemctl status | head -12`; task ids jump across + restarts). The engine's own ledger (`KV self size`, `n_slots`, + `n_ctx_slot`) outranks your model of it. + +## 5. The fork's data path (before blaming the link) + +Offload engines pick between DMA copy-in and read-through UVA dereference +inside the GEMM. Grep the installed package for the LRU/offload module and +read the BRANCH CONDITION, not just the env names. A token-count threshold +decides the path, and it interacts with spec decoding: an MTP step is k+1 +tokens, so the step can silently cross the threshold onto the slow path as +draft depth grows. Check the threshold's premise: a default tuned at +12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be +several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone +does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required); +check for the skip line before concluding autotuning ran. Gather-geometry +knobs read at import time → each arm is a full reload. + +## 6. RAM-bandwidth-bound MoE ceiling (before recommending a quant) + +``` +routed experts/token = top_k × n_layers +bytes/token = routed_params × bytes_per_param + shared_expert_bytes +ceiling (tok/s) = measured_bandwidth / bytes_per_token +``` + +Measure the HOST's real bandwidth, never assume DIMM ratings: 4-DIMM AM5 +boards downclock to DDR5-3600 regardless of 5600 sticks (two large DIMMs beat +four small). Single-threaded reads under-report dual-channel ~4x — use the +multi-threaded plateau of a checksumming reader (`scripts` pattern: +`/tmp/bw.c`, 1/2/4/8/16 threads). Worked example (DeepSeek-V4.1-Flash): +6×40 experts × 35.4M params = 8.5B params/token; at 3 bpw + 0.53 GB shared = +**3.7 GB/token** → 42 GB/s ⇒ **11.3 tok/s ceiling**. Sanity anchors: H100 +NVL + CPU offload 226 GB ⇒ 5.8–9.0 tok/s; 2×3090 + 768 GB DDR4 ⇒ 12.2. +CPU compute rarely binds below 4 bpw (~5.3 flops/byte ⇒ ~223 GFLOPS at 42 +GB/s); PCIe expert STREAMING is worse than CPU compute (Gen5 x8 ~22.7 GB/s +effective < half of RAM); prefill reads the union of a batch's experts +(prefill ≈ bandwidth / total_expert_bytes × ubatch). A quant that fits only +by ignoring the Engram table does not fit (Engram = sparse n-gram lookup, +NVMe-tolerant; the routed experts are not). An iGPU GTT aperture is an ALIAS +of the same RAM, not extra memory. + +## 7. IBS model-access attribution (AMD CPUs) + +Real CPU access patterns for mmap'd weights, live, no stop (established on +frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference: +`scripts/frank-primary-expert-activity.py` (self-contained dump + sample + +analyze; `--analyze-only`). + +1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents + from CONSECUTIVE offsets, never a type-size table** — a wrong per-element + size silently under-covers every tensor and dumps tail samples into a junk + bucket (first run lost 59 % of expert samples; a uniform gap/exps ratio + ~1.4 across parts is the tell). Data start = `(tell + 31) & ~31`. +2. `/proc//maps` VMAs (7-group regex; pgoff = group 4, path = group 7) + matched against part files; multi-VMA per part is normal. +3. `sudo perf record -e ibs_op// -c 50000 --data -o out.perf -p -- + sleep ` WHILE a mixed workload runs (idle perf numbers mean nothing). + `sudo perf script -i out.perf -F addr` for addresses. +4. Attribute: bisect VMAs, `foff = pgoff + (addr − vma_start)`, bisect tensor + starts; per-expert index = `int(lo / (span / N_EXPERTS))` (expert-major + rows). +5. Verify before believing histograms: gap ≈ 0, every tensor-bearing part + nonzero, and MoE blocks should be FLAT by construction (top-k fires every + block every token) — a non-flat block histogram means mis-attribution + first. Established law: expert routing is per-block flat, per-EXPERT + skewed. + +## 8. Prefix-cache-aware routing (multi-replica serving) + +Measure with a **sequential-turn client**, not only a batch benchmark — a +batch of distinct prompts parallelises prefill across replicas and hides the +problem. The two failure modes: round-robin loses every other cache hit +(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned); +pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s +pinned). **The rule that gets both: route by request shape** — a request +carrying more than one message is a continuation (agent loop re-sending its +context) → pin to the replica that served that prefix; anything else → spread +with a COUNTER, not a prefix hash. Add failover: demote a replica on +connection failure and retry on a live one BEFORE writing anything to the +client (safe pre-stream; never retry after streaming started). + +The wall no policy fixes: **a running sequence and its cached prefix draw on +the SAME pool.** Three concurrent 50–80k families against 122,636 tokens/card +thrash under every placement and admission policy tried (counter placement, +load-balanced placement, one-large-request-per-card admission, both chunk +directions — the last COSTS KV in both directions). When the working set +exceeds aggregate KV, say so: the levers are capacity, fewer concurrent +large contexts, or a different platform. Traps: a lifetime hit-rate counter +is not a per-request verdict (probe directly: same prompt twice to an idle +replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes; +`vllm bench serve` needs `--tokenizer `; verify metric names +against the live endpoint before dashboards. Test shape: drive the routing +function directly with synthetic bodies (stdlib-only policy module); +conversation identity is the FIRST N KiB of prompt — a shorter tested prefix +hashes differently per turn and pinning silently never happens. + +## Related + +- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law +- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling +- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms +- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound diff --git a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md new file mode 100644 index 0000000..ae47760 --- /dev/null +++ b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md @@ -0,0 +1,281 @@ +--- +type: reference +title: Model Swap and Rollback +description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline. +tags: [models, swap, gguf, huggingface, rollback, deployment, verification] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Model Swap and Rollback + +Applies to every endpoint swap on the fleet (frank-small / frank-large / +frank-primary). Repo law: all config flows through git; the live models are +host state under `/home/dank/models//`. + +## 1. Fit preflight — never deploy a swap without confirming it FITS + +1. Get the checkpoint's on-disk/weight size AND precision from the HF card + (`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params: + BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB. +2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that + stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload, + layer splits). Budget weights + KV pool + OS + page cache; **if weights + alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink + weights. +3. **The failure mode is a self-sustaining thrash, not a crash:** a large + swap makes the kernel PAGE instead of OOM-killing → sshd/user space + starved indefinitely, no recovery window opens by itself; systemd + `Restart=always` turns it into a crash-reload-rethrash loop; `enabled` + + `WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle + does NOT fix it. + +Recovery (may need console / SysRq-to-GRUB): + +```bash +sudo systemctl mask --now # break the reboot trap FIRST +cd && git fetch origin && git reset --hard origin/main +grep MODEL_DIR # confirm the fixed target +sudo systemctl unmask && sudo systemctl restart +curl -s # then a REAL completion +``` + +Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED — +confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no +FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose +quant by size fit, then accept the compute path the card supports. Before +choosing a smaller quant, state WHICH of the two you are buying — headroom, +or a design change (a smaller quant does not change what the runtime can +express; per-tensor cross-device placement stays refused under +`--override-tensor`, and a layer split moves only contiguous layer ranges). + +## 2. Ground truth first + +- Read the HF model card (runtime pin, hard constraints, recommended args) + + the repo trio `scripts/.env`, `scripts/serve-.sh`, + `scripts/.service` + the runbook under `serverAppConfigs-docs/LLM/`. +- Check the host before downloading anything: `ls /home/dank/models/`, + `df -h /home`, `du -sh` — bench-era or prior-deployment files often already + exist. + +## 3. Fetch and verify (HF payloads) + +**Inventory by QUANT NAME, never by extension** — large quants ship as +`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an +`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both +mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own +repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors +return 403 even with a valid token until the account accepts the gate — +prefer the ungated copy. + +```bash +# token out of band: mode-600 curl config; a token passed as -H is world-readable +printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc +chmod 600 /tmp/hf.curlrc +curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name" +``` + +`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes +the error body as the artifact (identical 153-byte shards, cheerful DONE, +zero bytes of model). Never `pgrep -a` a running fetch (token is in +`/proc//cmdline`); watch progress by size: +`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`. + +**Verify, then assemble:** per part, byte count == Hub's declared size AND +`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total == +sum of parts (compute the sum with python, not by hand) and `head -c 4` == +`GGUF`. The assembled file has no published hash — the per-part oids + exact +total are the chain. Keep the fragments when disk allows (only copies +re-verifiable against the Hub); failure artifacts and the replaced payload +go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants +and the local file may be the last copy. + +Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already +do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to +skip all three. HF cache snapshots SYMLINK every file into `blobs/` and +those links dangle inside containers — serve scripts and the deployer refuse +symlinks and missing tokenizer files for exactly this reason. + +## 4. Runtime build (when the card pins a fork/build) + +- Clone + build with the card's exact cmake line, plus + `-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui + bucket) aborts llama.cpp link at 98 % on these hosts; the API server is + unaffected. +- Verify every flag the serve script passes exists in the new binary BEFORE + touching the service: `llama-server --help | grep -q -- ""` for each; + list MISSes. Read the GGUF's tensor histogram first if the payload uses a + custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]). + +## 5. Config via git only + +Edit `scripts/.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve +script header/defaults locally; `bash -n`; `./tests/run.sh` must pass; +commit; push; `git pull --ff-only` on the target. Never edit configs on the +remote. Renaming a model dir (`mv`) is host state and allowed. The env-file +hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged +arms possible — keep it intact. + +## 6. Live validation + +- No scratch ports — one GPU cannot host two big models. Ensure the old unit + is truly gone first: `systemctl is-active` can hang in `stop-sigterm` + (llama.cpp never preempts an in-flight generation); poll until the PID + exits. +- Manual serve for smoke: `setsid nohup bash scripts/serve-.sh > + /tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session. +- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows + (`reasoning_content` non-empty); if the endpoint takes images, a + solid-color probe through an `image_url` content part (legacy top-level + `images` is silently ignored; probe with ≥256 px). +- vLLM specifics: first request after boot can take ~18 s (deferred Triton + loads) — warm and discard; `--reasoning-parser` / tool parsers are part of + the contract ([[vLLM Boot Forensics and Config Interrogation]]). + +## 7. systemd handoff — THE TRAP + +`./scripts/deploy.sh --service ` gates health, but a leftover +manual server holding the port makes the health check pass while the UNIT +crash-loops on bind. Always confirm ownership: +`systemctl show -p MainPID` equals the single `pgrep -x llama-server` +(or the container runtime PID); only then trust "healthy". If a real +long-context task is queued, do the handoff anyway — client aborts are +recoverable, drift is not. + +## 8. Close-out (mandatory) + +1. Docs: active runbook updated (build recipe, profile, validation numbers + incl. real-workload prefill/decode); superseded runbook gets a RETIRED + banner and stays as the rollback record; `LLM/index.md` mermaid + active + list; AGENTS.md fleet table + runtime section + history chain; work-log + entry in `serverAppConfigs-docs/log.md`. +2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot + context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides + the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports + `max-model-len` and its concurrency line. Overselling causes silent + client-side overflow or a discovered-wall 400. Also `maxTokens` vs the + engine's cap, `input` capabilities (text/image), and + `reasoningContentField` per runtime (llama.cpp/SGLang emit + `reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry + that named the old endpoint. +3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary + + payload kept on disk, rollback numbers recorded in the runbook. + +## 9. Revert-pin an ad hoc experiment (the three pins) + +A campaign is "done" only when all three exist — without them a +`podman pull`, an image update, or a lost scratch script destroys the way +back: + +1. **Git tag (annotated, pushed):** + `git tag -a -tuned-YYYYMMDD -m ""` — a future agent must reconstruct the + campaign from the tag message alone. +2. **Image digest in the serve script:** floating tags move. Capture + `podman inspect --format '{{.ImageName}} {{.Image}}'` + `podman + images --digests`, patch `IMAGE=` to `repo@sha256:` with a + comment (image ID, date, old ref), verify `podman image exists + repo@sha256:` and `bash -n`. Digest refs survive upstream pushes; + `podman pull repo@sha256:` re-fetches exact bytes. One host can + hold MULTIPLE images of one repo — pin by the digest of the RUNNING + container, never by tag name. Never edit the pinned line back to a + floating tag; new experiments branch from the digest. +3. **Local alias:** `podman tag -runtime:tuned-YYYYMMDD`. + +Record scratch-script paths + full flag surface in the runbook (hook env +vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`. +Pure measurement sessions that changed no state need only pin 1. + +## 10. Research verification discipline (for "can model X run here?") + +- Scouted research summaries have shipped fabricated citations on this fleet + (invented HF repos, plausible-but-wrong PR numbers, stale engine versions). + NEVER copy a scout's citations into a deliverable — re-verify each + load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it + enters anything. +- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim + about our own hosts. +- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active, + CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early + GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path + (heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K). + GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the + default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY, + flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in + unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant + ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp + MMLU-Pro vs Q8_0). +- GGUF parsing gotcha: expert tensors are + `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`; + a `len(parts) == 3` filter matches zero layers with no parse error. Filter + on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`. + +## 11. Adding a second LLM endpoint to a host + +When a host must serve two models at once (an iGPU runtime plus a second +card), or one endpoint is moving and a new one takes the freed hardware. + +**0. Decide the shape before touching hardware.** One provider per +HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs) +— a second endpoint is a new provider block, not a second model in the +existing one. Read the budget as an inequality: for unified memory the +engine's own fit check is the arbiter (`budget = host − pinned weights − +reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV +by counting only full-attention layers — and remember `--parallel N` +DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free +RAM by changing the POOL or the ARENA (the reserve is a fit-check term, +never an allocation) and drop optional charges (a vision tower is ~2 GiB) +rather than hoping. + +**1. Repo first, and retire in the same commit.** New service files +(`scripts/.env`, `.service`, `serve-.sh` with hard guards for +missing artifacts); planner entries (`SERVICE_ORDER` + the target's service +set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the +dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a +suffix — and check what pointed at them: bench harnesses, a chat template +only the old runtime read, and `scripts/test-changed.sh` mappings are the +usual victims (a mapping to a retired test file fails the pre-commit +wrapper silently for weeks). One bats file per service asserting +INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation), +not constants; a negative grep must strip comments first +(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining +the very thing you removed. + +**2. The port-bind race (rootless podman).** pasta binds the published port +at container CREATE; a just-stopped predecessor still holds it. Order the +handler: retire the old manager's unit → stop THIS service's own unit +(no-op on first cutover) → wait for every socket on the port to clear +([[Container Host Reachability]]) → install → start. Without the second +step a redeploy deadlocks against its own gate. + +**3. Wire the surfaces, refuse impossible signals.** Point the harness +small/fast + image roles at the new endpoint; move the vision role WITH the +vision tower (one owner per capability). `shared/homepage/services.json`: a +card per endpoint, probed from the monitor's own host. Prometheus: only add +a job if the runtime actually exposes Prometheus text (halogen publishes +`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the +reason written down rather than left as a permanent red). + +**4. Cutover, then verify by driving it.** `deploy.sh --service + --service ` (order matters); per endpoint: `/health`, +`/v1/models` reports the alias, a chat completion returns content AND +`reasoning_content`, a `tools` request returns `tool_calls`, an image +request is answered (or refused with a named reason), the runtime's own +speculation counters move, and the unit is `enabled`/`generated` with linger +on. Read the engine's boot ledger for a silent clamp. Finally spend one +real request through the harness per changed role — routing labels lie; the +servers' own journals do not. + +Traps that look like model faults: `500 tools param requires --jinja flag` +(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected +reasoning effort ` (fold via `compat.reasoningEffortMap`); the same +request works by hand but a subagent dies (subagents run at a different +thinking level — check both); the container dies instantly although the +predecessor "stopped" (the port race above). + +## Related + +- [[GGUF Compatibility]] — which runtime can load a payload at all +- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads +- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic +- [[LLM Bench Windows and Measurement]] — measuring the new deployment +- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint diff --git a/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md b/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md new file mode 100644 index 0000000..47fc48c --- /dev/null +++ b/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md @@ -0,0 +1,264 @@ +--- +type: reference +title: Tcclaviger Fork Ops +description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence. +tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Tcclaviger Fork Ops + +The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, +vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a +first-class expert-offload planner) served `frank-primary` at shape G until +the 2026-10-05 FreeToken cutover; it remains the rollback target +(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM +arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics: +[[vLLM Boot Forensics and Config Interrogation]]; window discipline: +[[LLM Bench Windows and Measurement]]. + +## The ad-hoc sweep harness (dankserver) + +Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP +depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`, +`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM` +(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config +JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`) +into every fresh container — `podman cp` files die on container recreate. + +- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or + `mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall − + ttft)`; chunk counting is INVALID under spec decode. +- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` / + `out NAME` ([[Remote Job Runner]]). +- **Validity check on ANY speed result:** `completion_tokens == max_tokens` + (else repetition-halt or early EOS), finish_reason, and inspect the text — + `bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A + loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode. + Acceptance swings 48–61% run-to-run — noise; decode is the metric. +- Each arm = full reload: ~8.5 min cold, ~12 min with measurement. +- Readiness gate: `podman logs 2>&1` (stdout+stderr merged) for + `Application startup complete` — it is on STDERR; an unmerged grep can + never match and burns the whole wait timeout. +- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture + are CPU-side); VRAM climbing is the load indicator. +- Config edit discipline: the serve script is patched by scp'd python + scripts with asserts (anchor must exist, addition must not); `bash -n` + after every patch. + +## Measured knob verdicts (2026-09-29 campaign) + +Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648 +TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx. + +| Knob | Verdict | Evidence | +|---|---|---| +| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob | +| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in | +| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact | +| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM | + +Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so +batch-shaped compute knobs move ~nothing. + +The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT, +SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are +the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec +depth. + +Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate +flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a +profile window OOM-kills the engine worker (~40 GB trace buffer; kernel +kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth +(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank.json`) +fires ONLY from the worker's profiler-stop path; worker-side stop can hang +>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate +measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies; +temporal reuse dominates, so never infer miss rates from resident-slot +fractions. + +## Sweep hygiene (fork-specific preflight) + +1. **Knob landed:** for each `-e` env knob, confirm in-container + `podman exec … env` (or the engine log's confirmed-values line). A knob + the fork silently ignores produces a "parity" arm that is really the + default — the v1 gather sweep lost all three arms to exactly this + (GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16). +2. **Prefetch uniformity:** every arm in a sweep MUST carry the same + expert-prefetch setting. The 2026-09-29 campaign measured losing arms + with prefetch ON and incumbents OFF; untaxed, its verdicts inverted + (no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed + verdict. +3. **Spec arms must prove engagement:** engine log + `SpecDecoding metrics: Mean acceptance length …` and + `curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off. +4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a + refused boot dies in one poll instead of 20 min of health waits. Capture + >8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init + refusal) lives higher up. +5. Report per-depth tables, never one number: g32×64's win grows with depth + (54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT + rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx + rows across arms only. +6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token + noise). + +Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`), +arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder +(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a +measure-stage bug costs decode tables but not boot-log findings. + +**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork +is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at +4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util +0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at +4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it +OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002 +tokens). Acceptance is real: mean length 2.4–2.8, per-position +0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is +exactly SEQS×(n+1) (`suboptimal` warning) → try 8192. + +## Serving a model larger than VRAM (dankserver recipe) + +Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB +RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives +a bogus 186.7 GiB (the Raphael iGPU is in the list — see +[[Llama Cpp Serving Sizing]] for the reading discipline). + +**Kernel gate — this decides which checkpoint can work at all.** +`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*` +but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with +BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup +otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`, +`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no +kernel (the class exists, the r4d.so entry does not); RFA has neither. Check +the checkpoint's quant against this list BEFORE downloading anything. + +Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve +"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`. +Known-good invocation: + +```bash +sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches' +podman run -d --name --ipc=host --ulimit memlock=-1:-1 \ + --cap-add SYS_PTRACE \ + --device /dev/kfd --device /dev/dri --device /dev/mem \ + -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \ + -v /home/dank/models/:/models:ro \ + docker.io/tcclaviger/vllm:latest /models \ + --served-model-name --port 7770 \ + --tensor-parallel-size 2 \ + --max-model-len 32768 \ + --kv-cache-memory-bytes 2147483648 \ + --max-num-seqs 4 \ + --gpu-memory-utilization 0.95 \ + --enable-prefix-caching --trust-remote-code +``` + +`-p 7770:7770` is not included, so the API is only reachable via +`podman exec` — add `-p` if host-side curl is needed. Load ~5 min; +`/health` answers connection-refused the whole time; watch for +`Application startup complete` in `podman logs `. Verified working: +`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131 +shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10, +multimodal, native 262,144). + +Traps that each cost a boot: + +1. **Mount the whole `/dev/dri`, never individual render nodes.** Render + nodes without the primary `card*` nodes → HIP enumerates two agents and + refuses a context on either (`RuntimeError: No CUDA GPUs are available`) + while `device_count()` cheerfully returns 2. Confirmed by direct A/B. +2. **memlock must be raised** (done on Dankserver via + `/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any + large pinned allocation fail as a misleading + `torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`; + it cannot be raised from inside a container or root's own shell — only + PAM applies it at login. Matters because `--expert-host-mlock` defaults + True. +3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** — + the planner sizes the arena against the budget and leaves KV the + remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed + at 0.64 GiB. +4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the + budget subtracts the KV reserve first, so a smaller context shrinks the + reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV; + 32768 → 0.34 GiB). +5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token + planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost). + Fix by pinning the pool: `--kv-cache-memory-bytes`. +6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena — + structurally orthogonal to `--expert-precise-memory` (verified + in-container 2026-10-05: precise-memory only swaps the GPU overhead term, + expert_offload_mem only sets the host bound; they rejoin only inside + `place_experts`, and the plan rewrites `expert_offload_mem` back to the + cap the plan met, so plan and allocation cannot drift). +7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one. +8. `podman run -d` without `-a` sends no stdout to a redirect — the log file + only receives the container ID; use `podman logs`. + +Status reads: use `scripts/dankstatus.sh`, never hand-rolled +`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a +TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and +awk prints `0.0` — a false "host out of memory" that invites an unnecessary +abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv +(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count +to the Hub's declared size; `-C -` resumes truncated shards. + +**Check whether the stack supports the model natively before porting an +overlay.** The image already registered `Glm5NextForConditionalGeneration`, +`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols +through two image tags to bring an overlay up was entirely wasted work. + +## Compile-cache persistence (survives `podman rm -f`) + +Serve scripts that recreate the container discard the writable layer, so +every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s` +cold, of which 173 s was one graph). + +1. **Prove it and locate it:** fresh container vs live + (`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs + `podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type + f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default + `/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds + `/root/.cache/comgr`. +2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists + both trees, safe whenever the image ships no such directory (step 1 + proves it; otherwise you would be hiding baked state). Wire as an opt-in + tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then + `${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the + checkout. +3. **Verify with two consecutive boots** — cold `Compiling a graph … takes + N s`, warm `Directly load the compiled graph(s) for compile range (…) + from the cache` with `torch.compile took` in single-digit seconds. +4. Traps: the cache key includes the compile RANGE — an MNBT change costs + ONE recompile, not a permanent tax; do not claim a proportional boot cut + (time to `/health` is dominated by weight loading: 490 s warm vs 188 s + cold compile); stale entries are misses, not wrong answers (version + + config hashed); if the serve script is seam-gated, assert the mkdir and + the exact mount target. + +## Plan-line verification checklist (after any fork deploy) + +``` +ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \ + | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"' +``` + +`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate +these columns to `17.`. Verify: new budget/graphs/arena in the plan; +`GPU KV cache size` matches the requested token count; `Estimated CUDA graph +memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture +exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the +real margin; `systemctl show frank-primary -p NRestarts --value` = 0; +`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat +completion from a PEER — `curl localhost:7770/health` inside an ssh session +on the host can return `000` and looks like an outage that isn't. + +## Related + +- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes +- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots +- [[LLM Bench Windows and Measurement]] — window discipline and bash laws +- [[Model Swap and Rollback]] — where the fork sits in the swap history +- [[Remote Job Runner]] — the job.sh harness referenced above diff --git a/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md b/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md index 85bbf53..4338c07 100644 --- a/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md +++ b/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md @@ -123,3 +123,27 @@ Stop and disable the root unit, trash it, restore the halogen quadlet from `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to `~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start; the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk. + +## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo) + +Why this deployment could not be tuned into a production endpoint: decode +3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at +1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts. +The decode root cause is structural: ~100 CPU↔GPU backend splits per token +with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4 +dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in +`pause` (ggml_barrier spin): threads wait on the serial critical path. +Memory placement (mmap/anon/THP/deferred) does not matter — all variants +landed 3.8–4.6. + +- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use + per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik + gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs + stay dead. +- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels + 4× slower than ik's iqk) — do not switch engines for speed. +- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the + NextN head is stripped (no `blk.48` tensors). +- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) → + decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx, + an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`. diff --git a/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md b/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md index 14eb9b0..1ed505b 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md +++ b/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md @@ -134,6 +134,59 @@ slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement 0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors. +## The split: re-measure it at the deployed shape + +The batch and the tensor split are **coupled**: compute buffers are not +symmetric across the two cards, so any batch change moves the balance point. +Order of work — sweep batch with llama-bench (above), THEN re-measure the +split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading +the real service with a per-config env override and reading `rocm-smi`. A +llama-bench number cannot decide the split: it has no `--parallel`, sizes +its own context as `n_prompt + n_gen + n_depth`, and cannot load the drafter. + +The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND +the port to be free (`ss -ltn | grep :7770` until gone) before the next +load, or the transient server dies with `couldn't bind HTTP server socket`. +Free bytes follow `card = weights_share + KV_share + constant + ubatch_term`, +where the ubatch term costs the drafter-carrying card roughly **1.85×** the +other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by +measurement only: one `-ts` step is one layer, worth ~0.43 GiB on a DeltaNet +boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is +about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91; +36,29 @512 → 6.01/7.32; **35,30 @512 → 6.44/6.90 (deployed)**. + +Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` at the new +deployed point and re-anchor the per-card constants; check neighbouring +splits still FAIL the balance gate or the seam has stopped discriminating +(keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy, +then verify live: flags in `/proc//cmdline`, `/slots`, VRAM matching +the pre-deploy measurement, a tool call, two concurrent requests, +`draft acceptance` lines on BOTH slots, 0 ROCm errors. + +**Per-card utilization (who is hot, and the split lever)** — a different +question from batch/prefill. Measure decode tok/s AND per-card +`rocm-smi --showuse` at 2–3 neighbouring splits in ONE session. Pitfalls +(each cost a re-run): run the transient unit as ROOT (a `pkill -9 +llama-server` as dank auto-restarts the service and `systemctl` as dank +fails "Access denied"); warm up 3 throwaway requests before the timed one +(first 384-token decode after load measured 1.77 tok/s against a true ~32); +`rocm-smi --showmeminfo vram --csv` columns are `$2`=Total, `$3`=**Used** — +parsing the wrong column spins the settle gate the full 600 s; the util% is +sampled post-hoc (which card finished last), so treat it as "which card is +the pipeline tail", not a duty cycle. + +**What the split does and does not buy (measured 2026-09-17, 393216 / +MTP n=3):** the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29 +all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at +35:30 because the MTP drafter + output layers + heavier compute buffers are +pinned to the card holding the LAST layers — not because PCIe is the +bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch). +36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput +cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing +preference is that the card with the FASTEST PCIe lane should be the more +utilized one; deploying it requires re-fitting the seam's slack/floor and +re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating. + ## Limitations - **`llama-bench` cannot run the drafter, the slots or the deployed context.** diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index 0d53e9e..a28c0a1 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -45,6 +45,98 @@ timestamp: 2026-09-30 > because the fork compiles its own Qwen trunk and not its GLM one. See > [The measured profile](#the-measured-profile-and-the-real-wall). +## DFlash2 target-side port — the four laws and the KV ladder + +Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder` +/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot. + +**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`): + +1. **Patch the instance the runner calls.** `model_runner.py:425` calls + `set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model — + GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper; + `self.language_model = Glm5NextForCausalLM`). Members placed only on the + CausalLM fail the check on the conditional — same error text, wrong file. + Patch BOTH wrappers. +2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is + `@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__` + (including base-protocol members) with `getattr_static`, breaking on the + first miss. Both wrappers need, as class attributes: + `supports_eagle3: ClassVar[Literal[True]] = True`, + `has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`. + Verify in-image with stock typing — no boot needed. +3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five + Protocol-derived bases; adding `SupportsEagle3` anywhere dies with + "Cannot create a consistent method resolution order" (sim-proven: the + ordering rejects it in every base position). The attribute route avoids + the MRO, so the two methods must be written by hand: the CausalLM + unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional + delegates to `self.language_model` (pixtral.py:498 pattern); + `get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`, + with the drafter config's `target_layer_ids` winning when present. +4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call + — GLM's loop carries the single-stream state entering the layer. + +**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`): + +1. EAGLE3 gate → presence-checked attrs on both wrappers (above). +2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads + whole token rows) but CAN grow block_size when the max page is an exact + multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to + the `lcm` of all MLA-family pages first (patch 02). +3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's + divisible branch grew KpoolTailSpec's block; its kernel pins block == + index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded` + instead of growing (patch 03). +4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why + GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs + exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend + `_get_kv_cache_groups_glm5_next` to partition + HiddenStateCacheSpec + drafter attention specs; the drafter forms its + own uniform group (per-group pages, NO unification → the MLA block + stays at the engine-auto-set 2,432 from interface.py:918, kpool guard + `block_size % 512 == 0` unstressed); layout returns a 10-tuple + (+draft_group, +hidden_groups); the builder sizes ONE shared backing + allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) = + aliased pool + appended draft/hidden tensors (patch 04). After 04 the + capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and + the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL + graphs on both TP ranks). +5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter + KV) + 5×hidden×2 (aux hidden cache) keyed on + `speculative_config.method in {dflash, dspark, extract_hidden_states}`. + NEVER key on `use_aux_hidden_state_outputs` — it lives on the + gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan + time, so the check silently fails and the reserve stays 13,600 + (patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 + + 5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB. + +**Verify cheaply before booting:** smoke-test the pure KV logic on CPU +torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`, +register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add +`from __future__ import annotations`, and run synthetic GLM+draft+hidden +spec sets through the patched grouping/layout/builder functions. A 7-minute +boot per hypothesis is the alternative. + +**Window discipline:** stop + mask `frank-primary` FIRST and gate on +MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner +reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError: +host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe` +to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once +under pipefail via SIGPIPE before log-save, losing the capture. The abort +path must restore prod (unmask + daemon-reload + reset-failed + start), +never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts +the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method": +"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's +echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH. +Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter +torch.compile is the frontier (~380 s). + +**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s +with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s +prefill. Reconcile with a clean single-request measurement before trusting +either. + Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same hardware — its conclusion was drawn from throughput arithmetic and is superseded by this page's kernel-time measurement), diff --git a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md index 730aab7..ece22a3 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md +++ b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md @@ -110,6 +110,32 @@ retires the lucebox user quadlet to dated home trash truly free, relinks + enables + restarts the root unit, and waits up to 1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs). +### Preflight the GGUF without downloading it + +GGUF metadata lives in the header and HF honours ranged GETs, so a few MB +answers every sizing question before a 13 GB transfer +(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64 +n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY += u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers +forget). Walk arrays by byte size even for a summary: a naive parser +desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read: +`general.architecture`, `.block_count`, `.context_length`, +`.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp` +build carries `blk..nextn.*`; `--spec-type draft-mtp` then loads the +head against the MAIN model's weights, and `--model-draft` would load the +file twice), `.head_count_kv`, `.key_length`/`value_length`, +`.full_attention_interval`, `.ssm.state_size`/`group_count`/ +`time_step_rank`, and for the projector `clip.projector_type` + +`clip.has_vision_encoder`. + +KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)): +only full-attention layers take KV +(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B, +q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot +buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the +MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size` +between slots — it does not multiply. + ## Verification (2026-09-26 deploy) - `/v1/models` reports `frank-primary`. diff --git a/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md b/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md index f3cfd93..97c56b0 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md +++ b/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md @@ -146,3 +146,70 @@ systemctl --user start frank-medium`) and stop/mask frank-primary. The _Filled in after the cutover: health, real completion, tool call, Open WebUI completion, measured prefill/decode, `systemctl --user is-enabled/is-active` state._ + +## Dual-R9700 follow-up (2026-09-28 probe era) + +After this single-card runbook was retired, the DS41 engine got a dual-R9700 +attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with +`unsupported model architecture 'deepseek41'`). Findings that outlive the +probe: + +**Budget both cards explicitly, or you waste half the box.** The engine +sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs +every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB +looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a +cache of things still streaming from NVMe. Pin all four: + +| env | card | meaning | +|---|---|---| +| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) | +| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) | +| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache | +| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them | + +Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB, +card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360. + +**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded +`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not +gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary +tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0 +secondary=hip:1`. It is the only lever that materially raises resident +expert count, and resident count is the entire bottleneck. Confirm every run +with `expert ownership:` and `hybrid experts ready:` lines. + +**The placement file must live under the mounted models dir.** JSON +`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values +`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device +keeps what fits its budget and demotes the rest. Anywhere outside +`/models/...` gives `cannot open expert placement` and a SILENT single-tier +fallback — a silent-wrongness failure, not a crash. + +**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill +exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode` +is refused there. With ~22% of experts resident each token pulls ~187 cold +experts × ~12 MiB; if `/proc//io` `read_bytes` climbs into terabytes +the model is larger than the box's RAM and you are in page-cache thrash at +the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util +with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound +decode, not a broken run. + +**Harness traps, each of which produced a wrong number:** never `env -i` the +podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes +PODMAN try `/root/.config` and die `stat /root/.config: permission denied` +before the container exists — pass container env with `-e`); never put a +large prompt on curl's argv (`Argument list too long` at ~128 KB — use +`--data-binary @file`); size prompts by growing the file until `wc -c` +reaches the byte target AND assert depth against the engine's own +`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop +that appends a fixed ~152-byte line per nominal 200-byte chunk undershot +30%); the server runs as the container's PID 1, so tear down with +`podman rm -f`, never host `pkill`; drop page cache between arms +(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and +every later one is a lie. + +Read the placement log — it is the whole diagnosis +(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304 +hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14] +… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never +mapped, so they do not evict the expert page cache). diff --git a/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md index 67851e3..4f666d3 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md +++ b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md @@ -190,6 +190,70 @@ repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified on :7770 (`--enable-prefix-caching` in the container argv, health 200, live completion answering). +## Authoring a new profile (the qualification pipeline) + +Order matters at every step: + +1. **Package manifest** `packages/models//package.json` — target shards + + aux artifacts with sha256. Verify every target's byte count AND sha256 + after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`). +2. **Placement bootstrap** from GGUF headers: parse all shards' tensor + tables (a KV-only shard1 is normal for split parts; each part carries its + own tensor table). Per-expert full per layer = + (`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices + rank0 = full×416//640, rank1 = full−rank0. Hot counts = + floor(measured_budget / per-expert total). **Anchor budgets to the + MEASURED hot_bytes of the qualified sibling profile, never invented.** +3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then + `expert_budget.validate_manifest(m)` + + `expert_budget.expert_memory(m, slots, {cache_rank}, runtime=)` + (runtime/catalog are keyword-only). Declared hot_bytes must match + computed or it raises. +4. **PLE derive** (containerized `tools/prepare_ple.py` with + `--expected-bytes/--expected-type/--expected-shape`); setup skips + re-derivation when the file exists with matching size. +5. **profile.env**: set values BEFORE `source dual-r9700/profile.env` + (set-first-wins; an empty string does NOT survive `:=`). + +The canonical bootstrap.json wrapper schema is exactly +`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path, +artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the +wrapper `id` MUST equal the id in the profile.json placement descriptor +(setup rejects mismatches: "does not match profile placement"). Never +fabricate qualification metadata — use +`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`. + +GGUF parsing laws for the bootstrap: expert tensor names are +`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is +parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes +`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte +extents from consecutive OFFSETS, never a type-size table (mis-buckets tails +silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536), +54,400,261,120 B, an entire shard. + +Setup/start: `./r9v setup --state-dir --model-dir --image +sha256: --local-image --reuse-from +--accept-model-license` (aux artifacts reuse byte-identical files; F16 vs +Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed +placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises. +The host-only doctor FAILs when another instance holds the cards (it wants +3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second +instance CANNOT coexist, so qualification requires a stop+mask window for +the incumbent, then unmask+restart. + +**Qualification gate (unqualified → qualified):** first start runs the +workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning +proof (`VmLck` ≈ PLE bytes in `/proc//status` when +RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If +headroom is violated, lower hot counts via re-trim and re-derive +MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook. + +Remote-execution gotchas from the same windows: `nohup X &` inside an ssh +command races ssh teardown — the script can die before writing a byte; run +long remote scripts synchronously, or setsid + disown, or via a system +unit. zsh login shells abort on unquoted `===` separators; grep globs need +quoting through ssh+zsh layers. + ## Ops - **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 4d2586a..866f20a 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -407,6 +407,42 @@ frank-small, port 7771] - [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations - [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo) +## Cross-Cutting References (skill distillation, 2026-10-05) + +One polished reference page per skill-cluster, distilled from the managed +skill library into the vault (see [log](../log.md) for the wave plan): + +- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand + of a loader; how the runtime is chosen by the file +- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape + laddering, llama-bench discipline +- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot` + laws, drafter placement +- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the + heterogeneous-pair verdicts +- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers +- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena + lever, and why free VRAM is not arena +- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that + died, and interrogating a running fork's config +- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM + versions/forks; torch.compile/dynamo breakage triage +- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B, + the window script laws +- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live + endpoints, IBS attribution, prefix-cache routing +- [[Model Swap and Rollback]] — the swap procedure, payload preflight, + research verification, second-endpoint bring-up +- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob + verdicts, large-MoE recipe, compile-cache persistence +- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split + proposals (the no-go verdict format) +- [[FreeToken Builds]] — the CUDA and ROCm build paths with their + dependency traps + +These sit alongside the operational pages under +`../Operations/index.md` and `../Agent/index.md`. + ## Related - [Project knowledge base](../index.md) — the Podman app stacks on the same diff --git a/serverAppConfigs-docs/Operations/Container Host Reachability.md b/serverAppConfigs-docs/Operations/Container Host Reachability.md new file mode 100644 index 0000000..22fc58c --- /dev/null +++ b/serverAppConfigs-docs/Operations/Container Host Reachability.md @@ -0,0 +1,128 @@ +--- +type: reference +title: Container Host Reachability +description: The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate. +tags: [podman, netavark, networking, rootless, reachability, caddy] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Container Host Reachability + +Two failure families on rootless podman + netavark hosts (Dankclaw), both +reading as "service down" when the service is up and the *path* is wrong. +Fleet source of truth: `serverAppConfigs-docs/Architecture/Container Network +Namespaces.md` (commit `2e179b0`). + +## Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly + +A container cannot reach a `Network=host` service at the host's LAN IP. +netavark DNATs **published** ports only: from a bridge container, +`192.168.0.143:` answers while `192.168.0.143:` and the +gateway `10.89.1.1` are both refused. This is what 502'd +`hermes.franksdank.cloud` until the Caddyfile upstream became +`host.containers.internal` on 2026-09-16 — its vhost comment had claimed the +LAN IP was open from that bridge; it never was. + +Probe that proves it in one command: + +```bash +ssh "podman run --rm --network --entrypoint sh \ + docker.io/library/busybox:latest -c \ + 'for p in ; do + nc -w2 -z -v \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD + done'" +podman exec bash -c 'exec 3<>/dev/tcp/host.containers.internal/ && echo ALIAS-OK' +``` + +Published OK + target BAD ⇒ quirk confirmed. Note busybox `nc -z` reports any +failure as failure — use it for bulk sweeps; use `/dev/tcp` (or `nc -v`) when +refusal-vs-timeout matters. + +Fix: point the service's host-facing URL at +`http://host.containers.internal:` in compose/quadlet env, redeploy via +git (`deploy.sh --service `), verify +`podman exec printenv ` and the connect log line (~40 s). +**Do not publish the daemon port** to make the LAN IP work — publishing is +exactly what makes other ports refuse. + +## Quirk 2 — addressability is per netns, not per host + +- `NetworkMode=host` → the container **is** the host netns: dials all of + `127.0.0.1`, binds any free port. +- bridge/named net → own netns; only published ports are reachable, via the + network's **gateway IP**. Unpublished host listeners refuse **instantly + (RST)** — no firewall rule, no stale DNAT, `nft`/`iptables -t nat` empty. Do + not hunt for REJECT rules; probe. + +Probe sequence (read-only): + +```bash +podman inspect --format '{{.HostConfig.NetworkMode}}' +sudo ss -ltnp 'sport = :' # host netns only — why 0.0.0.0 can still be unreachable +podman exec cat /proc/net/route # default route = the gateway IP to dial +podman exec sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp//"; echo rc=$?' +``` + +Interpretation without blurring: + +- **instant refusal** = nothing listening in a reachable namespace (or + rootlessport rejected an unpublished port map). +- **timeout** = no route to that address at all. `host.containers.internal` + (`169.254.1.2`) is an alias on the **default `podman` net only**; containers + on a named network time out against it. + +Tool gotchas (podman 5.x, verified live): + +- `podman network inspect --format '{{json .containers}}'` errors + (`can't evaluate field containers in type interface {}`) — list membership + from the container side: `podman ps -a --format '{{.Names}} {{.Networks}}'`. +- An **empty** `Networks` field = host network or shared namespace + (`--network=container:`), not "no network". +- Quadlet `Network=.network` yields the runtime name **`systemd-`** + — probe with `podman network ls`, never the `.network` filename. + +Fix options when a bridge-net container must reach a host-netns listener: +(a) move it to `Network=host`, or (b) publish the port — (b) makes the port +reachable from every namespace **and the LAN**, an exposure change needing +explicit operator approval, never a silent fix. + +## Related trap — the pasta port gate (rootless published ports) + +On hosts publishing ports through pasta, the port is bound in **userspace at +container CREATE**, not when the app binds. A predecessor stopped moments +earlier can still hold listener/TIME_WAIT state, so the replacement dies +`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals +seconds later** under `Restart=on-failure`. The failure looks like it failed, +then looks fine, and the missing gate stays invisible because the retry masks +it. Observed 2026-09-16 cutting `frank-large` from a root unit to a rootless +Quadlet. + +Gate procedure: stop the old unit AND the new one (a redeploy's running +replacement is itself the port-holder — without that stop the gate deadlocks +against itself), then wait for the port to be empty across **all socket +states**: + +```bash +wait_port_free() { + local port="$1" limit="${2:-90}" i + for ((i = 1; i <= limit; i++)); do + if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi + sleep 1 + done + echo "FAIL: port $port still held after ${limit}s:" >&2 + ss -tan "sport = :$port" >&2 || true + return 1 +} +``` + +`sport` catches accepted-connection server sockets and TIME_WAIT residue that +`ss -ltn` misses; fail loudly with the socket table attached. Verify the gate +by re-running the same deploy against a running service (idempotency) and +assert it in seam tests (`wait_port_free` present in the deploy handler). + +## Related + +- [[Quadlet Lifecycle]] — the unit layer these containers run under +- [[Container Network Namespaces]] — fleet network namespace source of truth +- [[Observability]] — the alerting paths that depend on reachability +- [[Deploy State Triage]] — the deploy flow that installs these configs diff --git a/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md b/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md new file mode 100644 index 0000000..c0af835 --- /dev/null +++ b/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md @@ -0,0 +1,331 @@ +--- +type: reference +title: Fleet Audit and Posture +description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work. +tags: [audit, security, prometheus, caddy, monitoring, git] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Fleet Audit and Posture + +## Monitoring gap audit + +Use after any reboot, or whenever Prometheus targets look wrong. Two failure +modes, only the first announces itself. + +**1. A service that should exist and doesn't is invisible.** A missing +exporter shows up as nothing — no failing target if the job was configured +for a host that no longer serves, no alert because the series is absent. So +check **existence, not health**: + +```bash +podman ps -a --format '{{.Names}} {{.Status}}' # includes dead ones +ls ~/.config/containers/systemd/*.container # what should be boot-persistent +systemctl --user is-enabled .service # expect: generated +``` + +A container in `ps -a` that is `Exited(…)` **and has no unit** is the classic +case: rootless podman does not restart bare containers after the stop that +shutdown performs, so a compose-only service dies at the first reboot and +stays dead. If the compose file still names it, the repo's own rule is being +violated — fix by writing the quadlet (mirror the same service on the other +host), deleting the compose entry (one owner per container name), and making +the deploy path reach it: planner service-set + path rule, dispatcher +allowlist `:`, handler installing from `"$TARGET/..."` never +a hardcoded peer's path. + +**2. Classify every down target — do not report a count.** +`targets?state=active` gives health **and** last error; two down targets can +mean opposite things: + +| Symptom | Meaning | +|---|---| +| `connection refused` | service not listening → real outage (podman ps, logs, exit codes) | +| `404` on `/metrics` | exporter path wrong or metrics disabled → config defect, service fine | +| `unsupported Content-Type …` | scraping an HTML *page* as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter | +| target absent | §1 | + +```bash +curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c " +import json,sys +for t in json.load(sys.stdin)['data']['activeTargets']: + if t['health'] != 'up': + print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])" +curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http:///metrics # hand-probe before believing the scraper +``` + +"7 of 11 up" without the four reasons is how a real outage hid behind three +config defects on this fleet — the crash-looping GPU service was one of four +failures reported as a number. + +**3. Exposure posture while you are in there.** An exporter bound to the +wildcard publishes an unauthenticated API on every interface. Expect exactly +two listeners per host-exporter — loopback for host-local probes, the host's +own LAN address for the Prometheus container: `ss -ltn | grep ` must +show two explicit addresses, never `0.0.0.0`. + +**4. Leave decisions that are not yours.** Fix mechanical drift (missing +unit, wrong path, wildcard bind). Do NOT disable a probe to green a +dashboard, or "fix" a crash-looping service whose posture depends on owner +intent — capture the evidence (restart count, exit code, last error line) and +hand it over intact. + +## vhost proxy health probes + +After deploying or auditing a Caddy/Anubis front door, verify **from a +client**, not from the backend's localhost — that is where transport +mismatches, auth walls, and dead upstreams show up. + +``` +curl -s -o /dev/null -m 6 -w "%{http_code}\n" https:///ping +curl -s -o /dev/null -m 6 -w "%{http_code}\n" https:/// +``` + +A status code is a diagnosis, not a failure: + +- `200` on `/ping` from *arrs = alive; **auth state unknown** (`/ping` + bypasses auth by design) — probe `/` to decide. +- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404 + there = healthy). +- `502` = Caddy could not reach/negotiate with the upstream: dead backend + **or transport mismatch**. Probe the backend port directly from the proxy's + network (`ssh curl http://:/…`). Direct 200 + + vhost 502 ⇒ config bug, not an outage. +- Login page to anonymous (grep body for `password`) = auth ON; instant + redirect to `/transferlist/`-style routes = auth OFF = open door. + +**Transport-mismatch mechanism:** `transport http {versions h2c 1.1}` makes +Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it +→ 502 per request while the same port answers 200 on plain HTTP/1.1. Fix = +drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain +HTTP off-loopback and needs a TLS transport. Match the transport to what the +backend speaks. + +**Shell trap when gating commits on a test suite:** `grep -c "^not ok"` exits +**1 when zero mismatches** — inside an `&&` chain it silently kills +everything after it while still printing `0`. Wrap: +`(./tests/run.sh | grep -cE '^not ok' || true) && git commit …`. + +## Security posture audit (behavioral) + +Answers "what can the internet touch, and is everything else actually +gated?" — **all findings must be behavioral probes**, not config reads. + +**1. Inventory (source of truth).** `dankclaw/Caddyfile`: classify every +vhost — guarded (`reverse_proxy anubis:8923`), exempt (direct upstream), or +redirect-only. Public DNS: `dig +short +time=3 +tries=1 .franksdank.cloud +@1.1.1.1` per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by +DNS absence. + +**2. Guard chain (expect the challenge, not the app).** Per guarded vhost: +`curl -sk https:///` and grep the body for +`anubis|challenge|proof-of-work` (~8 markers on a live challenge page). A 200 +with NO markers means the app is served directly — the guard is bypassed. + +**3. Exempt vhosts — per-app gating probes.** OpenWebUI `/api/v1/users/me` → +401. Gitea `/api/v1/user` → 401; repo visibility `/api/v1/repos/dank/` +→ 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex `/web/` +→ 302 (login is JS-rendered). Calibre-Web `/` → 302 to `/login`. Jellyfin +`/Users` → 401. Prowlarr/Sonarr/Radarr `/api/v3/system/status`; Lidarr/Readarr +`/api/v1/system/status` → 401 = gated, 200 = fully open. qBittorrent +`/api/v2/torrents/info` → 403 without SID. Hermes/hermesadmin `/` → 302 to +`/login`. IdP (`auth.`): `/` → 302 → `/if/flow/default-authentication-flow/` +→ 200 (flow HTML is React; grep "password" finds nothing — the 200 on the +flow URL is the proof). + +**4. WAN port scan — from an EXTERNAL vantage.** LAN→WAN hairpin is +unreliable; scan from gluetun's ProtonVPN egress: + +```bash +ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \ + 8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \ + 5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \ + >/dev/null 2>&1 && echo OPEN: \$p; done"' +``` + +Confirm the egress is truly external first (`wget -qO- https://api.ipify.org` +→ ProtonVPN IP, not Comcast). **Trap: plain-HTTP probes against TLS ports +(443, 32400) are false negatives** — port open, protocol mismatch; re-probe +with `wget -qO- https://IP[:port]/`. busybox wget has no `--secure` flag; it +errors instantly and the `&&` chain silently skips the echo (a sub-second +wall time on a "timeout" scan means the flag failed, not that ports closed). + +**5. Residuals.** Grafana basic-auth residual: `curl -u admin:admin +http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live +(loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds: +`sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no +auth (OpenAI API, no token). Stale-record check: current WAN IP = +`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress); +compare against every public A record — a record at a different Comcast IP +is stale. + +**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`, +`search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all +verified) + `auth` (IdP, by design); everything else LAN-only. Open findings: +LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback +residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP); +`plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin +192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener +(dead config, no exposure). + +## Stash supersession audit before drop + +Use when a repo (especially a live-deployment checkout where the working tree +IS the deployment) has an old stash whose content may have been superseded by +later commits, and dropping blindly risks losing unique facts. + +1. **Inventory**: `git stash show --stat stash@{0}`. +2. **Probe for unique content per file**: `git show "stash@{0}:"` and + diff against HEAD; count lines present only in the stash version. A file + with zero unique-vs-HEAD lines is fully superseded — safe to drop. +3. **Classify each unique-content hit**: *stale fact* (superseded by a later + refactor, e.g. old model naming) → drop is correct; *live durable fact* + (an operational invariant never re-recorded) → extract it into live + docs/memory or flag to the user before dropping. +4. **Archive before dropping** — zero loss: tag the stash commit, then drop: + +```bash +git tag archive/stash---wip "stash@{0}" +git stash drop "stash@{0}" +``` + +The tag pins the full stash object including its index parent; recoverable +via `git show archive/stash-…` or `git checkout archive/stash-…^3 -- ` +for the untracked-files third parent. Stashes are invisible in normal history +and expire from reflogs (~90 days); tagging converts "drop" into a +reversible archive at zero workflow cost. + +## Whole-fleet audit (git/test/live/docs cross-check) + +Deep-audit the homelab fleet — git/test state, live service health, and +docs-vs-live cross-check. Run from the dev checkout +(`/home/blank/Projects/serverAppConfig`). + +**1. Probe the live fleet — self-identifying, ONE ssh per host.** Do NOT +trust the display order of parallel SSH outputs: they can render crossed and +you will attribute host A's output to host B (2026-09-21: a false host↔IP +"conflict" dissolved by a single self-contained per-host probe). Put +`hostname` + a distinctive marker + the LLM identity in the SAME command: + +``` +for ip in 192.168.0.143 192.168.0.235; do + ssh -o BatchMode=yes dank@$ip \ + 'echo "host=$(hostname)"; \ + echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \ + echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo' +done +``` + +Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw += AI stack + `frank-large` :7770 + `frank-small` :7771 (RTX 3090 + Strix +Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary` +:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is +hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so +it is unaffected by IP drift. + +**2. LLM endpoints — health + identity.** `curl /health` per endpoint, then +`/v1/models` (first 120 chars) — a 200 with the right model `id`/`name` is +the pass condition. + +**3. Tests — the deploy completeness gate.** `./tests/run.sh` (bats) must +end `ALL TESTS PASSED`; `tests/e2e/run.sh` runs Playwright against the LIVE +fleet. **A single-test flake is normal**, not a regression: the plex crawl +intermittently lands on `app.plex.tv`, the grafana SSO test occasionally +times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just +that spec (`npx playwright test --project=standard specs/-.spec.ts`) +to classify before alarming. A `deploy.sh` e2e gate can fail on a flake +while the change itself is fine — verify the actual change live, then re-run +the suite. + +**4. Repo state.** `git status --short --branch; git log --oneline -20; +git stash list; git worktree list` (prune stale worktrees), and +`du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a +**shared checkout with parallel instances**: uncommitted peer WIP and +stash-list churn are expected. Never commit or drop a peer's files. To +deploy with a dirty tree: `git stash push -u -m "..." -- `, +deploy, `git stash pop`. `deploy.sh` refuses a dirty tree. + +**5. Prometheus — expected-down targets (not outages).** Query +`/api/v1?query=up` from inside the container; the structurally red-by- +construction jobs (`comfyui` HTML scrape, `services` up/down-as-scrape, +`gitea` exporter off) are documented in [[Observability]] — anything else +down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a +same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is +a *host bind*, not a published port, so the bridge-networked Prometheus +container cannot reach it at the host's LAN IP — netavark DNATs published +ports only. Scrape same-host services via +`host.containers.internal:`; verify the fix from inside the container +before editing: `podman exec prometheus wget -qO- +http://host.containers.internal:/metrics`. + +**6. Cross-check docs vs live.** Confirm the AGENTS.md host↔IP↔hardware↔ +stack mapping against step 1. The LLM endpoint table and `models.yml` are +probed live and stay current; the prose host labels are the part that goes +stale after an IP move. Report discrepancies; do not "fix" prose from a +single crossed probe. + +## Bulk-redact an identity from a docs + config tree + +Removing a retired host / machine / project name from a repo that holds both +documentation and live config. Evidence-first, surgical, reversible. + +**1. Inventory before touching anything.** Count hits **per file** (the +count decides whether this is a 5-minute job or a user-input decision), +excluding logs/`*.db`/runtime state, and capture `git status --short` first +— you must know which files carry in-flight peer WIP so you never stage +them. + +**2. Classify every hit:** + +| Bucket | Test | Action | +|---|---|---| +| **Narrative** | prose describing the thing | Reword to the class ("the workstation", "a client box") | +| **Load-bearing** | the identifier IS a lookup key, hostname match, or config value | **Keep.** Prove it: read the code that consumes it | +| **Evidence** | quoted logs, transcripts, reports | Keep timestamps/lines; de-identify only the host field | + +Load-bearing examples that look like prose but are not: `install.sh` doing +`PKGS=("common.pkgs" "$HOST.pkgs")` with `HOST="$(hostname)"` (the manifest +*filename* is the key); a `--host ` case list; a Wi-Fi SSID. Hardware +model strings, driver versions, PCI addresses are subject matter, not +identity. State the boundary in the commit message. + +**3. Subdirectories: distill, don't rename or keep.** For a per-host doc +tree (`docs/system//`), ask before acting — rename/trash/distill are +genuinely different. When distilling: read every page, sort into +still-true-and-machine-independent (fold into a general page) vs +specific-to-the-retired-box (let go), then `mv` the directory to +`.AGENTTRASH/YYYYMMDD-topic/` and repoint every index. Wikilinks resolve by +filename anywhere in the tree, so deleting a page can break a link in a +file you never grepped — run the link checker. + +**4. Execute with count assertions, never blind sed.** Per-file mapping of +`(path, old_exact_string, new_string, expected_count)`; abort the entry if +the count differs; exact strings, not regex. **Do not hand this to a +subagent** — observed: dropped list markers so bullets stopped rendering, a +numbered list with two `3.` items, "corrected" paths to nonexistent +directories, a deleted sentence where a reword was required. If a subagent +already did it: `git checkout --` its touched paths (excluding the WIP +files from step 1) and redo it. + +**5. Audit:** `grep -rniE '|' ` (expect nothing), then +`git diff -U0 -- | grep -E '^\+[^+]'` and **read every added line** +— the diff is the step that catches damage; the grep only proves a string +is gone. Then the repo's own linter. + +**6. Commit scoped** — only your own paths, never `git add -A`. If the +checkout IS the live deployment, committing on the current branch is usually +correct; a branch switch can swap live config out from under a running +process. Push/PR only when asked. + +**7. Report the residue honestly** — what still carries the name and why: +excluded scopes, load-bearing config, git history, and the cost of rewriting +history (force-push + re-clones). "Purged from the working tree" is a true +claim; "purged" alone is not. + +## Related + +- [[Observability]] — the monitored stack +- [[Stale Signal Audit]] — the down-target decision table +- [[Observability Provisioning]] — adding scrape targets and rules +- [[Homepage Status Probes]] — the status page that renders probe states +- [[Deploy State Triage]] — the deploy plumbing the audits exercise diff --git a/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md b/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md new file mode 100644 index 0000000..f46533d --- /dev/null +++ b/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md @@ -0,0 +1,272 @@ +--- +type: reference +title: Fleet Ops Playbook +description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host. +tags: [operations, incidents, gpu, systemd, wol, forensics] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Fleet Ops Playbook + +## Endpoint down, connection refused — GPU holder reclaim + +Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the +service was not crashed. Applies to `frank-medium` (dankserver, root unit), +`frank-small` (dankclaw, root unit), `frank-large` (dankclaw, rootless +Quadlet). + +**1. Classify the stop before touching anything** + +```bash +ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \ + systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts" +ls -l /etc/systemd/system/.service # symlink to repo = unmasked; -> /dev/null = masked +``` + +`inactive` + `enabled` + unmasked + `NRestarts=0` → **intentional stop**, +never restored. No crash; do not look for segfaults. `NRestarts>0` / +`activating` → different path; read the journal first. + +**2. Are the GPUs actually free?** A check, never an assumption: + +```bash +rocm-smi --showmeminfo vram | grep Used # idle baseline ~60 MiB/card +ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep +``` + +Cards idle → `systemctl start ` and verify (step 5). Cards busy with no +prod unit → get the holder, and compare its bytes against the service's +requirement before concluding a start would even fit (frank-medium 35:30 +needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the +restart on allocation, not config). + +**3. Identify the holder and prove it is abandoned** + +```bash +ps -o pid,ppid,user,lstart,etime,comm -p +sudo readlink /proc//cwd /proc//exe +tr '\0' ' ' < /proc//cmdline # port, ctx, split +sudo ls -l /proc//fd # where stdout/stderr go +sudo ss -tnp | grep +awk '{print $14+$15}' /proc//stat; sleep 10; awk '{print $14+$15}' /proc//stat +``` + +Abandoned **only if all** hold: utime+stime **identical** across the 10 s +sample (frozen CPU, not just low); **zero** established connections; bind +scope `127.0.0.1` (loopback scratch, invisible to endpoint monitors); log +file (fd 1/2 target) last written long ago, last lines showing success not a +hang. Active CPU or live connections → someone's running session. Stop and +ask. + +**4. Reclaim** + +```bash +cp -a .preserved-YYYYMMDD # evidence outlives the kill +kill # SIGTERM first; exits <1 s +rocm-smi --showmeminfo vram | grep Used # expect ~60 MiB baseline +``` + +Never `pkill -f` a pattern present in your own command line. + +**5. Restore and verify — health 200 alone is NOT proof** + +```bash +sudo systemctl start +for i in $(seq 1 60); do curl -s -m 5 -o /dev/null -w '%{http_code}\n' \ + http://127.0.0.1:/health; sleep 10; done # 503 while loading -> 200 +sudo tr '\0' ' ' < /proc//cmdline | grep -oE '\-\-(tensor-split|split-mode|ctx-size|port|spec-type|parallel|flash-attn)' +``` + +Prove with a **real completion** from off-host — content, reasoning, spec +counters: + +```bash +curl -sS -m 180 http://:/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":64,"temperature":0}' +# choices[0].message.content / .reasoning_content / timings.draft_n, draft_n_accepted +``` + +Record the runtime fingerprint (` --version` → `b, commit `) +and confirm it is the pinned runtime. Append a dated entry to +`serverAppConfigs-docs/log.md`, run `./tests/run.sh`, commit, push (origin is +the local Gitea on Dankclaw — push allowed). + +**Gotchas seen here:** dankserver's login shell is ZSH — never start an +unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A +freshly restarted endpoint under live agent-queue backlog may return empty +bodies/timeouts for minutes while `/health` is 200 — re-probe with a long +timeout. First request after a halogen (`frank-large`) restart can go silent +45–60 s and self-recover; do not restart over it. Rootless `frank-large`: +read logs **without** sudo (per-user storage). If the restore never happened, +the sudo journal names who and when: `journalctl -b | grep sudo`. + +## Scratch vLLM boot on a fleet GPU host — stop-first law + +Any scratch/experimental vLLM container on a host that also runs a production +LLM service (`frank-primary`, `frank-large`, `frank-small`) MUST stop the +production unit first, and the window script must verify it. Two independent +failure modes if skipped: + +1. **Expert-offload planner dies on host RAM.** + `vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at + plan time. A production rank worker holds tens of GiB of unevictable + pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod + boot sees ~86 GiB instead of ~239 and dies at plan time: + `ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`. + Looks like a planner bug; is pure RAM contention. Signature: container + exits within ~60 s, after `Resolved architecture: …` but before worker + init; `free -g` showing `available ≈ 86` while prod runs is the smoking + gun. +2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless + (the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine + only once the prod unit is down. + +Window skeleton (dankserver): + +```bash +sudo systemctl stop frank-primary.service +sudo systemctl mask --runtime frank-primary.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot +sleep 3 +podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; } +# ... launch scratch container, watch logs ... +podman rm -f # stuck vLLM ignores SIGTERM +sudo systemctl unmask --runtime frank-primary.service +sudo systemctl reset-failed frank-primary.service +sudo systemctl start frank-primary.service +# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink +``` + +The deploy restores the unit after bench windows, so stop is sufficient +there; `mask --runtime` is for scratch boots that must survive a stray +restart. Always end a window by re-checking the conjunction before +reporting — if a restore after failure doesn't happen automatically, prod +stays DOWN. See the repo's benchmarking policy (AGENTS.md) and +[[LLM Bench Windows and Measurement]]. + +## Iterating on a fleet service before commit + +The repo's contract is git → `deploy.sh` → host, but live testing needs the +new file on the host first, and the naive way (copy into the host checkout) +blocks the very deploy that would install it. The loop: + +1. Edit locally; run `./tests/run.sh`. +2. Ship working files **for testing**: + `cat local/file | ssh 'cat > /home/dank/serverAppConfigs/'` + (then `chmod +x` if executed — `cat >`/scp drop the exec bit and a + container entrypoint fails status 126, "found but not executable"). +3. Test against the live service; iterate. +4. **Before committing**, prove the host copy is byte-identical to what you + are about to commit (`sha256sum` both ends) — otherwise step 6 loses work. +5. Commit and push. +6. Clean the host checkout so the deploy can pull: untracked hand copies → + `mv` to `/tmp/handcopy/`; modified tracked files → `git checkout -- ` + (safe only because of step 4); confirm `git status --short` prints nothing. +7. `./scripts/deploy.sh --service `. + +Traps, in the order they bite: the **pull aborts on hand copies** ("untracked +working tree files would be overwritten by merge" — exactly the file you +shipped); `git checkout --` silently eats uncommitted work without step 4; a +runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`, +`shared/grafana/provisioning/**`) so a hand copy takes effect at next +container restart with no deploy; root-owned files from a container write +need `chown` back before git operations; and **a restart is not a +verification** — verify the *specific* claim (new code path's own output, +file checksum on the host, the API serving the resource). + +Why not keep a scratch copy on the host? The deployed artifact must be the +committed one. Probes may live anywhere; runtime state lives in git. + +## Incident window reconstruction + +Use when the question is **"what was running when it broke?"** and the answer +must come from artifacts, not recollection. Three independent clocks, +cross-checked: + +1. **Per-run duration** — scripts wrap commands in `time` → `real 4m19.129s`. +2. **Log mtime** — last write of the log. +3. **Journal first-failure instant** — first error line from the affected + service. + +`start ≈ mtime − real` for the last run in a log; each run nests inside the +previous one's window; the run whose `[start, end]` contains the journal's +first-failure instant is the answer. + +```bash +ssh 'ls -l --time-style=+%m-%d_%H:%M:%S /*.log | awk "{print \$6, \$7}"' +ssh 'grep -ah "^real" /*.log' +ssh 'grep -aE "^####|^\+ /path/to/binary" /.log | sed "s/\(.\{130\}\)/\1\n /g"' +ssh 'journalctl --user --since " " | grep -a "invalid internal status" | head -2' +``` + +Traps (all hit for real): `/tmp` does not survive a reboot — launch +experiment scripts from a persistent path (`/home/dank/models/logs/.log` +survived while `/tmp/x4*.sh` died); a log's mtime is not proof a run finished +(truncation with the fd still held keeps growing); long command lines are +display-truncated — wrap before reading or you report parameters never used; +off-by-one filenames (`x4b.log` vs `x4b-experiment.log`) silently skip +evidence — list the directory and match exactly; **overlap is a hypothesis, +not an observation** unless a sampler proves it; do not invent the cause — +reconstruct the *window* and keep the *mechanism* separate (`journalctl -k | +grep -i oom`); publish "trigger not established" when the proof is absent. + +Fleet-specific: container healthchecks shell out to the **podman CLI**, so +the first `invalid internal status … could not find any running process` in +the user journal is an early, per-minute indicator of rootless-podman state +death — it timestamps the incident to within one healthcheck interval (how +the 18:58:30 Dankclaw break was pinned to a bench still loading). + +Worked example (2026-09-16, Dankclaw): log mtimes x4e-run 18:58:29 / +x4f-run 18:59:56; durations E1 `real 4m53.946s`, F `real 5m24.785s` (exit 143 += killed); journal first failure 18:58:30 ⇒ **F** was in flight (launched +18:54:31, inside its window). + +## Wake-on-LAN for a sleeping fleet host + +A fleet host is asleep (100% ping loss; `ip neigh show ` shows STALE or +FAILED) and you need it up — typically the FrankBook-Pro Mac mini +(192.168.0.112, **MAC `de:6b:6d:c1:52:23`**, sleeps and breaks the e2e +`11-admin` spec). + +- **Find the MAC**: `ip neigh show ` — even a STALE entry carries it; + fallback Pi-hole/DHCP leases or fleet docs. +- **Send the packet**: `6×0xFF + 16×MAC` = 102 bytes, UDP to port **9** (and + 7), broadcast. The dev workstation has **two interfaces on 192.168.0.0/24** + (Wi-Fi + wired USB-Ethernet, default route metric 100) — a plain + `sendto("255.255.255.255", 9)` leaves only the default-route interface. + Bind explicitly to each interface's local IP and target the subnet + broadcast `192.168.0.255` as well as `255.255.255.255`; burst ~15 packets + per interface ~0.2 s apart. `wakeonlan` is not installed; use a raw UDP + socket with `SO_BROADCAST=1`: + +```python +import socket, time +MAC = "de:6b:6d:c1:52:23" +mac = bytes(int(x,16) for x in MAC.split(":")) +pkt = b"\xff"*6 + mac*16 # 102 bytes +for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"), + ("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]: + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) + s.bind((local, 0)) + for _ in range(15): + s.sendto(pkt,(dest,9)); s.sendto(pkt,(dest,7)); time.sleep(0.2) + s.close() +``` + +- **Poll for the wake**: a Mac mini takes **30–90 s**; `ping -c 1 -W 1` every + 5 s for ~2–3 min before concluding failure. +- **If it does not wake** (the common case): macOS honors WoL only when BOTH + are enabled on the mini — **System Settings → Battery → Options → "Wake + for network access"** = ON, and the active port's **"Enable Wake for LAN"** + (Network → [interface] → Details). If either is off, no packet from any + source wakes it, and you cannot change it while asleep. A full power-off + (smart plug cycled) ignores WoL entirely. Ask the user to toggle the + setting, then re-fire. + +## Related + +- [[LLM Bench Windows and Measurement]] — the bench windows this playbook arms +- [[Remote Job Runner]] — supervising the long jobs that create these windows +- [[Container Host Reachability]] — the pasta port gate at container create +- [[Quadlet Lifecycle]] — the unit layer behind stop/mask/start +- [[Model Swap and Rollback]] — swap-time incidents and rollbacks diff --git a/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md b/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md new file mode 100644 index 0000000..019040d --- /dev/null +++ b/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md @@ -0,0 +1,204 @@ +--- +type: reference +title: Payload and Disk Reclaim +description: Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index, prove nothing live references the payload, stage to trash with fingerprints, purge only on approval; plus the purge-and-refresh protocol with fetch verification. +tags: [operations, disk, podman, payloads, huggingface, trash] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Payload and Disk Reclaim + +Standing procedure for "clean up old models" / "disk is full" on a +serverAppConfigs host. Order matters: **measure, prove nothing live uses it, +stage, ask.** Staging frees nothing until the user approves the purge — say so +up front or the request reads as unfulfilled. + +## 1. Measure the whole filesystem, not the category asked about + +```sh +du -sh /home/dank/models /home/dank/.local /home/dank/.cache \ + /home/dank/comfyui-data /home/dank/GitPackages 2>/dev/null +df -h /home/dank | tail -1 +podman system df # reclaimable image space +du -sh ~/.local/share/containers # where rootless storage physically is +``` + +The answer is usually NOT the category the user named. On Dankclaw +(2026-09-28) retired models were 510G, but podman images were **279G +reclaimable (95%)** and uv+pip caches **73G** — 352G of regenerable garbage, +bigger than the models. Lead with that finding; it changes the decision. + +Check for sparse files before claiming a saving: + +```sh +stat -c 'apparent=%s blocks=%b' ; du -sh --apparent-size ; du -sh +``` + +A 3.3G "log" (`logs/x4e-run.log`) was **4K real** (8 blocks × 512) — deleting +it frees nothing. `du` vs `du --apparent-size` finds these instantly. + +## 2. Classify against the runbook index + +Every payload directory maps to a runtime in the AGENTS.md retired-runbook +table. Three buckets: + +- **live** — referenced by a running service's env/unit. +- **rollback** — named as a rollback target in a runbook. ASK before + touching: they are the difference between a one-minute flip and a 30G + re-download. +- **retired** — maps to a retired runtime. Safe candidates. + +Trap: a directory can hold **both** a dead target and a live drafter +(frank-large's DFlash2 drafter lives beside the previous target). Move the +single file, never the directory, and say so in the manifest. + +## 3. The live-reference check — before moving anything + +```sh +for d in ; do + grep -rl "$d" --include="*.sh" --include="*.env" --include="*.service" \ + --include="*.container" --include="*.yaml" . | grep -v '^./old/' +done +``` + +Then **read every hit**; do not count them. Most are benign (comments, +retired fetch-script destinations, `retire_*_quadlet` helpers that move a +quadlet not a model) — but on Dankclaw this check found a live bug: +`serve-frank-small.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a +payload retired a week earlier. The env file always overrode it, so it was +invisible; a missing env file would have produced a "missing asset" naming a +model that no longer exists. + +**Durable lesson:** `:`-defaults in a serve script are never exercised in +production, so they rot silently. When reclaiming a payload, grep serve +scripts too — the fallback is a live reference in all but name. + +Also check service **env files** (`scripts/*.env`) for `MODEL_DIR`/`MODEL` +paths, not just unit `Volume=` lines: a payload can be mounted by path and +named by env. `models/logs` and small tool dirs are not cache — logs are +evidence, keep them. + +## 4. Stage, never delete + +```sh +STAGE=/home/dank/.AGENTTRASH/$(date +%Y%m%d)- +mkdir -p "$STAGE" +mv "$STAGE"/ +``` + +Same filesystem ⇒ instant and recoverable — **and reclaims no space**; say so +rather than implying space was freed. Write `MANIFEST.md` into the stage dir: +per item — path, size, owning runtime, when retired, the `mv` command to +restore. Record what you **deliberately kept** and why; that is what a future +reader needs. Always check for pre-existing staged trash +(`du -sh ~/.AGENTTRASH/*`): 118G from a 2026-09-21 cleanup sat un-purged; +report the running total, not just the new batch. Root-owned leftovers need +`sudo`, and check `swapon --show` first (a trashed swapfile must not be +active). + +## 5. Verify, then report — and do not purge + +```sh +systemctl is-active frank-large frank-small +curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health +``` + +Report: staged items and sizes, what was kept and why, what is **still not +freed**, remaining candidates untouched. Purge only on explicit approval. + +## Repo-flow gotcha: do NOT push before you deploy + +`scripts/deploy.sh` sets `BASE_REVISION=$(git rev-parse origin/main)`. If you +pushed separately, the diff is empty and the planner reports "changed +services: (none)" — looks like a planner bug, is not one. The tool is +designed for the **unpushed** state: edit → test → `deploy.sh` (it pushes, +then dispatches). If you already pushed, pull on the host instead, and reason +about whether a restart is actually owed. Any serve-script/env change needs +`./tests/run.sh` before commit; a `:`-default change is usually **inert** +(env overrides it) and owes no restart — but say so explicitly. + +# Purge and refresh protocol + +When the task is clearing/reclaiming/**refreshing** model caches on a serving +host, add these steps. + +## Fingerprint before deleting + +`ls -l --block-size=1 `; for LLM payloads compare each file against the +Hub's declared size. Fingerprints are the only way to answer "did the refresh +change anything" afterwards — **a refresh is also an update check**: if +upstream never changed, the bytes come back identical, which is itself the +finding. Sidecars are versioned **in place**: equal main-checkpoint sizes do +not prove the repo is unchanged, and a stale sidecar is invisible from its +name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560 while the main +checkpoint returned byte-identical). If a file is irreplaceable upstream +(delisted revision, gated repo), say so before deleting. + +## Confirm re-fetchability first + +Query the Hub anonymously; a gated repo 403s even with the host's token: + +```bash +curl -s "https://huggingface.co/api/models//tree/main?recursive=true&blobs=true" \ + | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]" +``` + +An error/empty result means it is NOT safe to delete. Filter by **quant +name**, never `endswith('.gguf')` — mradermacher ships large quants as +`.partNofM` fragments that filter hides. + +## Refresh per engine — know the recovery path BEFORE deleting + +- **frank-large (halogen)**: the engine fetches its own payload via + `HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but + leave it **existing and empty**, then start the unit. **Bind-mount trap:** + podman refuses a mount whose source is missing + (`Error: statfs /home/dank/models/halogen: no such file or directory`) and + the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB + at 130–240 MB/s ≈ 10–20 min, then load. +- **llama.cpp services**: no self-fetch; the serve script refuses to start + without weights. Pull yourself and verify: + +```bash +curl -fL -C - --retry 10 -o "$DEST/" \ + "https://huggingface.co//resolve/main/" # -f is mandatory +sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to LFS oid + GGUF magic +``` + +`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — a +gated repo once produced three 153-byte shards with identical sha256s and a +cheerful DONE line for 104 GiB. Always `-fL -C -`, compare byte counts to the +Hub's declared sizes, compare sha256 to the LFS oid. Assemble `.partNofM` +fragments with `cat part1 part2 part3 > name.gguf` in numeric order (per-part +hashes verify the whole — concatenation is deterministic). Public repos need +no token: omit `-H "Authorization: Bearer …"` from argv entirely (it is +world-readable in `/proc//cmdline`). + +## Restore, then verify behaviour — not health alone + +Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`), +`reset-failed`, `start` (root units need `sudo`). Watch the engine's own +fetch/load lines, then `/health`, `/v1/models`, and a **real completion** on +every restored endpoint. Read the engine's memory ledger (kv pool / weights / +blocks left) rather than assuming the config still fits. Sweep the vhosts too +(`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki` via +`curl -sk --resolve :443:192.168.0.143`) — a purge touches state the +whole host shares. + +## Things that bite + +- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark** — + the A* run commits ~116 GiB and leaves ~10 GiB available; cache churn risks + the OOM/state failures this fleet has already seen. +- **Reboot is the repair of last resort** if rootless podman's pause process + dies mid-work (`invalid internal status … could not find any running + process`); `podman system migrate` panics in 5.7.0. +- A same-filesystem trash reclaims no space; root-owned leftovers need + `sudo`; never `rm` without explicit permission. + +## Related + +- [[Model Swap and Rollback]] — the runbook index this classification reads +- [[Quadlet Lifecycle]] — restoring the unit after a payload refresh +- [[Remote Job Runner]] — supervising long re-fetches +- [[Deploy State Triage]] — the push-before-deploy trap in context diff --git a/serverAppConfigs-docs/Operations/Remote Job Runner.md b/serverAppConfigs-docs/Operations/Remote Job Runner.md new file mode 100644 index 0000000..fe428df --- /dev/null +++ b/serverAppConfigs-docs/Operations/Remote Job Runner.md @@ -0,0 +1,135 @@ +--- +type: reference +title: Remote Job Runner +description: Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally, the script, its invariants, verification, and reporting discipline. +tags: [operations, ssh, jobs, sentinel, detached] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Remote Job Runner + +Use for any long-running job on a remote host (benchmarks, model loads, +builds, sweeps) whose result must survive the session. Also whenever you catch +yourself writing `ssh host 'tail -f log | sed …'` as a watcher. + +## Why the obvious approach fails structurally + +`ssh host 'tail -f log | grep/sed …'` as a background watcher fails in ways +that all LOOK like "the job died": + +- the session broker **aborts long-lived ssh** — the watcher vanishes and the + result is lost; +- a `ready` regex can match the log's own **header** on the first line — the + supervisor reports success instantly while the job is still running; +- a `sed -n '/PAT/,$p'` filter **exits at the first match**, ending the pipe; +- output goes into the ssh stream, so an aborted session loses everything. + +## The mechanism: detach on the HOST, poll a sentinel file + +Completion is a **`.done` sentinel touched last**, not a log pattern. Status +is then one fast independent command with no long-lived process to lose. + +```bash +#!/usr/bin/env bash +# job.sh {start|status|wait|out|tail|watch} ... Jobs live in $JOBS. +set -uo pipefail +JOBS=/home/dank/jobs +mkdir -p "$JOBS" + +case "${1:-}" in +start) + NAME="$2"; shift 2 + [ "${1:-}" = "--" ] && shift + rm -f "$JOBS/$NAME.done" "$JOBS/$NAME.rc" + # setsid detaches from this session's process group: the critical bit. + setsid nohup bash -c ' + NAME="$1"; shift + "$@" > "'"$JOBS"'/$NAME.out" 2>&1 + echo $? > "'"$JOBS"'/$NAME.rc" + touch "'"$JOBS"'/$NAME.done" + ' _ "$NAME" "$@" < /dev/null > /dev/null 2>&1 & + echo "started $NAME" + ;; +status) + # Glob *.out (not *.done) or running jobs are invisible. + names="${2:-$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)}" + for n in $names; do + if [ -f "$JOBS/$n.done" ]; then + printf ' %-18s DONE rc=%s %s lines\n' "$n" \ + "$(cat "$JOBS/$n.rc" 2>/dev/null)" \ + "$(wc -l < "$JOBS/$n.out" | tr -d ' ')" + elif [ -f "$JOBS/$n.out" ]; then + printf ' %-18s RUN %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" | tr -d ' ')" + else + printf ' %-18s PENDING\n' "$n" + fi + done + ;; +wait) + NAME="$2"; LIMIT="${3:-3600}"; waited=0 + while [ ! -f "$JOBS/$NAME.done" ] && [ "$waited" -lt "$LIMIT" ]; do + sleep 5; waited=$((waited+5)) + done + [ -f "$JOBS/$NAME.done" ] \ + && echo "DONE $NAME rc=$(cat "$JOBS/$NAME.rc" 2>/dev/null)" \ + || echo "TIMEOUT $NAME after ${waited}s" + tail -"${4:-25}" "$JOBS/$NAME.out" 2>/dev/null + ;; +out) NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;; +tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;; +watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;; +*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;; +esac +``` + +```bash +job.sh start NAME -- # launch detached +job.sh status [NAME] # one-shot state +job.sh wait NAME 900 25 # block, then print the tail +job.sh out NAME 60 # retrieve any past result +job.sh tail [NAME] # live follow +job.sh watch # refreshing status table +``` + +## Invariants + +- **`.done` is touched LAST**, after the exit code is written — its presence + alone means the output file is final. +- **Glob `*.out` for status**, never `*.done` — globbing `.done` hides running + jobs entirely, the exact blindness this fixes. +- **Capture the exit code** (`rc=7` proved a self-test worked; without it a + job can "complete" having failed). +- **Never put the payload in a line you grep for** — the sentinel is the only + completion signal. +- **Never `pkill -f `** where the pattern also matches your own ssh + command line — it kills the session. Distinctive names or `pkill -x`. +- **Write helper scripts to a file and `scp` them.** Triple-nested + `ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated + quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line); + every failure disappeared once the script was a file. + +## Verify before trusting it (3am law) + +```bash +job.sh start selftest -- bash -c 'echo one; sleep 2; echo two; exit 7' +job.sh start slowtest -- bash -c 'echo start; sleep 30; echo end' +sleep 4; job.sh status # selftest DONE rc=7; slowtest RUN +``` + +Both paths, including the failure case, and the rc round-trip — a runner that +only proves the happy path is not trustworthy. + +## Reporting discipline + +- Give the user a real terminal when the run is long: + `alacritty --title "bench - dankserver" -e bash -c 'ssh -o + ServerAliveInterval=30 -t dank@192.168.0.235 "bash /home/dank/job.sh tail"' &` +- State the MECHANISM, not vibes: the job is durable on the host; + `job.sh out NAME` retrieves it later. Do not narrate "still loading" + repeatedly — check once, act, report the resolved result. +- When the user asks for status, one fast command, not a narration. + +## Related + +- [[LLM Bench Windows and Measurement]] — the sweeps this runner executes +- [[Deploy State Triage]] — the two-process deploy this lesson generalizes diff --git a/serverAppConfigs-docs/Operations/SSH and Remote Access.md b/serverAppConfigs-docs/Operations/SSH and Remote Access.md new file mode 100644 index 0000000..1ec5583 --- /dev/null +++ b/serverAppConfigs-docs/Operations/SSH and Remote Access.md @@ -0,0 +1,130 @@ +--- +type: reference +title: SSH and Remote Access +description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session. +tags: [ssh, remote, fish, gui, diagnosis] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# SSH and Remote Access + +## Passwordless SSH bootstrap without sshpass/expect + +Use when you must install your pubkey on remote hosts but `sshpass` and +`expect` are not installed and you only have a password. Modern OpenSSH +(≥ 8.4) can drive the password prompt from a helper program, so the password +flows through the **process env only** — never a file, never argv. + +1. **Generate a key** (if none): + ```bash + ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)" + ``` +2. **Create an askpass helper** that echoes the password from an env var (the + helper itself stores NO secret): + ```bash + printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh + chmod 700 /tmp/askpass.sh + ``` +3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY: + ```bash + env DEPLOY_PW='' \ + SSH_ASKPASS=/tmp/askpass.sh \ + SSH_ASKPASS_REQUIRE=force \ + setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host + ``` + Password via the transient `env` argument only. +4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names + (`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`). +5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` — + BatchMode fails fast if a password would be required; a clean success + proves the key works. + +Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a +TTY is present and prompts interactively (hangs in a non-interactive tool); +`setsid` detaches the controlling terminal — required on older OpenSSH even +with `force`; the helper must be executable (`chmod 700`) and print ONLY the +password; `IdentitiesOnly yes` stops ssh offering other agent keys that could +be rejected first ("Too many authentication failures"). Passwordless **sudo** +on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`). + +## Remote execution on a fish login shell + +Running remote commands or shipping files over SSH to a host whose login +shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN +`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the +workstation). + +1. **Auth**: `SSHPASS='' sshpass -e ssh -o + PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'` + — password via env, never argv (ps-visible). +2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login + shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at + *parse* time on fish (`Missing end to balance this for loop`). Always wrap + compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b || + echo MISSING; done"'` (escape inner `$`). +3. **File shipping**: never combine `cat > /remote/path < /remote/path' < localfile`, then verify byte-truth with + `md5sum` on both ends. +4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish + (`$status` there) — wrap in bash or drop the status echo; ssh's own exit + code propagates to the caller anyway. + +Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first +probe died on fish loop parsing; a `cat >` ship silently swallowed the +heredoc meant for `bash -s`, deploying wrong bytes until md5-compared. + +Related: dankserver's login shell is ZSH — never start an unquoted word with +`=` in an ssh one-liner (`echo ===` aborts the whole chain). + +## GUI hang — remote diagnosis over SSH + +Use when the desktop is frozen, VT switches do nothing, but SSH still works. + +1. **Find the smoking gun** — the kernel hung-task reporter already logged + it: + ```bash + journalctl -b -p err --since "-2 hours" --no-pager | tail -80 + ``` + Look for `INFO: task X blocked for more than N seconds` and `blocked on a + semaphore likely last held by task Y:PID` (modern kernels name the lock + holder outright). +2. **Full stacks** for victim and holder: + ```bash + journalctl -b -k --since "" --no-pager | tail -120 + cat /proc//status | head -5 # State: R (spinning) vs D (blocked) + ``` + Victim `D` + holder `R` = livelock/deadlock with the lock held; + `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning, + not progressing. +3. **Hardware or software?** GPU/CPU still alive, no Xid errors? + ```bash + nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv + journalctl -b -k --no-pager | grep -i xid + ``` + Instant answers with no Xid ⇒ driver lock issue, not hardware. +4. **Trigger window**: block start = first hung message timestamp − reported + seconds; then inspect user-session events around it + (`journalctl -b --since "" --until "" | grep -v audit`). +5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`), + USB workers, etc. + +**Recoverability verdict:** compositor in `D` state is unkillable and a +spinning kernel thread holding the lock cannot be poked — **the only exit is +reboot**: `sync && systemctl reboot --force --force` (double-force skips +stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on +something that can complete, e.g. storage), recovery may be possible by +clearing that dependency first. + +After diagnosing: map stack symbols to driver internals to name the exact +subsystem, check upstream for the exact symbols + driver version, and propose +targeted off-switches (`NVreg_RegistryDwords`, module params from +`/proc/driver/nvidia/params`) rather than blanket driver swaps. + +## Related + +- [[Fleet Ops Playbook]] — incident procedures that run over these sessions +- [[Remote Job Runner]] — long jobs that must survive session loss +- [[Deploy State Triage]] — the deploy flow these credentials serve diff --git a/serverAppConfigs-docs/Operations/index.md b/serverAppConfigs-docs/Operations/index.md new file mode 100644 index 0000000..ec58fa1 --- /dev/null +++ b/serverAppConfigs-docs/Operations/index.md @@ -0,0 +1,32 @@ +--- +type: index +description: Operational procedures for the fleet — job runners, disk and payload reclaim, container reachability, audits, SSH craft, and incident playbooks. +tags: [operations, index] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# Operations + +Procedures for operating the fleet. The LLM-specific pages live under +[LLM](../LLM/index.md); infrastructure and architecture under the +`../Infrastructure/` and `../Architecture/` folders. + +- [[Remote Job Runner]] — detached long-job execution with sentinel files; + why ssh tail-watchers fail +- [[Payload and Disk Reclaim]] — measure-classify-prove-stage-ask disk + procedure, plus the purge-and-refresh protocol +- [[Container Host Reachability]] — netavark/pasta reachability quirks and + the pasta port gate +- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot + stop-first, pre-commit iteration, incident-window reconstruction, WoL +- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security + posture, whole-fleet audit, bulk redaction, stash audits +- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish + login shells, GUI-hang diagnosis over ssh + +## Related +- [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md) + — monitoring config as code +- [Deploy State Triage](../Architecture/Deploy%20State%20Triage.md) — the + deploy plumbing +- [Quadlet Lifecycle](../Architecture/Quadlet%20Lifecycle.md) — the unit layer diff --git a/serverAppConfigs-docs/Research/PICO-8 Asset Standards.md b/serverAppConfigs-docs/Research/PICO-8 Asset Standards.md new file mode 100644 index 0000000..161745b --- /dev/null +++ b/serverAppConfigs-docs/Research/PICO-8 Asset Standards.md @@ -0,0 +1,62 @@ +--- +type: reference +title: PICO-8 Asset Standards +description: Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction, detail flow and hierarchy, sheet organization, and the cohesion checklist. +tags: [pico8, game-dev, pixel-art, standards] +timestamp: 2026-10-05T00:00:00-07:00 +--- + +# PICO-8 Asset Standards + +Ensure every PICO-8 asset (tileset, spritesheet, character sheet, or asset +sheet) is cohesive, production-grade, and ready to drop into a game without +rework. + +## Palette discipline + +- Use the PICO-8 16-color palette exclusively. +- Establish a dominant color family (2–3 related hues) for the main subject. +- Reserve accent colors for highlights, danger zones, and focal points only. +- Maintain consistent color roles across all assets in a sheet (if shadow is + dark blue on one tile, it is dark blue everywhere). + +## Perspective & lighting + +- Choose ONE perspective per sheet (top-down, side-view, or isometric) and + never mix. +- Apply light from a single consistent direction across all tiles and + sprites. +- Shadows follow the same shape logic on every asset (e.g. all cast shadows + point bottom-right). + +## Detail flow & hierarchy + +- Start tiles/props from the palette's dominant colors, then add edge + highlights and shading. +- Keep tile density uniform — avoid one hyper-detailed tile next to empty + flat tiles. +- Props and decorations should read clearly at 8×8 or 16×16 without zooming. + +## Sheet organization + +- Group related assets together (terrain together, props together, + characters together). +- Leave consistent spacing between assets for clean cropping. +- Label sections visually through grouping, not text. + +## Cohesion checks (before finishing a sheet) + +- All colors appear in the PICO-8 palette. +- Perspective does not flip mid-sheet. +- Lighting direction is consistent. +- Edge treatment (dither, outline, anti-aliasing style) matches across + assets. +- No orphan tiles that don't connect to any neighbor. + +## Common pitfalls + +- Mixing 8-bit and 16-bit detail levels. +- Using outline style on some tiles but not others. +- Forgetting the palette constraint — colors outside the 16 PICO-8 colors. +- Inconsistent tile edge connections (a grass tile that only connects to one + dirt variant). diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index db9589e..27a0ef6 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -115,6 +115,64 @@ macOS mini) as quadlets inside its podman machine — not on Dankserver. - [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack - [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices +## Skill Distillation (2026-10-05) + +The managed skill library distilled into the vault, one reference page per +skill-cluster. See [log.md](log.md) for the wave plan and results. + +**Operations:** + +- [[Remote Job Runner]] — detached long-job execution with sentinel files +- [[Payload and Disk Reclaim]] — disk reclaim: measure, classify, prove, + stage, ask; plus purge-and-refresh +- [[Container Host Reachability]] — netavark/pasta reachability quirks and + the pasta port gate +- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot + stop-first, pre-commit iteration, incident-window reconstruction, WoL +- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security + posture, whole-fleet audit, bulk redaction +- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells, + GUI-hang diagnosis over ssh + +**Architecture:** + +- [[Stale Signal Audit]] — classifying green-but-dead and down-but-fine + signals +- [[Quadlet Lifecycle]] — quadlet units, drift, crash-loops, compose + cutover +- [[Deploy State Triage]] — revision markers, pending lockouts, deploy + interruptions, shared-checkout hygiene + +**Infrastructure:** + +- [[Observability Provisioning]] — Prometheus/Grafana config as code +- [[Homepage Status Probes]] — the status page's checker, vantage rules, + and triage + +**LLM cross-cutting (full index: [LLM/index.md](LLM/index.md)):** + +- [[GGUF Compatibility]] · [[Llama Cpp Serving Sizing]] · + [[Llama Cpp Multi-Device Splits]] · [[PCIe Link and Atomics]] · + [[RDNA4 Container Serving]] · [[RDNA4 MoE Expert Offload VRAM Budget]] · + [[vLLM Boot Forensics and Config Interrogation]] · + [[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] · + [[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] · + [[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] · + [[FreeToken Builds]] + +**Agent:** + +- [[OMP Harness]] — compaction arithmetic, per-model samplers, preamble + audits +- [[Agent Docs and Edits]] — doc-structure audits, hashline edit recovery +- [[Command Center Dotfiles]] — the ~/.omp repo-as-deployment + +**Desktop / Research:** + +- [[Noctalia and Niri]] — the desktop shell, palette recolor, template + rendering, rotation pinning, iCX3 RGB +- [[PICO-8 Asset Standards]] — cohesive PICO-8 asset sheets + ## LLM Fleet The LLM inference fleet on the two hosts. Full index: diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 91ee769..bf465be 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,5 +1,6 @@ # Bundle Change Log + ## 2026-10-05 - **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on dankserver.** Built FreeToken at PR head `77a7201` on the host into the @@ -26,7 +27,45 @@ Dankclaw Prometheus llm-fleet scrape of :7770 reads down until repointed. Rollback: tag `frank-primary-vllm-shapeG-20261003`. Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`. -## 2026-10-05 +- **Skill library distilled into the vault (waves 1–7).** All ~173 managed + skills read in full and merged, one polished reference page per + skill-cluster: + - **LLM cross-cutting (16 pages, `LLM/`):** GGUF Compatibility, + Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and + Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, + vLLM Boot Forensics and Config Interrogation, vLLM Porting and Compile + Triage, LLM Bench Windows and Measurement, LLM Performance Diagnosis, + Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split + Triage, FreeToken Builds, plus deltas folded into the retired runbooks + (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, + FLM pack registration, ik hybrid envelope, frank-medium split + rebalance, frank-primary llama.cpp GGUF preflight, batch-ubatch split + section, bench-window D-state/stale-read notes, second-endpoint + bring-up in Model Swap). + - **Architecture (3):** Stale Signal Audit, Quadlet Lifecycle, Deploy + State Triage. + - **Operations (6 + index):** Remote Job Runner, Payload and Disk + Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit + and Posture, SSH and Remote Access. + - **Infrastructure (2 + updates):** Observability Provisioning, Homepage + Status Probes; updates to Authentik (guard-chain flip recipe, `ak + shell` mechanics, user-provisioning playbook, outpost netns probe), + Anubis (guard-chain layer diagnosis), LAN-DNS (per-client resolver + drop-in + browser DoH bypass, Porkbun zone ops), Swap Configuration + (filesystem/rename caveats), OpenWebUI (introspection, verification + battery, ComfyUI imagegen), Hermes (config-home drift, live config + edits, auxiliary routing, password rotation — the pre-existing page was + preserved and extended). + - **Host pages:** FrankBook-Pro ARR-Stack filled with the API-version + split, schema-first probing, auth-posture method, auth-config trap, and + the qBittorrent E2E verification procedure added to [[FrankBook-Pro/qbittorrent]]. + - **Desktop/Agent/Research:** Noctalia and Niri (config layers, remote + IPC, palette recolor, offline template render, rotation pinning, iCX3 + RGB), Agent/ (OMP Harness, Agent Docs and Edits, Command Center + Dotfiles, index), PICO-8 Asset Standards. + - Index updated: `LLM/index.md` gained a Cross-Cutting References + section; Operations/ and Agent/ got index pages. AGENTS.md now points + at the new sections. Link checker + full test suite green. - **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright project.** The `11-admin` walk included the six `*arr`/qBittorrent admin links that terminate on 192.168.0.112, so every deploy gate died there when